[root@localhost ~]# obdiag display-trace 3911f2de-0dee-11f0-9a32-00163eb23b0c
obdiag version: 3.2.0
[2025-03-31 13:08:45.935] [DEBUG] - cmd: obdiag rca run
[2025-03-31 13:08:45.935] [DEBUG] - opts: {'inner_config': None, 'scene': 'major_hold', 'store_dir': './obdiag_rca/', 'env': None, 'report_type': 'table', 'c': '/root/.obdiag/config.yml', 'config': None}
[2025-03-31 13:08:45.935] [INFO] obdiag version: 3.2.0
[2025-03-31 13:08:45.935] [DEBUG] - mkdir /opt/oceanbase-diagnostic-tool/conf/inner_config.yml
[2025-03-31 13:08:45.939] [DEBUG] - mkdir /root/.obdiag/config.yml
[2025-03-31 13:08:46.386] [DEBUG] - connect databse ...
[2025-03-31 13:08:46.595] [DEBUG] - connect databse ...
[2025-03-31 13:08:46.596] [DEBUG] - RCAHandler.init store dir: ./obdiag_rca/
[2025-03-31 13:08:46.596] [DEBUG] - rca result save_path is :./obdiag_rca/
[2025-03-31 13:08:46.596] [DEBUG] - get observer version, by sql
[2025-03-31 13:08:46.596] [DEBUG] - start get_observer_version_by_sql . input: 192.168.1.62:2881
[2025-03-31 13:08:46.598] [DEBUG] - connect databse ...
[2025-03-31 13:08:46.599] [DEBUG] - get_observer_version_by_sql ob_version_info is ('5.7.25-OceanBase_CE-v4.3.5.1',)
[2025-03-31 13:08:46.600] [DEBUG] - RCAHandler.init get observer version: 4.3.5.1
[2025-03-31 13:08:46.600] [DEBUG] - RCAHandler.init get observer version: 4.3.5.1
[2025-03-31 13:08:46.632] [DEBUG] - RCAHandler init.cluster:sgkydb, init.nodes:[{'ip': '192.168.1.62', 'ssh_username': 'root', 'ssh_port': '22', 'home_path': '/root/observer', 'data_dir': '/root/observer/store', 'redo_dir': '/root/observer/store', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': <src.common.ssh_client.ssh.SshClient object at 0x7ff2c72ed1c0>}, {'ip': '192.168.1.63', 'ssh_username': 'root', 'ssh_port': '22', 'home_path': '/root/observer', 'data_dir': '/root/observer/store', 'redo_dir': '/root/observer/store', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': <src.common.ssh_client.ssh.SshClient object at 0x7ff2c72f86a0>}, {'ip': '192.168.1.64', 'ssh_username': 'root', 'ssh_port': '22', 'home_path': '/root/observer', 'data_dir': '/root/observer/store', 'redo_dir': '/root/observer/store', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': <src.common.ssh_client.ssh.SshClient object at 0x7ff2c7285940>}, {'ip': '192.168.1.65', 'ssh_username': 'root', 'ssh_port': '22', 'home_path': '/root/observer', 'data_dir': '/root/observer/store', 'redo_dir': '/root/observer/store', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': <src.common.ssh_client.ssh.SshClient object at 0x7ff2c72899d0>}], init.obproxy_nodes:[], init.store_dir:./obdiag_rca/
[2025-03-31 13:08:46.632] [DEBUG] - major_hold store_dir:./obdiag_rca/obdiag_major_hold_20250331130846
[2025-03-31 13:08:46.633] [DEBUG] - get observer version, by sql
[2025-03-31 13:08:46.633] [DEBUG] - start get_observer_version_by_sql . input: 192.168.1.62:2881
[2025-03-31 13:08:46.635] [DEBUG] - connect databse ...
[2025-03-31 13:08:46.636] [DEBUG] - get_observer_version_by_sql ob_version_info is ('5.7.25-OceanBase_CE-v4.3.5.1',)
[2025-03-31 13:08:46.636] [DEBUG] - major_hold init success
[2025-03-31 13:08:46.636] [DEBUG] - add record: check major task is error or not
[2025-03-31 13:08:46.667] [DEBUG] - add record: CDB_OB_MAJOR_COMPACTION is not exist IS_ERROR='YES'
[2025-03-31 13:08:46.667] [DEBUG] - add record: check on CDB_OB_MAJOR_COMPACTION IS_ERROR is 'YES'.  sql:select * from
[2025-03-31 13:08:46.667] [DEBUG] oceanbase.CDB_OB_MAJOR_COMPACTION where IS_ERROR="YES";
[2025-03-31 13:08:46.678] [DEBUG] - add record: __all_virtual_compaction_diagnose_info have status='FAILED',the tenant is ['1002', '1002', '1002',
[2025-03-31 13:08:46.678] [DEBUG] '1', '1001', '1001', '1002', '1002', '1002', '1009']
[2025-03-31 13:08:46.685] [DEBUG] - add record: merge tasks that have not ended beyond the expected time,the tenant_id is ['1002', '1009', '1',
[2025-03-31 13:08:46.685] [DEBUG] '1001']
[2025-03-31 13:08:46.685] [DEBUG] - merge tasks that have not ended beyond the expected time,the tenant_id is ['1002', '1009', '1', '1001']
[2025-03-31 13:08:46.685] [DEBUG] - add suggest: some tenants need execute MajorHoldScene. :['1001', '1009', '1', '1002']
[2025-03-31 13:08:46.685] [DEBUG] - On CDB_OB_MAJOR_COMPACTION
[2025-03-31 13:08:46.685] [DEBUG] - tenant_id is 1001
[2025-03-31 13:08:46.685] [DEBUG] - add record: tenant_id is 1001
[2025-03-31 13:08:46.685] [DEBUG] - add record: step:1 get CDB_OB_MAJOR_COMPACTION data
[2025-03-31 13:08:46.689] [DEBUG] - add record: on CDB_OB_MAJOR_COMPACTION where status='COMPACTING';result:[{'TENANT_ID': 1001, 'FROZEN_SCN':
[2025-03-31 13:08:46.689] [DEBUG] 1743098400293280000, 'FROZEN_TIME': datetime.datetime(2025, 3, 28, 2, 0, 0, 293280),
[2025-03-31 13:08:46.689] [DEBUG] 'GLOBAL_BROADCAST_SCN': 1743098400293280000, 'LAST_SCN': 1743012003356867000, 'LAST_FINISH_TIME':
[2025-03-31 13:08:46.689] [DEBUG] datetime.datetime(2025, 3, 27, 2, 6, 10, 38376), 'START_TIME': datetime.datetime(2025, 3, 28, 2, 0,
[2025-03-31 13:08:46.689] [DEBUG] 0, 604113), 'STATUS': 'COMPACTING', 'IS_ERROR': 'NO', 'IS_SUSPENDED': 'NO', 'INFO': ''}]
[2025-03-31 13:08:46.690] [DEBUG] - add record: step:2&3 get __all_virtual_compaction_diagnose_info and check the diagnose type
[2025-03-31 13:08:46.726] [DEBUG] - add record: on __all_virtual_compaction_diagnose_info;result:[{'svr_ip': '192.168.1.62', 'svr_port': 2882,
[2025-03-31 13:08:46.726] [DEBUG] 'tenant_id': 1, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 215, 'status': 'FAILED',
[2025-03-31 13:08:46.726] [DEBUG] 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 4, 875838), 'diagnose_info': 'error_no=-4016,las
[2025-03-31 13:08:46.726] [DEBUG] t_error_time=1743397726341044,error_trace=YB420A0B093E-00063125A22842DA-0-0,location=ob_partition_me
[2025-03-31 13:08:46.726] [DEBUG] rger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=215;compaction_scn=1
[2025-03-31 13:08:46.726] [DEBUG] 743098403563081000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.65',
[2025-03-31 13:08:46.726] [DEBUG] 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE', 'ls_id': 1003, 'tablet_id': 200391,
[2025-03-31 13:08:46.726] [DEBUG] 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 3, 32, 430045),
[2025-03-31 13:08:46.726] [DEBUG] 'diagnose_info': 'error_no=-4034,last_error_time=1743397683450165,error_trace=YB420A0B0941-00063125A
[2025-03-31 13:08:46.726] [DEBUG] 55389B6-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls
[2025-03-31 13:08:46.727] [DEBUG] _id=1003;tablet_id=200391;compaction_scn=1743098400401915000;exec_mode="EXEC_MODE_LOCAL",concurrent_
[2025-03-31 13:08:46.727] [DEBUG] cnt=1"'}, {'svr_ip': '192.168.1.65', 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE',
[2025-03-31 13:08:46.727] [DEBUG] 'ls_id': 1003, 'tablet_id': 200391, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3,
[2025-03-31 13:08:46.727] [DEBUG] 28, 2, 3, 32, 430045), 'diagnose_info': 'error_no=-4034,last_error_time=1743397683450165,error_trace
[2025-03-31 13:08:46.727] [DEBUG] =YB420A0B0941-00063125A55389B6-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJO
[2025-03-31 13:08:46.727] [DEBUG] R_MERGE/MEDIUM_MERGE;ls_id=1003;tablet_id=200391;compaction_scn=1743098400401915000;exec_mode="EXEC_
[2025-03-31 13:08:46.727] [DEBUG] MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.65', 'svr_port': 2882, 'tenant_id': 1002,
[2025-03-31 13:08:46.727] [DEBUG] 'type': 'MINOR_MERGE', 'ls_id': 1003, 'tablet_id': 200391, 'status': 'FAILED', 'create_time':
[2025-03-31 13:08:46.727] [DEBUG] datetime.datetime(2025, 3, 27, 23, 55, 3, 228902), 'diagnose_info': 'error_no=-4034,last_error_time=
[2025-03-31 13:08:46.727] [DEBUG] 1743098464251531,error_trace=YB420A0B0941-00063125A5536BF5-0-0,location=ob_partition_merger.cpp:124(
[2025-03-31 13:08:46.727] [DEBUG] prepare_merge),warning="MINOR_EXECUTE;ls_id=1003;tablet_id=200391;compaction_scn=1743091213157973000
[2025-03-31 13:08:46.727] [DEBUG] ;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port': 2882,
[2025-03-31 13:08:46.727] [DEBUG] 'tenant_id': 1001, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 331, 'status': 'FAILED',
[2025-03-31 13:08:46.727] [DEBUG] 'create_time': datetime.datetime(2025, 3, 28, 2, 5, 9, 445088), 'diagnose_info': 'error_no=-4016,las
[2025-03-31 13:08:46.727] [DEBUG] t_error_time=1743397691267177,error_trace=YB420A0B093F-00063125A6486AF8-0-0,location=ob_partition_me
[2025-03-31 13:08:46.727] [DEBUG] rger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=331;compaction_scn=1
[2025-03-31 13:08:46.727] [DEBUG] 743098400293280000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63',
[2025-03-31 13:08:46.727] [DEBUG] 'svr_port': 2882, 'tenant_id': 1001, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 121, 'status':
[2025-03-31 13:08:46.727] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 6, 5, 301487), 'diagnose_info': 'error_no
[2025-03-31 13:08:46.727] [DEBUG] =-4016,last_error_time=1743397691440167,error_trace=YB420A0B093F-00063125A6486AF9-0-0,location=ob_pa
[2025-03-31 13:08:46.727] [DEBUG] rtition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=121;compac
[2025-03-31 13:08:46.727] [DEBUG] tion_scn=1743098400293280000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip':
[2025-03-31 13:08:46.727] [DEBUG] '192.168.1.63', 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE', 'ls_id': 1001,
[2025-03-31 13:08:46.727] [DEBUG] 'tablet_id': 201413, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 25,
[2025-03-31 13:08:46.727] [DEBUG] 890814), 'diagnose_info': 'error_no=-4034,last_error_time=1743397687211333,error_trace=YB420A0B093F-
[2025-03-31 13:08:46.727] [DEBUG] 00063125A6296AB2-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM
[2025-03-31 13:08:46.727] [DEBUG] _MERGE;ls_id=1001;tablet_id=201413;compaction_scn=1743098400401915000;exec_mode="EXEC_MODE_LOCAL",co
[2025-03-31 13:08:46.728] [DEBUG] ncurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port': 2882, 'tenant_id': 1002, 'type':
[2025-03-31 13:08:46.728] [DEBUG] 'MEDIUM_MERGE', 'ls_id': 1001, 'tablet_id': 201413, 'status': 'FAILED', 'create_time':
[2025-03-31 13:08:46.728] [DEBUG] datetime.datetime(2025, 3, 28, 2, 4, 25, 890814), 'diagnose_info': 'error_no=-4034,last_error_time=1
[2025-03-31 13:08:46.728] [DEBUG] 743397687211333,error_trace=YB420A0B093F-00063125A6296AB2-0-0,location=ob_partition_merger.cpp:124(p
[2025-03-31 13:08:46.728] [DEBUG] repare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1001;tablet_id=201413;compaction_scn=174309840
[2025-03-31 13:08:46.728] [DEBUG] 0401915000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port':
[2025-03-31 13:08:46.728] [DEBUG] 2882, 'tenant_id': 1002, 'type': 'MINOR_MERGE', 'ls_id': 1001, 'tablet_id': 201413, 'status':
[2025-03-31 13:08:46.728] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 1, 44, 17, 825107), 'diagnose_info': 'error_
[2025-03-31 13:08:46.728] [DEBUG] no=-4034,last_error_time=1743098494302962,error_trace=YB420A0B093F-00063125A6284486-0-0,location=ob_
[2025-03-31 13:08:46.728] [DEBUG] partition_merger.cpp:124(prepare_merge),warning="MINOR_EXECUTE;ls_id=1001;tablet_id=201413;compactio
[2025-03-31 13:08:46.728] [DEBUG] n_scn=1743097455815495000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63',
[2025-03-31 13:08:46.728] [DEBUG] 'svr_port': 2882, 'tenant_id': 1009, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 323, 'status':
[2025-03-31 13:08:46.728] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 10, 781639), 'diagnose_info': 'error_n
[2025-03-31 13:08:46.728] [DEBUG] o=-4016,last_error_time=1743397613587786,error_trace=YB420A0B093F-00063125BBE8516E-0-0,location=ob_p
[2025-03-31 13:08:46.728] [DEBUG] artition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=323;compa
[2025-03-31 13:08:46.728] [DEBUG] ction_scn=1743098404774537000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}]
[2025-03-31 13:08:46.728] [DEBUG] - add record: diagnose_info:error_no=-4016,last_error_time=1743397726341044,error_trace=YB420A0B093E-00063125A2284
[2025-03-31 13:08:46.728] [DEBUG] 2DA-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=
[2025-03-31 13:08:46.728] [DEBUG] 1;tablet_id=215;compaction_scn=1743098403563081000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"
[2025-03-31 13:08:46.728] [DEBUG] - add record: diagnose_info type is error_no
[2025-03-31 13:08:46.736] [DEBUG] - add record: #2&3 on __all_virtual_compaction_diagnose_info get data failed
[2025-03-31 13:08:46.737] [WARNING] #2&3 MajorHoldScene execute exception: tuple index out of range
[2025-03-31 13:08:46.737] [DEBUG] - add record: step:4 get GV$OB_COMPACTION_PROGRESS whit tenant_id:1001
[2025-03-31 13:08:46.741] [DEBUG] - add record: global_broadcast_scn is 1743098400293280000
[2025-03-31 13:08:46.744] [DEBUG] - add record: last_scn is 1743012003356867000
[2025-03-31 13:08:46.749] [DEBUG] - add record: tenant_id:1001 OB_COMPACTION_PROGRESS_data_global_broadcast_scn save on ./obdiag_rca/obdiag_major_ho
[2025-03-31 13:08:46.749] [DEBUG] ld_20250331130846/rca_major_hold_1001_OB_COMPACTION_PROGRESS_data_global_broadcast_scn
[2025-03-31 13:08:46.754] [DEBUG] - add record: tenant_id:1001 OB_COMPACTION_PROGRESS_data_last_scn save on ./obdiag_rca/obdiag_major_hold_202503311
[2025-03-31 13:08:46.754] [DEBUG] 30846/rca_major_hold_1001_OB_COMPACTION_PROGRESS_data_last_scn
[2025-03-31 13:08:46.758] [DEBUG] - add record: step:5 get OB_COMPACTION_SUGGESTIONS
[2025-03-31 13:08:46.761] [DEBUG] - add record: tenant_id:1001 OB_COMPACTION_PROGRESS_data_last_scn save on
[2025-03-31 13:08:46.761] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/rca_major_hold_1001_OB_COMPACTION_SUGGESTIONS_info
[2025-03-31 13:08:46.761] [DEBUG] - step:6 get dmesg by dmesg -T
[2025-03-31 13:08:46.762] [DEBUG] - Execute Shell command on server 192.168.1.62:dmesg -T > /tmp/dmesg_remote_10_11_9_62.log
[2025-03-31 13:08:46.785] [DEBUG] - download file: /tmp/dmesg_remote_10_11_9_62.log to ./obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log/dmesg_remote_10_11_9_62.log
[2025-03-31 13:08:46.844] [DEBUG] - Download 192.168.1.62:/tmp/dmesg_remote_10_11_9_62.log
[2025-03-31 13:08:46.848] [DEBUG] - add record: download /tmp/dmesg_remote_10_11_9_62.log to
[2025-03-31 13:08:46.849] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log/dmesg_remote_10_11_9_62.log
[2025-03-31 13:08:46.849] [DEBUG] - Execute Shell command on server 192.168.1.63:dmesg -T > /tmp/dmesg_remote_10_11_9_63.log
[2025-03-31 13:08:46.872] [DEBUG] - download file: /tmp/dmesg_remote_10_11_9_63.log to ./obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log/dmesg_remote_10_11_9_63.log
[2025-03-31 13:08:46.930] [DEBUG] - Download 192.168.1.63:/tmp/dmesg_remote_10_11_9_63.log
[2025-03-31 13:08:46.934] [DEBUG] - add record: download /tmp/dmesg_remote_10_11_9_63.log to
[2025-03-31 13:08:46.935] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log/dmesg_remote_10_11_9_63.log
[2025-03-31 13:08:46.935] [DEBUG] - Execute Shell command on server 192.168.1.64:dmesg -T > /tmp/dmesg_remote_10_11_9_64.log
[2025-03-31 13:08:46.957] [DEBUG] - download file: /tmp/dmesg_remote_10_11_9_64.log to ./obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log/dmesg_remote_10_11_9_64.log
[2025-03-31 13:08:47.016] [DEBUG] - Download 192.168.1.64:/tmp/dmesg_remote_10_11_9_64.log
[2025-03-31 13:08:47.021] [DEBUG] - add record: download /tmp/dmesg_remote_10_11_9_64.log to
[2025-03-31 13:08:47.021] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log/dmesg_remote_10_11_9_64.log
[2025-03-31 13:08:47.021] [DEBUG] - Execute Shell command on server 192.168.1.65:dmesg -T > /tmp/dmesg_remote_10_11_9_65.log
[2025-03-31 13:08:47.044] [DEBUG] - download file: /tmp/dmesg_remote_10_11_9_65.log to ./obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log/dmesg_remote_10_11_9_65.log
[2025-03-31 13:08:47.103] [DEBUG] - Download 192.168.1.65:/tmp/dmesg_remote_10_11_9_65.log
[2025-03-31 13:08:47.108] [DEBUG] - add record: download /tmp/dmesg_remote_10_11_9_65.log to
[2025-03-31 13:08:47.108] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log/dmesg_remote_10_11_9_65.log
[2025-03-31 13:08:47.108] [DEBUG] - add suggest: send the ./obdiag_rca/obdiag_major_hold_20250331130846 to the oceanbase community
[2025-03-31 13:08:47.108] [DEBUG] - tenant_id is 1009
[2025-03-31 13:08:47.108] [DEBUG] - add record: tenant_id is 1009
[2025-03-31 13:08:47.108] [DEBUG] - add record: step:1 get CDB_OB_MAJOR_COMPACTION data
[2025-03-31 13:08:47.141] [DEBUG] - add record: on CDB_OB_MAJOR_COMPACTION where status='COMPACTING';result:[{'TENANT_ID': 1009, 'FROZEN_SCN':
[2025-03-31 13:08:47.141] [DEBUG] 1743098404774537000, 'FROZEN_TIME': datetime.datetime(2025, 3, 28, 2, 0, 4, 774537),
[2025-03-31 13:08:47.141] [DEBUG] 'GLOBAL_BROADCAST_SCN': 1743098404774537000, 'LAST_SCN': 1743012002904300000, 'LAST_FINISH_TIME':
[2025-03-31 13:08:47.141] [DEBUG] datetime.datetime(2025, 3, 27, 2, 6, 18, 195597), 'START_TIME': datetime.datetime(2025, 3, 28, 2, 0,
[2025-03-31 13:08:47.141] [DEBUG] 6, 39257), 'STATUS': 'COMPACTING', 'IS_ERROR': 'NO', 'IS_SUSPENDED': 'NO', 'INFO': ''}]
[2025-03-31 13:08:47.141] [DEBUG] - add record: step:2&3 get __all_virtual_compaction_diagnose_info and check the diagnose type
[2025-03-31 13:08:47.175] [DEBUG] - add record: on __all_virtual_compaction_diagnose_info;result:[{'svr_ip': '192.168.1.62', 'svr_port': 2882,
[2025-03-31 13:08:47.176] [DEBUG] 'tenant_id': 1, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 215, 'status': 'FAILED',
[2025-03-31 13:08:47.176] [DEBUG] 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 4, 875838), 'diagnose_info': 'error_no=-4016,las
[2025-03-31 13:08:47.176] [DEBUG] t_error_time=1743397726341044,error_trace=YB420A0B093E-00063125A22842DA-0-0,location=ob_partition_me
[2025-03-31 13:08:47.176] [DEBUG] rger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=215;compaction_scn=1
[2025-03-31 13:08:47.176] [DEBUG] 743098403563081000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.65',
[2025-03-31 13:08:47.176] [DEBUG] 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE', 'ls_id': 1003, 'tablet_id': 200391,
[2025-03-31 13:08:47.176] [DEBUG] 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 3, 32, 430045),
[2025-03-31 13:08:47.176] [DEBUG] 'diagnose_info': 'error_no=-4034,last_error_time=1743397683450165,error_trace=YB420A0B0941-00063125A
[2025-03-31 13:08:47.176] [DEBUG] 55389B6-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls
[2025-03-31 13:08:47.176] [DEBUG] _id=1003;tablet_id=200391;compaction_scn=1743098400401915000;exec_mode="EXEC_MODE_LOCAL",concurrent_
[2025-03-31 13:08:47.176] [DEBUG] cnt=1"'}, {'svr_ip': '192.168.1.65', 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE',
[2025-03-31 13:08:47.176] [DEBUG] 'ls_id': 1003, 'tablet_id': 200391, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3,
[2025-03-31 13:08:47.176] [DEBUG] 28, 2, 3, 32, 430045), 'diagnose_info': 'error_no=-4034,last_error_time=1743397683450165,error_trace
[2025-03-31 13:08:47.176] [DEBUG] =YB420A0B0941-00063125A55389B6-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJO
[2025-03-31 13:08:47.176] [DEBUG] R_MERGE/MEDIUM_MERGE;ls_id=1003;tablet_id=200391;compaction_scn=1743098400401915000;exec_mode="EXEC_
[2025-03-31 13:08:47.176] [DEBUG] MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.65', 'svr_port': 2882, 'tenant_id': 1002,
[2025-03-31 13:08:47.176] [DEBUG] 'type': 'MINOR_MERGE', 'ls_id': 1003, 'tablet_id': 200391, 'status': 'FAILED', 'create_time':
[2025-03-31 13:08:47.176] [DEBUG] datetime.datetime(2025, 3, 27, 23, 55, 3, 228902), 'diagnose_info': 'error_no=-4034,last_error_time=
[2025-03-31 13:08:47.176] [DEBUG] 1743098464251531,error_trace=YB420A0B0941-00063125A5536BF5-0-0,location=ob_partition_merger.cpp:124(
[2025-03-31 13:08:47.176] [DEBUG] prepare_merge),warning="MINOR_EXECUTE;ls_id=1003;tablet_id=200391;compaction_scn=1743091213157973000
[2025-03-31 13:08:47.176] [DEBUG] ;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port': 2882,
[2025-03-31 13:08:47.176] [DEBUG] 'tenant_id': 1001, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 331, 'status': 'FAILED',
[2025-03-31 13:08:47.176] [DEBUG] 'create_time': datetime.datetime(2025, 3, 28, 2, 5, 9, 445088), 'diagnose_info': 'error_no=-4016,las
[2025-03-31 13:08:47.176] [DEBUG] t_error_time=1743397691267177,error_trace=YB420A0B093F-00063125A6486AF8-0-0,location=ob_partition_me
[2025-03-31 13:08:47.176] [DEBUG] rger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=331;compaction_scn=1
[2025-03-31 13:08:47.176] [DEBUG] 743098400293280000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63',
[2025-03-31 13:08:47.176] [DEBUG] 'svr_port': 2882, 'tenant_id': 1001, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 121, 'status':
[2025-03-31 13:08:47.176] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 6, 5, 301487), 'diagnose_info': 'error_no
[2025-03-31 13:08:47.176] [DEBUG] =-4016,last_error_time=1743397691440167,error_trace=YB420A0B093F-00063125A6486AF9-0-0,location=ob_pa
[2025-03-31 13:08:47.176] [DEBUG] rtition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=121;compac
[2025-03-31 13:08:47.176] [DEBUG] tion_scn=1743098400293280000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip':
[2025-03-31 13:08:47.176] [DEBUG] '192.168.1.63', 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE', 'ls_id': 1001,
[2025-03-31 13:08:47.176] [DEBUG] 'tablet_id': 201413, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 25,
[2025-03-31 13:08:47.176] [DEBUG] 890814), 'diagnose_info': 'error_no=-4034,last_error_time=1743397687211333,error_trace=YB420A0B093F-
[2025-03-31 13:08:47.177] [DEBUG] 00063125A6296AB2-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM
[2025-03-31 13:08:47.177] [DEBUG] _MERGE;ls_id=1001;tablet_id=201413;compaction_scn=1743098400401915000;exec_mode="EXEC_MODE_LOCAL",co
[2025-03-31 13:08:47.177] [DEBUG] ncurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port': 2882, 'tenant_id': 1002, 'type':
[2025-03-31 13:08:47.177] [DEBUG] 'MEDIUM_MERGE', 'ls_id': 1001, 'tablet_id': 201413, 'status': 'FAILED', 'create_time':
[2025-03-31 13:08:47.177] [DEBUG] datetime.datetime(2025, 3, 28, 2, 4, 25, 890814), 'diagnose_info': 'error_no=-4034,last_error_time=1
[2025-03-31 13:08:47.177] [DEBUG] 743397687211333,error_trace=YB420A0B093F-00063125A6296AB2-0-0,location=ob_partition_merger.cpp:124(p
[2025-03-31 13:08:47.177] [DEBUG] repare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1001;tablet_id=201413;compaction_scn=174309840
[2025-03-31 13:08:47.177] [DEBUG] 0401915000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port':
[2025-03-31 13:08:47.177] [DEBUG] 2882, 'tenant_id': 1002, 'type': 'MINOR_MERGE', 'ls_id': 1001, 'tablet_id': 201413, 'status':
[2025-03-31 13:08:47.177] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 1, 44, 17, 825107), 'diagnose_info': 'error_
[2025-03-31 13:08:47.177] [DEBUG] no=-4034,last_error_time=1743098494302962,error_trace=YB420A0B093F-00063125A6284486-0-0,location=ob_
[2025-03-31 13:08:47.177] [DEBUG] partition_merger.cpp:124(prepare_merge),warning="MINOR_EXECUTE;ls_id=1001;tablet_id=201413;compactio
[2025-03-31 13:08:47.177] [DEBUG] n_scn=1743097455815495000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63',
[2025-03-31 13:08:47.177] [DEBUG] 'svr_port': 2882, 'tenant_id': 1009, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 323, 'status':
[2025-03-31 13:08:47.177] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 10, 781639), 'diagnose_info': 'error_n
[2025-03-31 13:08:47.177] [DEBUG] o=-4016,last_error_time=1743397613587786,error_trace=YB420A0B093F-00063125BBE8516E-0-0,location=ob_p
[2025-03-31 13:08:47.177] [DEBUG] artition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=323;compa
[2025-03-31 13:08:47.177] [DEBUG] ction_scn=1743098404774537000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}]
[2025-03-31 13:08:47.177] [DEBUG] - add record: diagnose_info:error_no=-4016,last_error_time=1743397726341044,error_trace=YB420A0B093E-00063125A2284
[2025-03-31 13:08:47.177] [DEBUG] 2DA-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=
[2025-03-31 13:08:47.177] [DEBUG] 1;tablet_id=215;compaction_scn=1743098403563081000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"
[2025-03-31 13:08:47.177] [DEBUG] - add record: diagnose_info type is error_no
[2025-03-31 13:08:47.185] [DEBUG] - add record: #2&3 on __all_virtual_compaction_diagnose_info get data failed
[2025-03-31 13:08:47.185] [WARNING] #2&3 MajorHoldScene execute exception: tuple index out of range
[2025-03-31 13:08:47.185] [DEBUG] - add record: step:4 get GV$OB_COMPACTION_PROGRESS whit tenant_id:1009
[2025-03-31 13:08:47.190] [DEBUG] - add record: global_broadcast_scn is 1743098404774537000
[2025-03-31 13:08:47.193] [DEBUG] - add record: last_scn is 1743012002904300000
[2025-03-31 13:08:47.198] [DEBUG] - add record: tenant_id:1009 OB_COMPACTION_PROGRESS_data_global_broadcast_scn save on ./obdiag_rca/obdiag_major_ho
[2025-03-31 13:08:47.198] [DEBUG] ld_20250331130846/rca_major_hold_1009_OB_COMPACTION_PROGRESS_data_global_broadcast_scn
[2025-03-31 13:08:47.203] [DEBUG] - add record: tenant_id:1009 OB_COMPACTION_PROGRESS_data_last_scn save on ./obdiag_rca/obdiag_major_hold_202503311
[2025-03-31 13:08:47.203] [DEBUG] 30846/rca_major_hold_1009_OB_COMPACTION_PROGRESS_data_last_scn
[2025-03-31 13:08:47.206] [DEBUG] - add record: step:5 get OB_COMPACTION_SUGGESTIONS
[2025-03-31 13:08:47.210] [DEBUG] - add record: tenant_id:1009 OB_COMPACTION_PROGRESS_data_last_scn save on
[2025-03-31 13:08:47.210] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/rca_major_hold_1009_OB_COMPACTION_SUGGESTIONS_info
[2025-03-31 13:08:47.210] [DEBUG] - step:6 get dmesg by dmesg -T
[2025-03-31 13:08:47.210] [WARNING] MajorHoldScene execute 6 get dmesg exception: [Errno 17] File exists: './obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log'
[2025-03-31 13:08:47.210] [DEBUG] - add suggest: send the ./obdiag_rca/obdiag_major_hold_20250331130846 to the oceanbase community
[2025-03-31 13:08:47.210] [DEBUG] - tenant_id is 1
[2025-03-31 13:08:47.210] [DEBUG] - add record: tenant_id is 1
[2025-03-31 13:08:47.210] [DEBUG] - add record: step:1 get CDB_OB_MAJOR_COMPACTION data
[2025-03-31 13:08:47.214] [DEBUG] - add record: on CDB_OB_MAJOR_COMPACTION where status='COMPACTING';result:[{'TENANT_ID': 1, 'FROZEN_SCN':
[2025-03-31 13:08:47.214] [DEBUG] 1743098403563081000, 'FROZEN_TIME': datetime.datetime(2025, 3, 28, 2, 0, 3, 563081),
[2025-03-31 13:08:47.214] [DEBUG] 'GLOBAL_BROADCAST_SCN': 1743098403563081000, 'LAST_SCN': 1743012001814465000, 'LAST_FINISH_TIME':
[2025-03-31 13:08:47.214] [DEBUG] datetime.datetime(2025, 3, 27, 2, 7, 5, 769136), 'START_TIME': datetime.datetime(2025, 3, 28, 2, 0,
[2025-03-31 13:08:47.214] [DEBUG] 3, 805730), 'STATUS': 'COMPACTING', 'IS_ERROR': 'NO', 'IS_SUSPENDED': 'NO', 'INFO': ''}]
[2025-03-31 13:08:47.214] [DEBUG] - add record: step:2&3 get __all_virtual_compaction_diagnose_info and check the diagnose type
[2025-03-31 13:08:47.250] [DEBUG] - add record: on __all_virtual_compaction_diagnose_info;result:[{'svr_ip': '192.168.1.62', 'svr_port': 2882,
[2025-03-31 13:08:47.251] [DEBUG] 'tenant_id': 1, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 215, 'status': 'FAILED',
[2025-03-31 13:08:47.251] [DEBUG] 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 4, 875838), 'diagnose_info': 'error_no=-4016,las
[2025-03-31 13:08:47.251] [DEBUG] t_error_time=1743397726341044,error_trace=YB420A0B093E-00063125A22842DA-0-0,location=ob_partition_me
[2025-03-31 13:08:47.251] [DEBUG] rger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=215;compaction_scn=1
[2025-03-31 13:08:47.251] [DEBUG] 743098403563081000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.65',
[2025-03-31 13:08:47.251] [DEBUG] 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE', 'ls_id': 1003, 'tablet_id': 200391,
[2025-03-31 13:08:47.251] [DEBUG] 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 3, 32, 430045),
[2025-03-31 13:08:47.251] [DEBUG] 'diagnose_info': 'error_no=-4034,last_error_time=1743397683450165,error_trace=YB420A0B0941-00063125A
[2025-03-31 13:08:47.251] [DEBUG] 55389B6-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls
[2025-03-31 13:08:47.251] [DEBUG] _id=1003;tablet_id=200391;compaction_scn=1743098400401915000;exec_mode="EXEC_MODE_LOCAL",concurrent_
[2025-03-31 13:08:47.251] [DEBUG] cnt=1"'}, {'svr_ip': '192.168.1.65', 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE',
[2025-03-31 13:08:47.251] [DEBUG] 'ls_id': 1003, 'tablet_id': 200391, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3,
[2025-03-31 13:08:47.251] [DEBUG] 28, 2, 3, 32, 430045), 'diagnose_info': 'error_no=-4034,last_error_time=1743397683450165,error_trace
[2025-03-31 13:08:47.251] [DEBUG] =YB420A0B0941-00063125A55389B6-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJO
[2025-03-31 13:08:47.251] [DEBUG] R_MERGE/MEDIUM_MERGE;ls_id=1003;tablet_id=200391;compaction_scn=1743098400401915000;exec_mode="EXEC_
[2025-03-31 13:08:47.251] [DEBUG] MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.65', 'svr_port': 2882, 'tenant_id': 1002,
[2025-03-31 13:08:47.251] [DEBUG] 'type': 'MINOR_MERGE', 'ls_id': 1003, 'tablet_id': 200391, 'status': 'FAILED', 'create_time':
[2025-03-31 13:08:47.251] [DEBUG] datetime.datetime(2025, 3, 27, 23, 55, 3, 228902), 'diagnose_info': 'error_no=-4034,last_error_time=
[2025-03-31 13:08:47.251] [DEBUG] 1743098464251531,error_trace=YB420A0B0941-00063125A5536BF5-0-0,location=ob_partition_merger.cpp:124(
[2025-03-31 13:08:47.251] [DEBUG] prepare_merge),warning="MINOR_EXECUTE;ls_id=1003;tablet_id=200391;compaction_scn=1743091213157973000
[2025-03-31 13:08:47.251] [DEBUG] ;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port': 2882,
[2025-03-31 13:08:47.251] [DEBUG] 'tenant_id': 1001, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 331, 'status': 'FAILED',
[2025-03-31 13:08:47.251] [DEBUG] 'create_time': datetime.datetime(2025, 3, 28, 2, 5, 9, 445088), 'diagnose_info': 'error_no=-4016,las
[2025-03-31 13:08:47.251] [DEBUG] t_error_time=1743397691267177,error_trace=YB420A0B093F-00063125A6486AF8-0-0,location=ob_partition_me
[2025-03-31 13:08:47.251] [DEBUG] rger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=331;compaction_scn=1
[2025-03-31 13:08:47.251] [DEBUG] 743098400293280000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63',
[2025-03-31 13:08:47.251] [DEBUG] 'svr_port': 2882, 'tenant_id': 1001, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 121, 'status':
[2025-03-31 13:08:47.251] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 6, 5, 301487), 'diagnose_info': 'error_no
[2025-03-31 13:08:47.251] [DEBUG] =-4016,last_error_time=1743397691440167,error_trace=YB420A0B093F-00063125A6486AF9-0-0,location=ob_pa
[2025-03-31 13:08:47.251] [DEBUG] rtition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=121;compac
[2025-03-31 13:08:47.252] [DEBUG] tion_scn=1743098400293280000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip':
[2025-03-31 13:08:47.252] [DEBUG] '192.168.1.63', 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE', 'ls_id': 1001,
[2025-03-31 13:08:47.252] [DEBUG] 'tablet_id': 201413, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 25,
[2025-03-31 13:08:47.252] [DEBUG] 890814), 'diagnose_info': 'error_no=-4034,last_error_time=1743397687211333,error_trace=YB420A0B093F-
[2025-03-31 13:08:47.252] [DEBUG] 00063125A6296AB2-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM
[2025-03-31 13:08:47.252] [DEBUG] _MERGE;ls_id=1001;tablet_id=201413;compaction_scn=1743098400401915000;exec_mode="EXEC_MODE_LOCAL",co
[2025-03-31 13:08:47.252] [DEBUG] ncurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port': 2882, 'tenant_id': 1002, 'type':
[2025-03-31 13:08:47.252] [DEBUG] 'MEDIUM_MERGE', 'ls_id': 1001, 'tablet_id': 201413, 'status': 'FAILED', 'create_time':
[2025-03-31 13:08:47.252] [DEBUG] datetime.datetime(2025, 3, 28, 2, 4, 25, 890814), 'diagnose_info': 'error_no=-4034,last_error_time=1
[2025-03-31 13:08:47.252] [DEBUG] 743397687211333,error_trace=YB420A0B093F-00063125A6296AB2-0-0,location=ob_partition_merger.cpp:124(p
[2025-03-31 13:08:47.252] [DEBUG] repare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1001;tablet_id=201413;compaction_scn=174309840
[2025-03-31 13:08:47.252] [DEBUG] 0401915000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port':
[2025-03-31 13:08:47.252] [DEBUG] 2882, 'tenant_id': 1002, 'type': 'MINOR_MERGE', 'ls_id': 1001, 'tablet_id': 201413, 'status':
[2025-03-31 13:08:47.252] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 1, 44, 17, 825107), 'diagnose_info': 'error_
[2025-03-31 13:08:47.252] [DEBUG] no=-4034,last_error_time=1743098494302962,error_trace=YB420A0B093F-00063125A6284486-0-0,location=ob_
[2025-03-31 13:08:47.252] [DEBUG] partition_merger.cpp:124(prepare_merge),warning="MINOR_EXECUTE;ls_id=1001;tablet_id=201413;compactio
[2025-03-31 13:08:47.252] [DEBUG] n_scn=1743097455815495000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63',
[2025-03-31 13:08:47.252] [DEBUG] 'svr_port': 2882, 'tenant_id': 1009, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 323, 'status':
[2025-03-31 13:08:47.252] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 10, 781639), 'diagnose_info': 'error_n
[2025-03-31 13:08:47.252] [DEBUG] o=-4016,last_error_time=1743397613587786,error_trace=YB420A0B093F-00063125BBE8516E-0-0,location=ob_p
[2025-03-31 13:08:47.252] [DEBUG] artition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=323;compa
[2025-03-31 13:08:47.252] [DEBUG] ction_scn=1743098404774537000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}]
[2025-03-31 13:08:47.252] [DEBUG] - add record: diagnose_info:error_no=-4016,last_error_time=1743397726341044,error_trace=YB420A0B093E-00063125A2284
[2025-03-31 13:08:47.252] [DEBUG] 2DA-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=
[2025-03-31 13:08:47.252] [DEBUG] 1;tablet_id=215;compaction_scn=1743098403563081000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"
[2025-03-31 13:08:47.252] [DEBUG] - add record: diagnose_info type is error_no
[2025-03-31 13:08:47.258] [DEBUG] - add record: #2&3 on __all_virtual_compaction_diagnose_info get data failed
[2025-03-31 13:08:47.258] [WARNING] #2&3 MajorHoldScene execute exception: tuple index out of range
[2025-03-31 13:08:47.258] [DEBUG] - add record: step:4 get GV$OB_COMPACTION_PROGRESS whit tenant_id:1
[2025-03-31 13:08:47.261] [DEBUG] - add record: global_broadcast_scn is 1743098403563081000
[2025-03-31 13:08:47.263] [DEBUG] - add record: last_scn is 1743012001814465000
[2025-03-31 13:08:47.267] [DEBUG] - add record: tenant_id:1 OB_COMPACTION_PROGRESS_data_global_broadcast_scn save on ./obdiag_rca/obdiag_major_hold_
[2025-03-31 13:08:47.267] [DEBUG] 20250331130846/rca_major_hold_1_OB_COMPACTION_PROGRESS_data_global_broadcast_scn
[2025-03-31 13:08:47.271] [DEBUG] - add record: tenant_id:1 OB_COMPACTION_PROGRESS_data_last_scn save on
[2025-03-31 13:08:47.271] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/rca_major_hold_1_OB_COMPACTION_PROGRESS_data_last_scn
[2025-03-31 13:08:47.274] [DEBUG] - add record: step:5 get OB_COMPACTION_SUGGESTIONS
[2025-03-31 13:08:47.277] [DEBUG] - add record: tenant_id:1 OB_COMPACTION_PROGRESS_data_last_scn save on
[2025-03-31 13:08:47.278] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/rca_major_hold_1_OB_COMPACTION_SUGGESTIONS_info
[2025-03-31 13:08:47.278] [DEBUG] - step:6 get dmesg by dmesg -T
[2025-03-31 13:08:47.278] [WARNING] MajorHoldScene execute 6 get dmesg exception: [Errno 17] File exists: './obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log'
[2025-03-31 13:08:47.278] [DEBUG] - add suggest: send the ./obdiag_rca/obdiag_major_hold_20250331130846 to the oceanbase community
[2025-03-31 13:08:47.278] [DEBUG] - tenant_id is 1002
[2025-03-31 13:08:47.278] [DEBUG] - add record: tenant_id is 1002
[2025-03-31 13:08:47.278] [DEBUG] - add record: step:1 get CDB_OB_MAJOR_COMPACTION data
[2025-03-31 13:08:47.281] [DEBUG] - add record: on CDB_OB_MAJOR_COMPACTION where status='COMPACTING';result:[{'TENANT_ID': 1002, 'FROZEN_SCN':
[2025-03-31 13:08:47.281] [DEBUG] 1743357600990044000, 'FROZEN_TIME': datetime.datetime(2025, 3, 31, 2, 0, 0, 990044),
[2025-03-31 13:08:47.281] [DEBUG] 'GLOBAL_BROADCAST_SCN': 1743357600990044000, 'LAST_SCN': 1743271204136735000, 'LAST_FINISH_TIME':
[2025-03-31 13:08:47.281] [DEBUG] datetime.datetime(2025, 3, 31, 0, 29, 55, 685366), 'START_TIME': datetime.datetime(2025, 3, 31, 2,
[2025-03-31 13:08:47.281] [DEBUG] 0, 1, 260204), 'STATUS': 'COMPACTING', 'IS_ERROR': 'NO', 'IS_SUSPENDED': 'NO', 'INFO': ''}]
[2025-03-31 13:08:47.281] [DEBUG] - add record: step:2&3 get __all_virtual_compaction_diagnose_info and check the diagnose type
[2025-03-31 13:08:47.290] [DEBUG] - add record: on __all_virtual_compaction_diagnose_info;result:[{'svr_ip': '192.168.1.62', 'svr_port': 2882,
[2025-03-31 13:08:47.290] [DEBUG] 'tenant_id': 1, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 215, 'status': 'FAILED',
[2025-03-31 13:08:47.290] [DEBUG] 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 4, 875838), 'diagnose_info': 'error_no=-4016,las
[2025-03-31 13:08:47.290] [DEBUG] t_error_time=1743397726341044,error_trace=YB420A0B093E-00063125A22842DA-0-0,location=ob_partition_me
[2025-03-31 13:08:47.290] [DEBUG] rger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=215;compaction_scn=1
[2025-03-31 13:08:47.290] [DEBUG] 743098403563081000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.65',
[2025-03-31 13:08:47.290] [DEBUG] 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE', 'ls_id': 1003, 'tablet_id': 200391,
[2025-03-31 13:08:47.290] [DEBUG] 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 3, 32, 430045),
[2025-03-31 13:08:47.290] [DEBUG] 'diagnose_info': 'error_no=-4034,last_error_time=1743397683450165,error_trace=YB420A0B0941-00063125A
[2025-03-31 13:08:47.290] [DEBUG] 55389B6-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls
[2025-03-31 13:08:47.290] [DEBUG] _id=1003;tablet_id=200391;compaction_scn=1743098400401915000;exec_mode="EXEC_MODE_LOCAL",concurrent_
[2025-03-31 13:08:47.290] [DEBUG] cnt=1"'}, {'svr_ip': '192.168.1.65', 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE',
[2025-03-31 13:08:47.290] [DEBUG] 'ls_id': 1003, 'tablet_id': 200391, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3,
[2025-03-31 13:08:47.290] [DEBUG] 28, 2, 3, 32, 430045), 'diagnose_info': 'error_no=-4034,last_error_time=1743397683450165,error_trace
[2025-03-31 13:08:47.290] [DEBUG] =YB420A0B0941-00063125A55389B6-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJO
[2025-03-31 13:08:47.291] [DEBUG] R_MERGE/MEDIUM_MERGE;ls_id=1003;tablet_id=200391;compaction_scn=1743098400401915000;exec_mode="EXEC_
[2025-03-31 13:08:47.291] [DEBUG] MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.65', 'svr_port': 2882, 'tenant_id': 1002,
[2025-03-31 13:08:47.291] [DEBUG] 'type': 'MINOR_MERGE', 'ls_id': 1003, 'tablet_id': 200391, 'status': 'FAILED', 'create_time':
[2025-03-31 13:08:47.291] [DEBUG] datetime.datetime(2025, 3, 27, 23, 55, 3, 228902), 'diagnose_info': 'error_no=-4034,last_error_time=
[2025-03-31 13:08:47.291] [DEBUG] 1743098464251531,error_trace=YB420A0B0941-00063125A5536BF5-0-0,location=ob_partition_merger.cpp:124(
[2025-03-31 13:08:47.291] [DEBUG] prepare_merge),warning="MINOR_EXECUTE;ls_id=1003;tablet_id=200391;compaction_scn=1743091213157973000
[2025-03-31 13:08:47.291] [DEBUG] ;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port': 2882,
[2025-03-31 13:08:47.291] [DEBUG] 'tenant_id': 1001, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 331, 'status': 'FAILED',
[2025-03-31 13:08:47.291] [DEBUG] 'create_time': datetime.datetime(2025, 3, 28, 2, 5, 9, 445088), 'diagnose_info': 'error_no=-4016,las
[2025-03-31 13:08:47.291] [DEBUG] t_error_time=1743397691267177,error_trace=YB420A0B093F-00063125A6486AF8-0-0,location=ob_partition_me
[2025-03-31 13:08:47.291] [DEBUG] rger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=331;compaction_scn=1
[2025-03-31 13:08:47.291] [DEBUG] 743098400293280000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63',
[2025-03-31 13:08:47.291] [DEBUG] 'svr_port': 2882, 'tenant_id': 1001, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 121, 'status':
[2025-03-31 13:08:47.291] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 6, 5, 301487), 'diagnose_info': 'error_no
[2025-03-31 13:08:47.291] [DEBUG] =-4016,last_error_time=1743397691440167,error_trace=YB420A0B093F-00063125A6486AF9-0-0,location=ob_pa
[2025-03-31 13:08:47.291] [DEBUG] rtition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=121;compac
[2025-03-31 13:08:47.291] [DEBUG] tion_scn=1743098400293280000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip':
[2025-03-31 13:08:47.291] [DEBUG] '192.168.1.63', 'svr_port': 2882, 'tenant_id': 1002, 'type': 'MEDIUM_MERGE', 'ls_id': 1001,
[2025-03-31 13:08:47.291] [DEBUG] 'tablet_id': 201413, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 25,
[2025-03-31 13:08:47.291] [DEBUG] 890814), 'diagnose_info': 'error_no=-4034,last_error_time=1743397687211333,error_trace=YB420A0B093F-
[2025-03-31 13:08:47.291] [DEBUG] 00063125A6296AB2-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM
[2025-03-31 13:08:47.291] [DEBUG] _MERGE;ls_id=1001;tablet_id=201413;compaction_scn=1743098400401915000;exec_mode="EXEC_MODE_LOCAL",co
[2025-03-31 13:08:47.291] [DEBUG] ncurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port': 2882, 'tenant_id': 1002, 'type':
[2025-03-31 13:08:47.291] [DEBUG] 'MEDIUM_MERGE', 'ls_id': 1001, 'tablet_id': 201413, 'status': 'FAILED', 'create_time':
[2025-03-31 13:08:47.291] [DEBUG] datetime.datetime(2025, 3, 28, 2, 4, 25, 890814), 'diagnose_info': 'error_no=-4034,last_error_time=1
[2025-03-31 13:08:47.291] [DEBUG] 743397687211333,error_trace=YB420A0B093F-00063125A6296AB2-0-0,location=ob_partition_merger.cpp:124(p
[2025-03-31 13:08:47.291] [DEBUG] repare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1001;tablet_id=201413;compaction_scn=174309840
[2025-03-31 13:08:47.291] [DEBUG] 0401915000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63', 'svr_port':
[2025-03-31 13:08:47.291] [DEBUG] 2882, 'tenant_id': 1002, 'type': 'MINOR_MERGE', 'ls_id': 1001, 'tablet_id': 201413, 'status':
[2025-03-31 13:08:47.291] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 1, 44, 17, 825107), 'diagnose_info': 'error_
[2025-03-31 13:08:47.291] [DEBUG] no=-4034,last_error_time=1743098494302962,error_trace=YB420A0B093F-00063125A6284486-0-0,location=ob_
[2025-03-31 13:08:47.291] [DEBUG] partition_merger.cpp:124(prepare_merge),warning="MINOR_EXECUTE;ls_id=1001;tablet_id=201413;compactio
[2025-03-31 13:08:47.291] [DEBUG] n_scn=1743097455815495000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}, {'svr_ip': '192.168.1.63',
[2025-03-31 13:08:47.291] [DEBUG] 'svr_port': 2882, 'tenant_id': 1009, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 323, 'status':
[2025-03-31 13:08:47.291] [DEBUG] 'FAILED', 'create_time': datetime.datetime(2025, 3, 28, 2, 4, 10, 781639), 'diagnose_info': 'error_n
[2025-03-31 13:08:47.291] [DEBUG] o=-4016,last_error_time=1743397613587786,error_trace=YB420A0B093F-00063125BBE8516E-0-0,location=ob_p
[2025-03-31 13:08:47.291] [DEBUG] artition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=1;tablet_id=323;compa
[2025-03-31 13:08:47.292] [DEBUG] ction_scn=1743098404774537000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"'}]
[2025-03-31 13:08:47.292] [DEBUG] - add record: diagnose_info:error_no=-4016,last_error_time=1743397726341044,error_trace=YB420A0B093E-00063125A2284
[2025-03-31 13:08:47.292] [DEBUG] 2DA-0-0,location=ob_partition_merger.cpp:124(prepare_merge),warning="MAJOR_MERGE/MEDIUM_MERGE;ls_id=
[2025-03-31 13:08:47.292] [DEBUG] 1;tablet_id=215;compaction_scn=1743098403563081000;exec_mode="EXEC_MODE_LOCAL",concurrent_cnt=1"
[2025-03-31 13:08:47.292] [DEBUG] - add record: diagnose_info type is error_no
[2025-03-31 13:08:47.298] [DEBUG] - add record: #2&3 on __all_virtual_compaction_diagnose_info get data failed
[2025-03-31 13:08:47.298] [WARNING] #2&3 MajorHoldScene execute exception: tuple index out of range
[2025-03-31 13:08:47.298] [DEBUG] - add record: step:4 get GV$OB_COMPACTION_PROGRESS whit tenant_id:1002
[2025-03-31 13:08:47.303] [DEBUG] - add record: global_broadcast_scn is 1743357600990044000
[2025-03-31 13:08:47.308] [DEBUG] - add record: last_scn is 1743271204136735000
[2025-03-31 13:08:47.312] [DEBUG] - add record: tenant_id:1002 OB_COMPACTION_PROGRESS_data_global_broadcast_scn save on ./obdiag_rca/obdiag_major_ho
[2025-03-31 13:08:47.312] [DEBUG] ld_20250331130846/rca_major_hold_1002_OB_COMPACTION_PROGRESS_data_global_broadcast_scn
[2025-03-31 13:08:47.316] [DEBUG] - add record: tenant_id:1002 OB_COMPACTION_PROGRESS_data_last_scn save on ./obdiag_rca/obdiag_major_hold_202503311
[2025-03-31 13:08:47.316] [DEBUG] 30846/rca_major_hold_1002_OB_COMPACTION_PROGRESS_data_last_scn
[2025-03-31 13:08:47.320] [DEBUG] - add record: step:5 get OB_COMPACTION_SUGGESTIONS
[2025-03-31 13:08:47.323] [DEBUG] - add record: tenant_id:1002 OB_COMPACTION_PROGRESS_data_last_scn save on
[2025-03-31 13:08:47.323] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250331130846/rca_major_hold_1002_OB_COMPACTION_SUGGESTIONS_info
[2025-03-31 13:08:47.323] [DEBUG] - step:6 get dmesg by dmesg -T
[2025-03-31 13:08:47.323] [WARNING] MajorHoldScene execute 6 get dmesg exception: [Errno 17] File exists: './obdiag_rca/obdiag_major_hold_20250331130846/dmesg_log'
[2025-03-31 13:08:47.323] [DEBUG] - add suggest: send the ./obdiag_rca/obdiag_major_hold_20250331130846 to the oceanbase community
[2025-03-31 13:08:47.323] [DEBUG] - save record to ./obdiag_rca/obdiag_major_hold_20250331130846/record.table
[2025-03-31 13:08:47.372] [INFO] rca finished. For more details, the result on './obdiag_rca/obdiag_major_hold_20250331130846' 
[2025-03-31 13:08:47.372] [INFO] You can get the suggest by 'cat ./obdiag_rca/obdiag_major_hold_20250331130846/record.table'
[2025-03-31 13:08:47.372] [INFO] Trace ID: 3911f2de-0dee-11f0-9a32-00163eb23b0c
[2025-03-31 13:08:47.372] [INFO] If you want to view detailed obdiag logs, please run: obdiag display-trace 3911f2de-0dee-11f0-9a32-00163eb23b0c