[root@localhost ~]# obdiag display-trace e80cdf26-1422-11f0-9c75-00163eb20c6f
obdiag version: 3.2.0
[2025-04-08 10:41:00.312] [DEBUG] - cmd: obdiag rca run
[2025-04-08 10:41:00.323] [DEBUG] - opts: {'inner_config': None, 'scene': 'major_hold', 'store_dir': './obdiag_rca/', 'env': None, 'report_type': 'table', 'c': '/root/.obdiag/config.yml', 'config': None}
[2025-04-08 10:41:00.323] [INFO] obdiag version: 3.2.0
[2025-04-08 10:41:00.323] [DEBUG] - mkdir /opt/oceanbase-diagnostic-tool/conf/inner_config.yml
[2025-04-08 10:41:00.334] [DEBUG] - mkdir /root/.obdiag/config.yml
[2025-04-08 10:41:01.036] [DEBUG] - connect databse ...
[2025-04-08 10:41:01.430] [DEBUG] - connect databse ...
[2025-04-08 10:41:01.431] [DEBUG] - RCAHandler.init store dir: ./obdiag_rca/
[2025-04-08 10:41:01.431] [DEBUG] - rca result save_path is :./obdiag_rca/
[2025-04-08 10:41:01.432] [DEBUG] - get observer version, by sql
[2025-04-08 10:41:01.432] [DEBUG] - start get_observer_version_by_sql . input: 192.168.1.61:2881
[2025-04-08 10:41:01.433] [DEBUG] - connect databse ...
[2025-04-08 10:41:01.434] [DEBUG] - get_observer_version_by_sql ob_version_info is ('5.7.25-OceanBase_CE-v4.2.1.8',)
[2025-04-08 10:41:01.434] [DEBUG] - RCAHandler.init get observer version: 4.2.1.8
[2025-04-08 10:41:01.434] [DEBUG] - RCAHandler.init get observer version: 4.2.1.8
[2025-04-08 10:41:01.632] [DEBUG] - Execute Shell command on server 192.168.1.61:/root/obproxy/bin/obproxy --version
[2025-04-08 10:41:01.707] [DEBUG] - get obproxy version, run cmd = [/root/obproxy/bin/obproxy --version] 
[2025-04-08 10:41:01.707] [DEBUG] - RCAHandler.init get obproxy version: 4.3.3.0
[2025-04-08 10:41:01.749] [DEBUG] - RCAHandler init.cluster:sgkyocp, init.nodes:[{'ip': '192.168.1.61', 'ssh_username': 'root', 'ssh_port': 22, 'home_path': '/root/oceanbase', 'data_dir': '/data/1', 'redo_dir': '/data/log1', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': <src.common.ssh_client.ssh.SshClient object at 0x145f85a6c0d0>}], init.obproxy_nodes:[{'ip': '192.168.1.61', 'ssh_username': 'root', 'ssh_port': 22, 'home_path': '/root/obproxy', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': None, 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBPROXY', 'ssher': <src.common.ssh_client.ssh.SshClient object at 0x145f85a763a0>}], init.store_dir:./obdiag_rca/
[2025-04-08 10:41:01.749] [DEBUG] - major_hold store_dir:./obdiag_rca/obdiag_major_hold_20250408104101
[2025-04-08 10:41:01.749] [DEBUG] - get observer version, by sql
[2025-04-08 10:41:01.749] [DEBUG] - start get_observer_version_by_sql . input: 192.168.1.61:2881
[2025-04-08 10:41:01.750] [DEBUG] - connect databse ...
[2025-04-08 10:41:01.751] [DEBUG] - get_observer_version_by_sql ob_version_info is ('5.7.25-OceanBase_CE-v4.2.1.8',)
[2025-04-08 10:41:01.751] [DEBUG] - major_hold init success
[2025-04-08 10:41:01.751] [DEBUG] - add record: check major task is error or not
[2025-04-08 10:41:01.760] [DEBUG] - add record: CDB_OB_MAJOR_COMPACTION is not exist IS_ERROR='YES'
[2025-04-08 10:41:01.760] [DEBUG] - add record: check on CDB_OB_MAJOR_COMPACTION IS_ERROR is 'YES'.  sql:select * from
[2025-04-08 10:41:01.760] [DEBUG] oceanbase.CDB_OB_MAJOR_COMPACTION where IS_ERROR="YES";
[2025-04-08 10:41:01.808] [DEBUG] - add record: __all_virtual_compaction_diagnose_info have status='FAILED',the tenant is ['1004', '1004', '1004']
[2025-04-08 10:41:01.811] [DEBUG] - add record: merge tasks that have not ended beyond the expected time,the tenant_id is ['1004']
[2025-04-08 10:41:01.812] [DEBUG] - merge tasks that have not ended beyond the expected time,the tenant_id is ['1004']
[2025-04-08 10:41:01.812] [DEBUG] - add suggest: some tenants need execute MajorHoldScene. :['1004']
[2025-04-08 10:41:01.812] [DEBUG] - On CDB_OB_MAJOR_COMPACTION
[2025-04-08 10:41:01.812] [DEBUG] - tenant_id is 1004
[2025-04-08 10:41:01.812] [DEBUG] - add record: tenant_id is 1004
[2025-04-08 10:41:01.812] [DEBUG] - add record: step:1 get CDB_OB_MAJOR_COMPACTION data
[2025-04-08 10:41:01.815] [DEBUG] - add record: on CDB_OB_MAJOR_COMPACTION where status='COMPACTING';result:[{'TENANT_ID': 1004, 'FROZEN_SCN':
[2025-04-08 10:41:01.815] [DEBUG] 1744048801036449184, 'FROZEN_TIME': datetime.datetime(2025, 4, 8, 2, 0, 1, 36449),
[2025-04-08 10:41:01.815] [DEBUG] 'GLOBAL_BROADCAST_SCN': 1744048801036449184, 'LAST_SCN': 1, 'LAST_FINISH_TIME':
[2025-04-08 10:41:01.815] [DEBUG] datetime.datetime(1970, 1, 1, 8, 0), 'START_TIME': datetime.datetime(2025, 4, 8, 2, 0, 1, 265508),
[2025-04-08 10:41:01.815] [DEBUG] 'STATUS': 'COMPACTING', 'IS_ERROR': 'NO', 'IS_SUSPENDED': 'NO', 'INFO': ''}]
[2025-04-08 10:41:01.815] [DEBUG] - add record: step:2&3 get __all_virtual_compaction_diagnose_info and check the diagnose type
[2025-04-08 10:41:01.865] [DEBUG] - add record: on __all_virtual_compaction_diagnose_info;result:[{'svr_ip': '192.168.1.61', 'svr_port': 2882,
[2025-04-08 10:41:01.865] [DEBUG] 'tenant_id': 1004, 'type': 'MEDIUM_MERGE', 'ls_id': 1, 'tablet_id': 182, 'status': 'FAILED',
[2025-04-08 10:41:01.865] [DEBUG] 'create_time': datetime.datetime(2025, 4, 8, 2, 1, 38, 322007), 'diagnose_info': 'error_no=-4034,las
[2025-04-08 10:41:01.865] [DEBUG] t_error_time=1744080039840836,error_trace=YB420A0B093D-0006322C65074D4D-0-0,warning="info="DAG_MAJOR
[2025-04-08 10:41:01.865] [DEBUG] _MERGE";ls_id=1;tablet_id=182;compaction_scn=1744048801036449184;extra_info="merge_type="MEDIUM_MERG
[2025-04-08 10:41:01.865] [DEBUG] E"";"'}, {'svr_ip': '192.168.1.61', 'svr_port': 2882, 'tenant_id': 1004, 'type': 'MINOR_MERGE',
[2025-04-08 10:41:01.865] [DEBUG] 'ls_id': 1001, 'tablet_id': 1152921504606847785, 'status': 'FAILED', 'create_time':
[2025-04-08 10:41:01.865] [DEBUG] datetime.datetime(2025, 4, 8, 7, 47, 26, 213582), 'diagnose_info': 'error_no=-4034,last_error_time=1
[2025-04-08 10:41:01.865] [DEBUG] 744079969807357,error_trace=YB420A0B093D-0006322C65074D4C-0-0,warning="info="DAG_MINOR_MERGE";ls_id=
[2025-04-08 10:41:01.865] [DEBUG] 1001;tablet_id=1152921504606847785;compaction_scn=0;extra_info="merge_type="MINOR_MERGE"";"'},
[2025-04-08 10:41:01.865] [DEBUG] {'svr_ip': '192.168.1.61', 'svr_port': 2882, 'tenant_id': 1004, 'type': 'MEDIUM_MERGE', 'ls_id': 1,
[2025-04-08 10:41:01.865] [DEBUG] 'tablet_id': 100002, 'status': 'FAILED', 'create_time': datetime.datetime(2025, 4, 8, 2, 2, 0,
[2025-04-08 10:41:01.866] [DEBUG] 680330), 'diagnose_info': 'error_no=-4034,last_error_time=1744080039844490,error_trace=YB420A0B093D-
[2025-04-08 10:41:01.866] [DEBUG] 0006322C65074D4E-0-0,warning="info="DAG_MAJOR_MERGE";ls_id=1;tablet_id=100002;compaction_scn=1744048
[2025-04-08 10:41:01.866] [DEBUG] 801036449184;extra_info="merge_type="MEDIUM_MERGE"";"'}]
[2025-04-08 10:41:01.866] [DEBUG] - add record: diagnose_info:error_no=-4034,last_error_time=1744080039840836,error_trace=YB420A0B093D-0006322C65074
[2025-04-08 10:41:01.866] [DEBUG] D4D-0-0,warning="info="DAG_MAJOR_MERGE";ls_id=1;tablet_id=182;compaction_scn=1744048801036449184;ext
[2025-04-08 10:41:01.866] [DEBUG] ra_info="merge_type="MEDIUM_MERGE"";"
[2025-04-08 10:41:01.866] [DEBUG] - add record: diagnose_info type is error_no
[2025-04-08 10:41:01.881] [DEBUG] - add record: #2&3 on __all_virtual_compaction_diagnose_info get data failed
[2025-04-08 10:41:01.881] [WARNING] #2&3 MajorHoldScene execute exception: tuple index out of range
[2025-04-08 10:41:01.881] [DEBUG] - add record: step:4 get GV$OB_COMPACTION_PROGRESS whit tenant_id:1004
[2025-04-08 10:41:01.883] [DEBUG] - add record: global_broadcast_scn is 1744048801036449184
[2025-04-08 10:41:01.887] [DEBUG] - add record: last_scn is 1
[2025-04-08 10:41:01.890] [DEBUG] - add record: tenant_id:1004 OB_COMPACTION_PROGRESS_data_global_broadcast_scn save on ./obdiag_rca/obdiag_major_ho
[2025-04-08 10:41:01.890] [DEBUG] ld_20250408104101/rca_major_hold_1004_OB_COMPACTION_PROGRESS_data_global_broadcast_scn
[2025-04-08 10:41:01.891] [DEBUG] - add record: tenant_id:1004 OB_COMPACTION_PROGRESS_data_last_scn save on ./obdiag_rca/obdiag_major_hold_202504081
[2025-04-08 10:41:01.891] [DEBUG] 04101/rca_major_hold_1004_OB_COMPACTION_PROGRESS_data_last_scn
[2025-04-08 10:41:01.894] [DEBUG] - add record: step:5 get OB_COMPACTION_SUGGESTIONS
[2025-04-08 10:41:01.900] [DEBUG] - add record: tenant_id:1004 OB_COMPACTION_PROGRESS_data_last_scn save on
[2025-04-08 10:41:01.901] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250408104101/rca_major_hold_1004_OB_COMPACTION_SUGGESTIONS_info
[2025-04-08 10:41:01.901] [DEBUG] - step:6 get dmesg by dmesg -T
[2025-04-08 10:41:01.901] [DEBUG] - Execute Shell command on server 192.168.1.61:dmesg -T > /tmp/dmesg_remote_10_11_9_61.log
[2025-04-08 10:41:01.924] [DEBUG] - download file: /tmp/dmesg_remote_10_11_9_61.log to ./obdiag_rca/obdiag_major_hold_20250408104101/dmesg_log/dmesg_remote_10_11_9_61.log
[2025-04-08 10:41:01.983] [DEBUG] - Download 192.168.1.61:/tmp/dmesg_remote_10_11_9_61.log
[2025-04-08 10:41:01.986] [DEBUG] - add record: download /tmp/dmesg_remote_10_11_9_61.log to
[2025-04-08 10:41:01.986] [DEBUG] ./obdiag_rca/obdiag_major_hold_20250408104101/dmesg_log/dmesg_remote_10_11_9_61.log
[2025-04-08 10:41:01.986] [DEBUG] - add suggest: send the ./obdiag_rca/obdiag_major_hold_20250408104101 to the oceanbase community
[2025-04-08 10:41:01.986] [DEBUG] - save record to ./obdiag_rca/obdiag_major_hold_20250408104101/record.table
[2025-04-08 10:41:01.994] [INFO] rca finished. For more details, the result on './obdiag_rca/obdiag_major_hold_20250408104101' 
[2025-04-08 10:41:01.994] [INFO] You can get the suggest by 'cat ./obdiag_rca/obdiag_major_hold_20250408104101/record.table'
[2025-04-08 10:41:01.994] [INFO] Trace ID: e80cdf26-1422-11f0-9c75-00163eb20c6f
[2025-04-08 10:41:01.994] [INFO] If you want to view detailed obdiag logs, please run: obdiag display-trace e80cdf26-1422-11f0-9c75-00163eb20c6f