[root@localhost ~]# obdiag display-trace edd9d5d0-09ff-11f0-852a-00163eb23b0c [2025-03-26 13:05:26.028] [DEBUG] - cmd: obdiag rca run [2025-03-26 13:05:26.028] [DEBUG] - opts: {'inner_config': None, 'scene': 'major_hold', 'store_dir': './obdiag_rca/', 'env': None, 'report_type': 'table', 'c': '/root/.obdiag/config.yml', 'config': None} [2025-03-26 13:05:26.029] [DEBUG] - mkdir /usr/local/oceanbase-diagnostic-tool/conf/inner_config.yml [2025-03-26 13:05:26.069] [DEBUG] - mkdir /root/.obdiag/config.yml [2025-03-26 13:05:26.339] [DEBUG] - connect databse ... [2025-03-26 13:05:26.597] [DEBUG] - connect databse ... [2025-03-26 13:05:26.598] [DEBUG] - RCAHandler.init store dir: ./obdiag_rca/ [2025-03-26 13:05:26.599] [DEBUG] - rca result save_path is :./obdiag_rca/ [2025-03-26 13:05:26.599] [DEBUG] - get observer version, by sql [2025-03-26 13:05:26.599] [DEBUG] - start get_observer_version_by_sql . input: 192.168.1.62:2881 [2025-03-26 13:05:26.600] [DEBUG] - connect databse ... [2025-03-26 13:05:26.601] [DEBUG] - get_observer_version_by_sql ob_version_info is ('5.7.25-OceanBase_CE-v4.3.5.1',) [2025-03-26 13:05:26.602] [DEBUG] - RCAHandler.init get observer version: 4.3.5.1 [2025-03-26 13:05:26.602] [DEBUG] - RCAHandler.init get observer version: 4.3.5.1 [2025-03-26 13:05:26.694] [DEBUG] - RCAHandler init.cluster:sgkydb, init.nodes:[{'ip': '192.168.1.62', 'ssh_username': 'root', 'ssh_port': '22', 'home_path': '/root/observer', 'data_dir': '/root/observer/store', 'redo_dir': '/root/observer/store', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': }, {'ip': '192.168.1.63', 'ssh_username': 'root', 'ssh_port': '22', 'home_path': '/root/observer', 'data_dir': '/root/observer/store', 'redo_dir': '/root/observer/store', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': }, {'ip': '192.168.1.64', 'ssh_username': 'root', 'ssh_port': '22', 'home_path': '/root/observer', 'data_dir': '/root/observer/store', 'redo_dir': '/root/observer/store', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': }, {'ip': '192.168.1.65', 'ssh_username': 'root', 'ssh_port': '22', 'home_path': '/root/observer', 'data_dir': '/root/observer/store', 'redo_dir': '/root/observer/store', 'ssh_key_file': '', 'ssh_type': 'remote', 'container_name': '', 'namespace': '', 'pod_name': '', 'kubernetes_config_file': '', 'host_type': 'OBSERVER', 'ssher': }], init.obproxy_nodes:[], init.store_dir:./obdiag_rca/ [2025-03-26 13:05:26.694] [DEBUG] - major_hold store_dir:./obdiag_rca/obdiag_major_hold_20250326130526 [2025-03-26 13:05:26.694] [DEBUG] - get observer version, by sql [2025-03-26 13:05:26.694] [DEBUG] - start get_observer_version_by_sql . input: 192.168.1.62:2881 [2025-03-26 13:05:26.696] [DEBUG] - connect databse ... [2025-03-26 13:05:26.697] [DEBUG] - get_observer_version_by_sql ob_version_info is ('5.7.25-OceanBase_CE-v4.3.5.1',) [2025-03-26 13:05:26.697] [DEBUG] - major_hold init success [2025-03-26 13:05:26.697] [DEBUG] - add record: check major task is error or not [2025-03-26 13:05:26.748] [DEBUG] - add record: CDB_OB_MAJOR_COMPACTION is not exist IS_ERROR='YES' [2025-03-26 13:05:26.748] [DEBUG] - add record: check on CDB_OB_MAJOR_COMPACTION IS_ERROR is 'YES'. sql:select * from [2025-03-26 13:05:26.748] [DEBUG] oceanbase.CDB_OB_MAJOR_COMPACTION where IS_ERROR="YES"; [2025-03-26 13:05:26.756] [DEBUG] - add record: __all_virtual_compaction_diagnose_info have status='FAILED',the tenant is ['1006'] [2025-03-26 13:05:26.766] [DEBUG] - add record: merge tasks that have not ended beyond the expected time,the tenant_id is ['1006'] [2025-03-26 13:05:26.766] [DEBUG] - merge tasks that have not ended beyond the expected time,the tenant_id is ['1006'] [2025-03-26 13:05:26.766] [DEBUG] - add suggest: some tenants need execute MajorHoldScene. :['1006'] [2025-03-26 13:05:26.766] [DEBUG] - On CDB_OB_MAJOR_COMPACTION [2025-03-26 13:05:26.766] [DEBUG] - tenant_id is 1006 [2025-03-26 13:05:26.814] [WARNING] #2&3 MajorHoldScene execute exception: tuple index out of range [2025-03-26 13:05:26.834] [ERROR] MajorHoldScene execute 4 exception: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use near 'LAST_SCN': 1}';' at line 1") [2025-03-26 13:05:26.835] [DEBUG] - Traceback (most recent call last): [2025-03-26 13:05:26.835] [DEBUG] File "/root/.obdiag/rca/major_hold.py", line 158, in execute [2025-03-26 13:05:26.835] [DEBUG] OB_COMPACTION_PROGRESS_data_last_scn = self.ob_connector.execute_sql_return_cursor_dictionary(sql).fetchall() [2025-03-26 13:05:26.835] [DEBUG] File "src/common/ob_connector.py", line 131, in execute_sql_return_cursor_dictionary [2025-03-26 13:05:26.836] [DEBUG] File "pymysql/cursors.py", line 153, in execute [2025-03-26 13:05:26.836] [DEBUG] File "pymysql/cursors.py", line 322, in _query [2025-03-26 13:05:26.836] [DEBUG] File "pymysql/connections.py", line 563, in query [2025-03-26 13:05:26.836] [DEBUG] File "pymysql/connections.py", line 825, in _read_query_result [2025-03-26 13:05:26.836] [DEBUG] File "pymysql/connections.py", line 1199, in read [2025-03-26 13:05:26.836] [DEBUG] File "pymysql/connections.py", line 775, in _read_packet [2025-03-26 13:05:26.836] [DEBUG] File "pymysql/protocol.py", line 219, in raise_for_error [2025-03-26 13:05:26.836] [DEBUG] File "pymysql/err.py", line 150, in raise_mysql_exception [2025-03-26 13:05:26.836] [DEBUG] pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use near 'LAST_SCN': 1}';' at line 1") [2025-03-26 13:05:26.836] [DEBUG] [2025-03-26 13:05:26.836] [DEBUG] During handling of the above exception, another exception occurred: [2025-03-26 13:05:26.836] [DEBUG] [2025-03-26 13:05:26.836] [DEBUG] Traceback (most recent call last): [2025-03-26 13:05:26.836] [DEBUG] File "src/handler/rca/rca_handler.py", line 186, in __execute [2025-03-26 13:05:26.836] [DEBUG] File "/root/.obdiag/rca/major_hold.py", line 187, in execute [2025-03-26 13:05:26.836] [DEBUG] raise RCAExecuteException("MajorHoldScene execute 4 exception: {0}".format(e)) [2025-03-26 13:05:26.836] [DEBUG] src.handler.rca.rca_exception.RCAExecuteException: MajorHoldScene execute 4 exception: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use near 'LAST_SCN': 1}';' at line 1") [2025-03-26 13:05:26.836] [DEBUG] [2025-03-26 13:05:26.836] [ERROR] rca_scene.execute err: MajorHoldScene execute 4 exception: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use near 'LAST_SCN': 1}';' at line 1") [2025-03-26 13:05:26.836] [INFO] Trace ID: edd9d5d0-09ff-11f0-852a-00163eb23b0c [2025-03-26 13:05:26.836] [INFO] If you want to view detailed obdiag logs, please run: obdiag display-trace edd9d5d0-09ff-11f0-852a-00163eb23b0c