[2025-03-25 13:48:09] INFO do_upgrade_post.py:156 parameters from cmd: host="192.168.1.62", port=2881, user="root", password="******", timeout="0", module="set(['post_check', 'health_check', 'begin_upgrade', 'end_rolling_upgrade', 'begin_rolling_upgrade', 'end_upgrade', 'tenant_upgrade', 'special_action'])", log-file="upgrade_post.log" [2025-03-25 13:48:09] INFO actions.py:233 select distinct(substring_index(build_version, '_', 1)) from __all_server [2025-03-25 13:48:09] INFO actions.py:239 check server version success [2025-03-25 13:48:09] INFO do_upgrade_post.py:73 ================begin to run health check action =============== [2025-03-25 13:48:09] INFO upgrade_health_checker.py:295 zone is empty, check all servers in cluster [2025-03-25 13:48:09] INFO upgrade_health_checker.py:53 succeed to execute query: select distinct tenant_id from oceanbase.__all_tenant order by tenant_id desc, rowcount = 11 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:318 use default timeout caculated by tenants, timeout(s):600, tenant_count:11, timeout_per_tenant(s):10, min_timeout(s):600 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:53 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:382 check value is 0 success [2025-03-25 13:48:09] INFO upgrade_health_checker.py:53 succeed to execute query: select distinct tenant_id from oceanbase.__all_tenant order by tenant_id desc, rowcount = 11 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:318 use default timeout caculated by tenants, timeout(s):600, tenant_count:11, timeout_per_tenant(s):10, min_timeout(s):600 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:53 succeed to execute query: select count(*) from oceanbase.GV$OB_LOG_STAT where in_sync = 'NO', rowcount = 1 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:382 check value is 0 success [2025-03-25 13:48:09] INFO upgrade_health_checker.py:343 check paxos replica success [2025-03-25 13:48:09] INFO upgrade_health_checker.py:53 succeed to execute query: select distinct tenant_id from oceanbase.__all_tenant order by tenant_id desc, rowcount = 11 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:318 use default timeout caculated by tenants, timeout(s):600, tenant_count:11, timeout_per_tenant(s):30, min_timeout(s):600 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:53 succeed to execute query: select if (a.cnt = b.cnt, 1, 0) as passed from (select count(*) as cnt from oceanbase.__all_virtual_server_schema_info where refreshed_schema_version > 1 and refreshed_schema_version % 8 = 0) as a join (select count(*) as cnt from oceanbase.__all_server join oceanbase.__all_tenant) as b, rowcount = 1 [2025-03-25 13:48:09] INFO upgrade_health_checker.py:382 check value is 1 success [2025-03-25 13:48:09] INFO upgrade_health_checker.py:326 skip check server version by cluster [2025-03-25 13:48:09] INFO do_upgrade_post.py:75 ================succeed to run health check action =============== [2025-03-25 13:48:09] INFO do_upgrade_post.py:78 ================begin to run end rolling upgrade action =============== [2025-03-25 13:48:09] INFO actions.py:245 select * from oceanbase.GV$OB_PARAMETERS where name = 'enable_upgrade_mode' and value = 'False' [2025-03-25 13:48:09] INFO actions.py:245 select * from oceanbase.GV$OB_PARAMETERS where name = '_upgrade_stage' and value = 'POSTUPGRADE' [2025-03-25 13:48:09] INFO actions.py:245 select * from oceanbase.GV$OB_PARAMETERS where name = 'min_observer_version' and value = '4.3.5.1' [2025-03-25 13:48:09] INFO actions.py:132 set @@session.ob_query_timeout = 60000000 [2025-03-25 13:48:09] INFO actions.py:275 select count(*) as cnt from oceanbase.GV$OB_PARAMETERS where name = 'min_observer_version' and value != '4.3.5.1' [2025-03-25 13:48:09] INFO actions.py:282 min_observer_version is sync, value is 4.3.5.1 [2025-03-25 13:48:09] INFO actions.py:132 set @@session.ob_query_timeout = 10000000 [2025-03-25 13:48:09] INFO actions.py:132 set @@session.ob_query_timeout = 60000000 [2025-03-25 13:48:09] INFO actions.py:275 select count(*) as cnt from oceanbase.GV$OB_PARAMETERS where name = '_upgrade_stage' and value != 'POSTUPGRADE' [2025-03-25 13:48:09] INFO actions.py:282 _upgrade_stage is sync, value is POSTUPGRADE [2025-03-25 13:48:09] INFO actions.py:132 set @@session.ob_query_timeout = 10000000 [2025-03-25 13:48:09] INFO do_upgrade_post.py:83 ================succeed to run end rolling upgrade action =============== [2025-03-25 13:48:09] INFO do_upgrade_post.py:86 ================begin to run tenant upgrade action =============== [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:46 select count(*) from oceanbase.__all_table where table_name = '__all_virtual_core_table' [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:46 select tenant_id from oceanbase.__all_tenant [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:46 select count(*) from __all_virtual_core_table where column_name in ('target_data_version', 'current_data_version', 'upgrade_begin_data_version') and column_value = 17180067073 [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:95 target_data_version/current_data_version/upgrade_begin_data_version not match with 4.3.5.1, tenant_cnt:11, result_cnt:11 [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:231 start to run upgrade job, job_name:UPGRADE_ALL [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:46 select job_status from oceanbase.__all_rootservice_job where job_type = 'UPGRADE_ALL' order by job_id desc limit 1 [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:158 execute upgrade job failed, should run again [2025-03-25 13:48:09] INFO actions.py:183 select value from oceanbase.__all_sys_parameter where name = 'enable_ddl' [2025-03-25 13:48:09] INFO actions.py:132 set @@session.ob_query_timeout = 60000000 [2025-03-25 13:48:09] INFO actions.py:275 select count(*) as cnt from oceanbase.GV$OB_PARAMETERS where name = 'enable_ddl' and value != 'True' [2025-03-25 13:48:09] INFO actions.py:282 enable_ddl is sync, value is True [2025-03-25 13:48:09] INFO actions.py:132 set @@session.ob_query_timeout = 10000000 [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:46 select job_id from oceanbase.__all_rootservice_job order by job_id desc limit 1 [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:130 get max_used_job_id:7 [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:243 alter system run upgrade job 'UPGRADE_ALL' [2025-03-25 13:48:09] INFO actions.py:421 succeed to execute query: select distinct tenant_id from oceanbase.__all_tenant order by tenant_id desc, rowcount = 11 [2025-03-25 13:48:09] INFO actions.py:146 use default timeout caculated by tenants, timeout(s):3600, tenant_count:11, timeout_per_tenant(s):100, min_timeout(s):3600 [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:46 select job_status, rs_svr_ip, rs_svr_port, gmt_create from oceanbase.__all_rootservice_job where job_type = 'UPGRADE_ALL' and job_id > 7 order by job_id desc limit 1 [2025-03-25 13:48:09] INFO tenant_upgrade_action.py:180 upgrade job not created yet [2025-03-25 13:48:19] INFO tenant_upgrade_action.py:46 select job_status, rs_svr_ip, rs_svr_port, gmt_create from oceanbase.__all_rootservice_job where job_type = 'UPGRADE_ALL' and job_id > 7 order by job_id desc limit 1 [2025-03-25 13:48:19] WARNING tenant_upgrade_action.py:214 execute upgrade job failed [2025-03-25 13:48:19] WARNING tenant_upgrade_action.py:226 failed to check upgrade job result [2025-03-25 13:48:19] WARNING tenant_upgrade_action.py:251 run upgrade job failed, :UPGRADE_ALL [2025-03-25 13:48:19] ERROR do_upgrade_post.py:110 run error Traceback (most recent call last): File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/do_upgrade_post.py", line 88, in do_upgrade tenant_upgrade_action.do_upgrade(conn, cur, timeout, my_user, my_passwd) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 21, in do_upgrade run_upgrade_job(conn, cur, "UPGRADE_ALL", timeout) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 246, in run_upgrade_job check_upgrade_job_result(cur, job_name, timeout, max_used_job_id) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 215, in check_upgrade_job_result raise MyError("execute upgrade job failed") MyError: 'execute upgrade job failed' [2025-03-25 13:48:19] INFO do_upgrade_post.py:49 ================================================================================== [2025-03-25 13:48:19] INFO do_upgrade_post.py:50 ============================== STATISTICS BEGIN ================================== [2025-03-25 13:48:19] INFO do_upgrade_post.py:51 ================================================================================== [2025-03-25 13:48:19] INFO do_upgrade_post.py:52 succeed run sql(except sql of special actions): [2025-03-25 13:48:19] INFO do_upgrade_post.py:53 commited sql(except sql of special actions): [2025-03-25 13:48:19] INFO do_upgrade_post.py:54 ================================================================================== [2025-03-25 13:48:19] INFO do_upgrade_post.py:55 =============================== STATISTICS END =================================== [2025-03-25 13:48:19] INFO do_upgrade_post.py:56 ================================================================================== [2025-03-25 13:48:19] ERROR do_upgrade_post.py:123 normal error Traceback (most recent call last): File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/do_upgrade_post.py", line 88, in do_upgrade tenant_upgrade_action.do_upgrade(conn, cur, timeout, my_user, my_passwd) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 21, in do_upgrade run_upgrade_job(conn, cur, "UPGRADE_ALL", timeout) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 246, in run_upgrade_job check_upgrade_job_result(cur, job_name, timeout, max_used_job_id) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 215, in check_upgrade_job_result raise MyError("execute upgrade job failed") MyError: 'execute upgrade job failed' [2025-03-25 13:48:19] ERROR do_upgrade_post.py:163 normal error Traceback (most recent call last): File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/do_upgrade_post.py", line 157, in do_upgrade_by_argv do_upgrade(host, port, user, password, timeout, module_set, upgrade_params) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/do_upgrade_post.py", line 88, in do_upgrade tenant_upgrade_action.do_upgrade(conn, cur, timeout, my_user, my_passwd) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 21, in do_upgrade run_upgrade_job(conn, cur, "UPGRADE_ALL", timeout) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 246, in run_upgrade_job check_upgrade_job_result(cur, job_name, timeout, max_used_job_id) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 215, in check_upgrade_job_result raise MyError("execute upgrade job failed") MyError: 'execute upgrade job failed' [2025-03-25 13:48:19] ERROR do_upgrade_post.py:164 run error, maybe you can reference rollback_sql_post.txt to rollback it Traceback (most recent call last): File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/do_upgrade_post.py", line 157, in do_upgrade_by_argv do_upgrade(host, port, user, password, timeout, module_set, upgrade_params) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/do_upgrade_post.py", line 88, in do_upgrade tenant_upgrade_action.do_upgrade(conn, cur, timeout, my_user, my_passwd) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 21, in do_upgrade run_upgrade_job(conn, cur, "UPGRADE_ALL", timeout) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 246, in run_upgrade_job check_upgrade_job_result(cur, job_name, timeout, max_used_job_id) File "/tmp/192.168.1.62:2882_192.168.1.63:2882_192.168.1.64:2882_192.168.1.65:2882/3a4f23adb7973d6d1d6969bcd9ae108f8c564b66/upgrade_post_extract_files_2025_03_25_13_48_09_240896_lC22K256/tenant_upgrade_action.py", line 215, in check_upgrade_job_result raise MyError("execute upgrade job failed") MyError: 'execute upgrade job failed'