[2025-01-15 10:04:45] INFO __init__.py:1611 parameters from cmd: host="172.16.0.65", port=2881, user="root", password="******", timeout="0", module="set(['post_check', 'health_check', 'begin_upgrade', 'end_rolling_upgrade', 'begin_rolling_upgrade', 'end_upgrade', 'tenant_upgrade', 'special_action'])", log-file="upgrade_post.log"
[2025-01-15 10:04:45] INFO __init__.py:1611 select distinct(substring_index(build_version, '_', 1)) from __all_server
[2025-01-15 10:04:45] INFO __init__.py:1611 check server version success
[2025-01-15 10:04:45] INFO __init__.py:1611 ================begin to run health check action ===============
[2025-01-15 10:04:45] INFO __init__.py:1611 zone is empty, check all servers in cluster
[2025-01-15 10:04:45] INFO __init__.py:1611 succeed to execute query: select distinct tenant_id from oceanbase.__all_tenant order by tenant_id desc, rowcount = 5
[2025-01-15 10:04:45] INFO __init__.py:1611 use default timeout caculated by tenants, timeout(s):600, tenant_count:5, timeout_per_tenant(s):10, min_timeout(s):600
[2025-01-15 10:04:45] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2025-01-15 10:04:45] INFO __init__.py:1611 check value is 0 success
[2025-01-15 10:04:45] INFO __init__.py:1611 succeed to execute query: select distinct tenant_id from oceanbase.__all_tenant order by tenant_id desc, rowcount = 5
[2025-01-15 10:04:45] INFO __init__.py:1611 use default timeout caculated by tenants, timeout(s):600, tenant_count:5, timeout_per_tenant(s):10, min_timeout(s):600
[2025-01-15 10:04:45] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.GV$OB_LOG_STAT where in_sync = 'NO', rowcount = 1
[2025-01-15 10:04:45] INFO __init__.py:1611 check value is 0 success
[2025-01-15 10:04:45] INFO __init__.py:1611 check paxos replica success
[2025-01-15 10:04:45] INFO __init__.py:1611 succeed to execute query: select distinct tenant_id from oceanbase.__all_tenant order by tenant_id desc, rowcount = 5
[2025-01-15 10:04:45] INFO __init__.py:1611 use default timeout caculated by tenants, timeout(s):600, tenant_count:5, timeout_per_tenant(s):30, min_timeout(s):600
[2025-01-15 10:04:45] INFO __init__.py:1611 succeed to execute query: select if (a.cnt = b.cnt, 1, 0) as passed from (select count(*) as cnt from oceanbase.__all_virtual_server_schema_info where refreshed_schema_version > 1 and refreshed_schema_version % 8 = 0) as a join (select count(*) as cnt from oceanbase.__all_server join oceanbase.__all_tenant) as b, rowcount = 1
[2025-01-15 10:04:45] INFO __init__.py:1611 check value is 1 success
[2025-01-15 10:04:45] INFO __init__.py:1611 skip check server version by cluster
[2025-01-15 10:04:45] INFO __init__.py:1611 ================succeed to run health check action ===============
[2025-01-15 10:04:45] INFO __init__.py:1611 ================begin to run end rolling upgrade action ===============
[2025-01-15 10:04:45] INFO __init__.py:1611 select * from oceanbase.GV$OB_PARAMETERS
           where name = 'enable_upgrade_mode' and value = 'False'
[2025-01-15 10:04:45] INFO __init__.py:1611 select * from oceanbase.GV$OB_PARAMETERS
           where name = '_upgrade_stage' and value = 'POSTUPGRADE'
[2025-01-15 10:04:45] INFO __init__.py:1611 alter system end rolling upgrade
[2025-01-15 10:04:55] ERROR __init__.py:1585 run error
Traceback (most recent call last):
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/do_upgrade_post.py", line 80, in do_upgrade
    actions.do_end_rolling_upgrade(cur, timeout)
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/actions.py", line 337, in do_end_rolling_upgrade
    cur.execute(action_sql)
  File "/usr/obd/lib/executer/executer27/site-packages/mysql/connector/cursor_cext.py", line 266, in execute
    raw_as_string=self._raw_as_string)
  File "/usr/obd/lib/executer/executer27/site-packages/mysql/connector/connection_cext.py", line 492, in cmd_query
    sqlstate=exc.sqlstate)
DatabaseError: 4012 (HY000): Timeout
[2025-01-15 10:04:55] INFO __init__.py:1611 ==================================================================================
[2025-01-15 10:04:55] INFO __init__.py:1611 ============================== STATISTICS BEGIN ==================================
[2025-01-15 10:04:55] INFO __init__.py:1611 ==================================================================================
[2025-01-15 10:04:55] INFO __init__.py:1611 succeed run sql(except sql of special actions): 



[2025-01-15 10:04:55] INFO __init__.py:1611 commited sql(except sql of special actions): 



[2025-01-15 10:04:55] INFO __init__.py:1611 ==================================================================================
[2025-01-15 10:04:55] INFO __init__.py:1611 =============================== STATISTICS END ===================================
[2025-01-15 10:04:55] INFO __init__.py:1611 ==================================================================================
[2025-01-15 10:04:55] ERROR __init__.py:1585 connection error
Traceback (most recent call last):
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/do_upgrade_post.py", line 80, in do_upgrade
    actions.do_end_rolling_upgrade(cur, timeout)
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/actions.py", line 337, in do_end_rolling_upgrade
    cur.execute(action_sql)
  File "/usr/obd/lib/executer/executer27/site-packages/mysql/connector/cursor_cext.py", line 266, in execute
    raw_as_string=self._raw_as_string)
  File "/usr/obd/lib/executer/executer27/site-packages/mysql/connector/connection_cext.py", line 492, in cmd_query
    sqlstate=exc.sqlstate)
DatabaseError: 4012 (HY000): Timeout
[2025-01-15 10:04:55] ERROR __init__.py:1585 mysql connctor error
Traceback (most recent call last):
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/do_upgrade_post.py", line 157, in do_upgrade_by_argv
    do_upgrade(host, port, user, password, timeout, module_set, upgrade_params)
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/do_upgrade_post.py", line 80, in do_upgrade
    actions.do_end_rolling_upgrade(cur, timeout)
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/actions.py", line 337, in do_end_rolling_upgrade
    cur.execute(action_sql)
  File "/usr/obd/lib/executer/executer27/site-packages/mysql/connector/cursor_cext.py", line 266, in execute
    raw_as_string=self._raw_as_string)
  File "/usr/obd/lib/executer/executer27/site-packages/mysql/connector/connection_cext.py", line 492, in cmd_query
    sqlstate=exc.sqlstate)
DatabaseError: 4012 (HY000): Timeout
[2025-01-15 10:04:55] ERROR __init__.py:1585 run error, maybe you can reference rollback_sql_post.txt to rollback it
Traceback (most recent call last):
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/do_upgrade_post.py", line 157, in do_upgrade_by_argv
    do_upgrade(host, port, user, password, timeout, module_set, upgrade_params)
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/do_upgrade_post.py", line 80, in do_upgrade
    actions.do_end_rolling_upgrade(cur, timeout)
  File "/tmp/172.16.0.65:2882/48b61655aaa13e9b01b722928d1979c76b41937e/upgrade_post_extract_files_2025_01_15_10_04_45_053728_anDaHioJ/actions.py", line 337, in do_end_rolling_upgrade
    cur.execute(action_sql)
  File "/usr/obd/lib/executer/executer27/site-packages/mysql/connector/cursor_cext.py", line 266, in execute
    raw_as_string=self._raw_as_string)
  File "/usr/obd/lib/executer/executer27/site-packages/mysql/connector/connection_cext.py", line 492, in cmd_query
    sqlstate=exc.sqlstate)
DatabaseError: 4012 (HY000): Timeout