[2023-08-11 14:37:30] INFO __init__.py:1611 parameters from cmd: host="192.168.122.74", port=2881, user="root", password="LancooECPsys", timeout="0", module="set(['post_check', 'health_check', 'begin_upgrade', 'end_rolling_upgrade', 'begin_rolling_upgrade', 'end_upgrade', 'tenant_upgrade', 'special_action'])", log-file="upgrade_post.log"
[2023-08-11 14:37:30] INFO __init__.py:1611 select distinct(substring_index(build_version, '_', 1)) from __all_server
[2023-08-11 14:37:30] INFO __init__.py:1611 check server version success
[2023-08-11 14:37:30] INFO __init__.py:1611 ================begin to run health check action ===============
[2023-08-11 14:37:30] INFO __init__.py:1611 zone is empty, check all servers in cluster
[2023-08-11 14:37:30] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:37:30] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:37:40] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:37:40] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:37:50] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:37:50] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:38:00] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:38:00] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:38:10] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:38:10] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:38:20] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:38:20] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:38:30] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:38:30] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:38:40] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:38:40] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:38:50] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:38:50] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:39:00] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:39:00] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:39:10] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:39:10] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:39:20] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:39:20] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:39:30] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:39:30] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:39:40] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:39:40] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:39:50] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:39:50] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:40:00] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:40:00] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:40:10] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:40:10] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:40:20] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:40:20] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:40:30] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:40:30] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:40:40] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:40:40] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:40:50] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:40:50] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:41:00] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:41:00] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:41:10] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:41:10] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:41:20] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:41:20] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:41:30] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:41:30] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:41:40] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:41:40] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:41:50] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:41:50] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:42:00] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:42:00] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:42:10] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:42:10] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:42:20] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:42:20] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:42:30] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:42:30] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:42:40] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:42:40] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:42:50] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:42:50] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:43:00] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:43:00] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:43:10] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:43:10] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:43:20] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:43:20] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:43:30] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:43:30] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:43:40] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:43:40] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:43:50] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:43:50] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:44:00] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:44:00] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:44:10] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:44:10] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:44:20] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:44:20] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:44:30] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:44:30] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:44:40] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:44:40] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:44:50] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:44:50] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:45:00] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:45:00] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:45:10] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:45:10] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:45:20] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:45:20] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:45:31] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:45:31] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:45:41] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:45:41] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:45:51] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:45:51] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:46:01] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:46:01] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:46:11] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:46:11] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:46:21] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:46:21] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:46:31] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:46:31] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:46:41] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:46:41] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:46:51] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:46:51] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:47:01] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:47:01] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:47:11] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:47:11] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:47:21] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:47:21] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:47:31] INFO __init__.py:1611 succeed to execute query: select count(*) from oceanbase.__all_server where (start_service_time <= 0 or status='inactive'), rowcount = 1
[2023-08-11 14:47:31] INFO __init__.py:1611 value is 1, expected value is 0, not matched
[2023-08-11 14:47:31] ERROR __init__.py:1585 run error
Traceback (most recent call last):
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/upgrade_health_checker.py", line 373, in do_check
    check_observer_status(query_cur, zone, timeout)
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/upgrade_health_checker.py", line 318, in check_observer_status
    check_until_timeout(query_cur, sql, 0, timeout)
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/upgrade_health_checker.py", line 354, in check_until_timeout
    logging.warn("""check {0} job timeout""".format(job_name))
NameError: global name 'job_name' is not defined
[2023-08-11 14:47:31] ERROR __init__.py:1585 normal error
Traceback (most recent call last):
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/upgrade_health_checker.py", line 380, in do_check
    raise e
NameError: global name 'job_name' is not defined
[2023-08-11 14:47:31] ERROR __init__.py:1585 run error
Traceback (most recent call last):
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/do_upgrade_post.py", line 68, in do_upgrade
    upgrade_health_checker.do_check(my_host, my_port, my_user, my_passwd, upgrade_params, timeout)
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/upgrade_health_checker.py", line 389, in do_check
    raise e
NameError: global name 'job_name' is not defined
[2023-08-11 14:47:31] INFO __init__.py:1611 ==================================================================================
[2023-08-11 14:47:31] INFO __init__.py:1611 ============================== STATISTICS BEGIN ==================================
[2023-08-11 14:47:31] INFO __init__.py:1611 ==================================================================================
[2023-08-11 14:47:31] INFO __init__.py:1611 succeed run sql(except sql of special actions): 



[2023-08-11 14:47:31] INFO __init__.py:1611 commited sql(except sql of special actions): 



[2023-08-11 14:47:31] INFO __init__.py:1611 ==================================================================================
[2023-08-11 14:47:31] INFO __init__.py:1611 =============================== STATISTICS END ===================================
[2023-08-11 14:47:31] INFO __init__.py:1611 ==================================================================================
[2023-08-11 14:47:31] ERROR __init__.py:1585 normal error
Traceback (most recent call last):
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/do_upgrade_post.py", line 105, in do_upgrade
    raise e
NameError: global name 'job_name' is not defined
[2023-08-11 14:47:31] ERROR __init__.py:1585 normal error
Traceback (most recent call last):
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/do_upgrade_post.py", line 151, in do_upgrade_by_argv
    do_upgrade(host, port, user, password, timeout, module_set, upgrade_params)
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/do_upgrade_post.py", line 118, in do_upgrade
    raise e
NameError: global name 'job_name' is not defined
[2023-08-11 14:47:31] ERROR __init__.py:1585 run error, maybe you can reference rollback_sql_post.txt to rollback it
Traceback (most recent call last):
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/do_upgrade_post.py", line 151, in do_upgrade_by_argv
    do_upgrade(host, port, user, password, timeout, module_set, upgrade_params)
  File "/tmp/192.168.122.74:2882/5cc69b0ce9944adb57e36deb449bb70786d3ddc5/upgrade_post_extract_files_2023_08_11_14_37_30_188989_uettFB6O/do_upgrade_post.py", line 118, in do_upgrade
    raise e
NameError: global name 'job_name' is not defined