- open /root/.obd/version for a+ - cmd: ['4d07e48e-2a5d-11f0-8c63-5254009d7d10'] - opts: {} - mkdir /root/.obd/lock/ - set lock mode to NO_LOCK(0) - Get log by trace_id [2025-05-06 09:34:26.119] [DEBUG] - cmd: ['cluster_102166'] [2025-05-06 09:34:26.120] [DEBUG] - opts: {'tenant_name': 'tenant', 'ignore_standby': True} [2025-05-06 09:34:26.121] [DEBUG] - mkdir /root/.obd/lock/ [2025-05-06 09:34:26.121] [DEBUG] - unknown lock mode [2025-05-06 09:34:26.122] [DEBUG] - try to get share lock /root/.obd/lock/global [2025-05-06 09:34:26.123] [DEBUG] - share lock `/root/.obd/lock/global`, count 1 [2025-05-06 09:34:26.123] [DEBUG] - Get Deploy by name [2025-05-06 09:34:26.124] [DEBUG] - mkdir /root/.obd/cluster/ [2025-05-06 09:34:26.124] [DEBUG] - mkdir /root/.obd/config_parser/ [2025-05-06 09:34:26.126] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_cluster_102166 [2025-05-06 09:34:26.126] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_cluster_102166`, count 1 [2025-05-06 09:34:26.147] [DEBUG] - Deploy status judge [2025-05-06 09:34:26.148] [DEBUG] - Get deploy config [2025-05-06 09:34:26.216] [INFO] Get local repositories [2025-05-06 09:34:26.219] [DEBUG] - mkdir /root/.obd/repository [2025-05-06 09:34:26.220] [DEBUG] - Get local repository oceanbase-ce-4.3.5.1-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:26.221] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo [2025-05-06 09:34:26.221] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 1 [2025-05-06 09:34:26.229] [DEBUG] - Get local repository obproxy-ce-4.3.3.0-850604eabec8b341c3a121240f0ef87b56cdae8a [2025-05-06 09:34:26.230] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 2 [2025-05-06 09:34:26.237] [DEBUG] - Get local repository obagent-4.2.2-d8172a2bf3811219777124490558bf800dffef87 [2025-05-06 09:34:26.238] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 3 [2025-05-06 09:34:26.245] [DEBUG] - Searching param plugin for components ... [2025-05-06 09:34:26.246] [DEBUG] - Search param plugin for oceanbase-ce [2025-05-06 09:34:26.247] [DEBUG] - mkdir /root/.obd/plugins [2025-05-06 09:34:26.250] [DEBUG] - Found for oceanbase-ce-param-4.3.3.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:26.251] [DEBUG] - Applying oceanbase-ce-param-4.3.3.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:28.974] [DEBUG] - Search param plugin for obproxy-ce [2025-05-06 09:34:28.976] [DEBUG] - Found for obproxy-ce-param-4.3.0 for obproxy-ce-4.3.3.0 [2025-05-06 09:34:28.976] [DEBUG] - Applying obproxy-ce-param-4.3.0 for obproxy-ce-4.3.3.0-5.el8-850604eabec8b341c3a121240f0ef87b56cdae8a [2025-05-06 09:34:29.443] [DEBUG] - Search param plugin for obagent [2025-05-06 09:34:29.461] [DEBUG] - Found for obagent-param-4.2.2 for obagent-4.2.2 [2025-05-06 09:34:29.462] [DEBUG] - Applying obagent-param-4.2.2 for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2025-05-06 09:34:29.856] [INFO] Open ssh connection [2025-05-06 09:34:29.875] [DEBUG] - host: 172.36.102.166, port: 10022, user: root, password: ****** [2025-05-06 09:34:32.063] [DEBUG] - Searching drop_tenant template for components ... [2025-05-06 09:34:32.064] [DEBUG] - mkdir /root/.obd/workflows [2025-05-06 09:34:32.068] [DEBUG] - Call workflow oceanbase-ce-py_script_workflow_drop_tenant-3.1.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:32.070] [DEBUG] - mkdir /root/.obd/mirror [2025-05-06 09:34:32.071] [DEBUG] - mkdir /root/.obd/mirror/remote [2025-05-06 09:34:32.072] [DEBUG] - mkdir /root/.obd/mirror/local [2025-05-06 09:34:32.073] [DEBUG] - mkdir /root/.obd/optimize/ [2025-05-06 09:34:32.074] [DEBUG] - mkdir /root/.obd/tool/ [2025-05-06 09:34:32.076] [DEBUG] - import drop_tenant [2025-05-06 09:34:32.080] [DEBUG] - add drop_tenant ref count to 1 [2025-05-06 09:34:32.081] [DEBUG] - sub drop_tenant ref count to 0 [2025-05-06 09:34:32.081] [DEBUG] - export drop_tenant [2025-05-06 09:34:32.082] [DEBUG] - plugin oceanbase-ce-py_script_workflow_drop_tenant-3.1.0 result: True [2025-05-06 09:34:32.082] [DEBUG] - Found for oceanbase-ce-py_script_workflow_drop_tenant-3.1.0 for oceanbase-ce-3.1.0 [2025-05-06 09:34:32.083] [DEBUG] - Searching drop_tenant template for components ... [2025-05-06 09:34:33.115] [DEBUG] - No such drop_tenant template for obproxy-ce-4.3.3.0 [2025-05-06 09:34:33.116] [DEBUG] - Searching drop_tenant template for components ... [2025-05-06 09:34:33.119] [DEBUG] - No such drop_tenant template for obagent-4.2.2 [2025-05-06 09:34:33.120] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 4 [2025-05-06 09:34:33.128] [DEBUG] - Searching status plugin for components ... [2025-05-06 09:34:33.129] [DEBUG] - Searching status plugin for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.131] [DEBUG] - Found for oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:33.132] [DEBUG] - Call plugin oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.132] [DEBUG] - import status [2025-05-06 09:34:33.286] [DEBUG] - add status ref count to 1 [2025-05-06 09:34:33.288] [DEBUG] -- root@172.36.102.166 execute: cat /opt/oceanbase/run/observer.pid [2025-05-06 09:34:33.403] [DEBUG] -- exited code 0 [2025-05-06 09:34:33.404] [DEBUG] -- root@172.36.102.166 execute: ls /proc/241 [2025-05-06 09:34:33.528] [DEBUG] -- exited code 0 [2025-05-06 09:34:33.529] [DEBUG] - sub status ref count to 0 [2025-05-06 09:34:33.529] [DEBUG] - export status [2025-05-06 09:34:33.530] [DEBUG] - plugin oceanbase-ce-py_script_status-3.1.0 result: True [2025-05-06 09:34:33.531] [DEBUG] - Searching status_check plugin for components ... [2025-05-06 09:34:33.531] [DEBUG] - Searching status_check plugin for general-4.3.5.1--None [2025-05-06 09:34:33.533] [DEBUG] - Found for general-py_script_status_check-0.1 for general-4.3.5.1 [2025-05-06 09:34:33.534] [DEBUG] - Call plugin general-py_script_status_check-0.1 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.534] [DEBUG] - import status_check [2025-05-06 09:34:33.536] [DEBUG] - add status_check ref count to 1 [2025-05-06 09:34:33.537] [DEBUG] - sub status_check ref count to 0 [2025-05-06 09:34:33.538] [DEBUG] - export status_check [2025-05-06 09:34:33.538] [DEBUG] - plugin general-py_script_status_check-0.1 result: True [2025-05-06 09:34:33.538] [DEBUG] - Searching connect plugin for components ... [2025-05-06 09:34:33.539] [DEBUG] - Searching connect plugin for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.541] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:33.541] [DEBUG] - Call plugin oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.542] [DEBUG] - import connect [2025-05-06 09:34:33.545] [DEBUG] - add connect ref count to 1 [2025-05-06 09:34:33.546] [INFO] Connect to observer [2025-05-06 09:34:33.549] [DEBUG] -- connect 172.36.102.166 -P2881 -uroot -p****** [2025-05-06 09:34:33.557] [DEBUG] -- execute sql: select 1. args: None [2025-05-06 09:34:33.682] [DEBUG] - sub connect ref count to 0 [2025-05-06 09:34:33.683] [DEBUG] - export connect [2025-05-06 09:34:33.683] [DEBUG] - plugin oceanbase-ce-py_script_connect-3.1.0 result: True [2025-05-06 09:34:33.684] [DEBUG] - Searching get_relation_tenants plugin for components ... [2025-05-06 09:34:33.684] [DEBUG] - Searching get_relation_tenants plugin for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.687] [DEBUG] - Found for oceanbase-ce-py_script_get_relation_tenants-4.2.0.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:33.689] [DEBUG] - Call plugin oceanbase-ce-py_script_get_relation_tenants-4.2.0.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.690] [DEBUG] - import get_relation_tenants [2025-05-06 09:34:33.693] [DEBUG] - add get_relation_tenants ref count to 1 [2025-05-06 09:34:33.694] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_cluster_102166`, count 2 [2025-05-06 09:34:33.786] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_cluster_10346 [2025-05-06 09:34:33.787] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_cluster_10346`, count 1 [2025-05-06 09:34:33.868] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_cluster_10346`, count 2 [2025-05-06 09:34:33.944] [DEBUG] - sub get_relation_tenants ref count to 0 [2025-05-06 09:34:33.945] [DEBUG] - export get_relation_tenants [2025-05-06 09:34:33.945] [DEBUG] - plugin oceanbase-ce-py_script_get_relation_tenants-4.2.0.0 result: True [2025-05-06 09:34:33.945] [DEBUG] - Searching get_deployment_connections plugin for components ... [2025-05-06 09:34:33.946] [DEBUG] - Searching get_deployment_connections plugin for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.948] [DEBUG] - Found for oceanbase-ce-py_script_get_deployment_connections-4.2.0.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:33.948] [DEBUG] - Call plugin oceanbase-ce-py_script_get_deployment_connections-4.2.0.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:33.949] [DEBUG] - import get_deployment_connections [2025-05-06 09:34:33.951] [DEBUG] - add get_deployment_connections ref count to 1 [2025-05-06 09:34:33.953] [INFO] Get deployment connections [2025-05-06 09:34:34.086] [DEBUG] - sub get_deployment_connections ref count to 0 [2025-05-06 09:34:34.087] [DEBUG] - export get_deployment_connections [2025-05-06 09:34:34.087] [DEBUG] - plugin oceanbase-ce-py_script_get_deployment_connections-4.2.0.0 result: True [2025-05-06 09:34:34.088] [DEBUG] - Searching get_standbys plugin for components ... [2025-05-06 09:34:34.088] [DEBUG] - Searching get_standbys plugin for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:34.091] [DEBUG] - Found for oceanbase-ce-py_script_get_standbys-4.2.0.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:34.092] [DEBUG] - Call plugin oceanbase-ce-py_script_get_standbys-4.2.0.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:34.092] [DEBUG] - import get_standbys [2025-05-06 09:34:34.096] [DEBUG] - add get_standbys ref count to 1 [2025-05-06 09:34:34.098] [INFO] Get standbys info [2025-05-06 09:34:34.100] [DEBUG] -- execute sql: select TENANT_ID, group_concat(host separator ";") as ip_list from (select concat(svr_ip,":",SQL_PORT) as host,TENANT_ID from oceanbase.cdb_ob_access_point where tenant_name=%s). args: ('tenant',) [2025-05-06 09:34:34.183] [DEBUG] -- execute sql: select TENANT_ID from oceanbase.DBA_OB_TENANTS where tenant_name=%s. args: ('tenant',) [2025-05-06 09:34:34.223] [DEBUG] -- find primary, primary tenant cluster_102166:tenant's ip list:['172.36.102.166:2881'], relation_tenants:[('cluster_10346', 'tenant'), ('cluster_102166', 'tenant')], exclude_tenant:[] [2025-05-06 09:34:34.224] [DEBUG] -- execute sql: select TENANT_ROLE from oceanbase.DBA_OB_TENANTS where TENANT_NAME = %s. args: ('tenant',) [2025-05-06 09:34:34.363] [DEBUG] - sub get_standbys ref count to 0 [2025-05-06 09:34:34.363] [DEBUG] - export get_standbys [2025-05-06 09:34:34.364] [DEBUG] - plugin oceanbase-ce-py_script_get_standbys-4.2.0.0 result: True [2025-05-06 09:34:34.365] [DEBUG] - Searching check_exit_standby plugin for components ... [2025-05-06 09:34:34.365] [DEBUG] - Searching check_exit_standby plugin for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:34.367] [DEBUG] - Found for oceanbase-ce-py_script_check_exit_standby-4.2.0.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:34.368] [DEBUG] - Call plugin oceanbase-ce-py_script_check_exit_standby-4.2.0.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:34.368] [DEBUG] - import check_exit_standby [2025-05-06 09:34:34.370] [DEBUG] - add check_exit_standby ref count to 1 [2025-05-06 09:34:34.371] [DEBUG] - sub check_exit_standby ref count to 0 [2025-05-06 09:34:34.372] [DEBUG] - export check_exit_standby [2025-05-06 09:34:34.372] [DEBUG] - plugin oceanbase-ce-py_script_check_exit_standby-4.2.0.0 result: True [2025-05-06 09:34:34.372] [DEBUG] - Searching drop_tenant plugin for components ... [2025-05-06 09:34:34.373] [DEBUG] - Searching drop_tenant plugin for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:34.375] [DEBUG] - Found for oceanbase-ce-py_script_drop_tenant-4.0.0.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:34.375] [DEBUG] - Call plugin oceanbase-ce-py_script_drop_tenant-4.0.0.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:34.375] [DEBUG] - import drop_tenant [2025-05-06 09:34:34.379] [DEBUG] - add drop_tenant ref count to 1 [2025-05-06 09:34:34.381] [INFO] Drop tenant tenant [2025-05-06 09:34:34.383] [DEBUG] -- execute sql: select * from oceanbase.DBA_OB_TENANTS where tenant_name = %s. args: ['tenant'] [2025-05-06 09:34:34.416] [DEBUG] -- execute sql: select * from oceanbase.DBA_OB_RESOURCE_POOLS where tenant_id = 1016. args: None [2025-05-06 09:34:34.488] [DEBUG] -- execute sql: select * from oceanbase.DBA_OB_UNIT_CONFIGS where unit_config_id = 1008. args: None [2025-05-06 09:34:34.516] [DEBUG] - sub drop_tenant ref count to 0 [2025-05-06 09:34:34.517] [DEBUG] - export drop_tenant [2025-05-06 09:34:34.517] [DEBUG] - plugin oceanbase-ce-py_script_drop_tenant-4.0.0.0 result: True [2025-05-06 09:34:34.518] [DEBUG] - Searching delete_standby_info plugin for components ... [2025-05-06 09:34:34.518] [DEBUG] - Searching delete_standby_info plugin for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:34.521] [DEBUG] - Found for oceanbase-ce-py_script_delete_standby_info-4.2.0.0 for oceanbase-ce-4.3.5.1 [2025-05-06 09:34:34.521] [DEBUG] - Call plugin oceanbase-ce-py_script_delete_standby_info-4.2.0.0 for oceanbase-ce-4.3.5.1-101000042025031818.nonlse.el8-bd2d798325c8c75f083ddc15c65408ee43435ab1 [2025-05-06 09:34:34.522] [DEBUG] - import delete_standby_info [2025-05-06 09:34:34.525] [DEBUG] - add delete_standby_info ref count to 1 [2025-05-06 09:34:34.529] [DEBUG] - open /root/.obd/cluster/cluster_102166/inner_config.yaml for w [2025-05-06 09:34:34.557] [DEBUG] - open /root/.obd/cluster/cluster_102166/inner_config.yaml for w [2025-05-06 09:34:34.586] [DEBUG] - open /root/.obd/cluster/cluster_10346/inner_config.yaml for w [2025-05-06 09:34:34.619] [DEBUG] - open /root/.obd/cluster/cluster_10346/inner_config.yaml for w [2025-05-06 09:34:34.646] [DEBUG] - sub delete_standby_info ref count to 0 [2025-05-06 09:34:34.647] [DEBUG] - export delete_standby_info [2025-05-06 09:34:34.647] [DEBUG] - plugin oceanbase-ce-py_script_delete_standby_info-4.2.0.0 result: True [2025-05-06 09:34:34.648] [DEBUG] - exclusive lock /root/.obd/lock/deploy_cluster_10346 release, count 1 [2025-05-06 09:34:34.648] [DEBUG] - exclusive lock /root/.obd/lock/deploy_cluster_10346 release, count 0 [2025-05-06 09:34:34.649] [DEBUG] - unlock /root/.obd/lock/deploy_cluster_10346 [2025-05-06 09:34:34.649] [DEBUG] - exclusive lock /root/.obd/lock/deploy_cluster_102166 release, count 1 [2025-05-06 09:34:34.649] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 3 [2025-05-06 09:34:34.650] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 2 [2025-05-06 09:34:34.650] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1 [2025-05-06 09:34:34.650] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0 [2025-05-06 09:34:34.651] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo [2025-05-06 09:34:34.651] [DEBUG] - exclusive lock /root/.obd/lock/deploy_cluster_102166 release, count 0 [2025-05-06 09:34:34.652] [DEBUG] - unlock /root/.obd/lock/deploy_cluster_102166 [2025-05-06 09:34:34.652] [DEBUG] - share lock /root/.obd/lock/global release, count 0 [2025-05-06 09:34:34.652] [DEBUG] - unlock /root/.obd/lock/global [2025-05-06 09:34:34.653] [INFO] Trace ID: 4d07e48e-2a5d-11f0-8c63-5254009d7d10 [2025-05-06 09:34:34.653] [INFO] If you want to view detailed obd logs, please run: obd display-trace 4d07e48e-2a5d-11f0-8c63-5254009d7d10