[2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] Open ssh connection [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] Cluster param config check [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:08:52] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching status plugin for components ... [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching status plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Searching status plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Found for obproxy-py_script_status-3.1.0 for obproxy-3.2.0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - import status [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - add status ref count to 1 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ls /proc/29930 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/29930': No such file or directory [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - sub status ref count to 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - export status [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Call obproxy-py_script_status-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - import status [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - add status ref count to 1 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - sub status ref count to 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - export status [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] Check before start observer [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ls /proc/29930 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/29930': No such file or directory [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -1.48193359375 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] Check before start obproxy [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:08:53] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - import start [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] Start observer [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ls /proc/29930 [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/29930': No such file or directory [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 01:08:54] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] observer program health check [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- local execute: ls /proc/109 [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 109] started [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - export start [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - import connect [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] Connect to observer [2024-04-29 01:08:57] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 01:09:00] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 01:09:00] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 01:09:00] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - export connect [2024-04-29 01:09:00] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:09:00] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - import display [2024-04-29 01:09:00] [0b3d588a-05c5-11ef-b046-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 01:09:00] [0b3d588a-05c5-11ef-b046-0242ac110002] [INFO] Wait for observer init [2024-04-29 01:19:17] [7fcea8ec-05c6-11ef-9455-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 01:19:17] [7fcea8ec-05c6-11ef-9455-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 01:19:25] [84575a08-05c6-11ef-9cd1-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 01:19:25] [84575a08-05c6-11ef-9cd1-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 01:29:42] [0b3d588a-05c5-11ef-b046-0242ac110002] [ERROR] [ERROR] Running Error. [2024-04-29 01:29:42] [0b3d588a-05c5-11ef-b046-0242ac110002] [ERROR] Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 32, in display cursor.execute('select * from oceanbase.__all_server') File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/usr/obd/lib/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/usr/obd/lib/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist") During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 42, in display time.sleep(3) KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "obd.py", line 136, in do_command File "obd.py", line 381, in _do_command File "core.py", line 1025, in start_cluster File "_plugin.py", line 234, in __call__ File "_plugin.py", line 208, in _new_func File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 44, in display stdio.stop_loading('fail', 'observer need bootstarp') File "_stdio.py", line 313, in stop_loading File "_stdio.py", line 288, in _stop_sync_obj File "_stdio.py", line 294, in _stop_sync_obj File "_stdio.py", line 137, in fail File "_stdio.py", line 129, in stop_and_persist File "halo/halo.py", line 598, in stop_and_persist File "halo/halo.py", line 512, in stop File "threading.py", line 1011, in join File "threading.py", line 1027, in _wait_for_tstate_lock KeyboardInterrupt [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None} [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Check the deploy status [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:34:13] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Searching stop plugin for components ... [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Searching stop plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Searching stop plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Found for obproxy-py_script_stop-3.1.0 for obproxy-3.2.0 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [INFO] Open ssh connection [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - import stop [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [INFO] Stop observer [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- local execute: ps uax | egrep " 109 " | grep -v grep [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid:109] stopping ... [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- local execute: kill -9 109 [2024-04-29 01:34:14] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- 127.0.0.1 check whether the port is released [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- local execute: rm -f /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- 127.0.0.1 observer is stopped [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - export stop [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Call obproxy-py_script_stop-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - import stop [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [INFO] Stop obproxy [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:34:15] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:34:17] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 01:34:17] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - export stop [2024-04-29 01:34:17] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - Set obdemo deploy status to stopped [2024-04-29 01:34:17] [95e1cf22-05c8-11ef-9b76-0242ac110002] [DEBUG] - dump deploy info to /home/admin/.obd/cluster/obdemo/.data [2024-04-29 01:34:17] [95e1cf22-05c8-11ef-9b76-0242ac110002] [INFO] obdemo stopped [2024-04-29 01:35:13] [b94305f8-05c8-11ef-9652-0242ac110002] [DEBUG] - cmd: [] [2024-04-29 01:35:13] [b94305f8-05c8-11ef-9652-0242ac110002] [DEBUG] - opts: {} [2024-04-29 01:35:13] [b94305f8-05c8-11ef-9652-0242ac110002] [DEBUG] - Get deploy list [2024-04-29 01:35:13] [b94305f8-05c8-11ef-9652-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:35:13] [b94305f8-05c8-11ef-9652-0242ac110002] [INFO] +------------------------------------------------------------+ | Cluster List | +--------+---------------------------------+-----------------+ | Name | Configuration Path | Status (Cached) | +--------+---------------------------------+-----------------+ | obdemo | /home/admin/.obd/cluster/obdemo | stopped | +--------+---------------------------------+-----------------+ [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] Open ssh connection [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] Cluster param config check [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:35:21] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] Check before start observer [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 1, error output: cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -1.639404296875 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] Check before start obproxy [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - import start [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] Start observer [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 1, error output: cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 01:35:22] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] observer program health check [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- local execute: ls /proc/93 [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 93] started [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - export start [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - import connect [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] Connect to observer [2024-04-29 01:35:25] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 01:35:29] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 01:35:29] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 01:35:29] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - export connect [2024-04-29 01:35:29] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:35:29] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - import display [2024-04-29 01:35:29] [be114f5e-05c8-11ef-acbd-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 01:35:29] [be114f5e-05c8-11ef-acbd-0242ac110002] [INFO] Wait for observer init [2024-04-29 01:36:58] [be114f5e-05c8-11ef-acbd-0242ac110002] [ERROR] [ERROR] Running Error. [2024-04-29 01:36:58] [be114f5e-05c8-11ef-acbd-0242ac110002] [ERROR] Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 32, in display cursor.execute('select * from oceanbase.__all_server') File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/usr/obd/lib/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/usr/obd/lib/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist") During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 42, in display time.sleep(3) KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "obd.py", line 136, in do_command File "obd.py", line 381, in _do_command File "core.py", line 1025, in start_cluster File "_plugin.py", line 234, in __call__ File "_plugin.py", line 208, in _new_func File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 44, in display stdio.stop_loading('fail', 'observer need bootstarp') File "_stdio.py", line 313, in stop_loading File "_stdio.py", line 288, in _stop_sync_obj File "_stdio.py", line 294, in _stop_sync_obj File "_stdio.py", line 137, in fail File "_stdio.py", line 129, in stop_and_persist File "halo/halo.py", line 598, in stop_and_persist File "halo/halo.py", line 512, in stop File "threading.py", line 1011, in join File "threading.py", line 1027, in _wait_for_tstate_lock KeyboardInterrupt [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None} [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Check the deploy status [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Searching stop plugin for components ... [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Searching stop plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Searching stop plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:37:04] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Found for obproxy-py_script_stop-3.1.0 for obproxy-3.2.0 [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [INFO] Open ssh connection [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - import stop [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [INFO] Stop observer [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- local execute: ps uax | egrep " 93 " | grep -v grep [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid:93] stopping ... [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- local execute: kill -9 93 [2024-04-29 01:37:05] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- 127.0.0.1 check whether the port is released [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- local execute: rm -f /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- 127.0.0.1 observer is stopped [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - export stop [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Call obproxy-py_script_stop-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - import stop [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [INFO] Stop obproxy [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:37:06] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:37:07] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 01:37:07] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - export stop [2024-04-29 01:37:07] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - Set obdemo deploy status to stopped [2024-04-29 01:37:07] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [DEBUG] - dump deploy info to /home/admin/.obd/cluster/obdemo/.data [2024-04-29 01:37:07] [fb7139d6-05c8-11ef-9c3f-0242ac110002] [INFO] obdemo stopped [2024-04-29 01:37:18] [03de7804-05c9-11ef-af4a-0242ac110002] [DEBUG] - cmd: [] [2024-04-29 01:37:18] [03de7804-05c9-11ef-af4a-0242ac110002] [DEBUG] - opts: {} [2024-04-29 01:37:18] [03de7804-05c9-11ef-af4a-0242ac110002] [DEBUG] - Get deploy list [2024-04-29 01:37:18] [03de7804-05c9-11ef-af4a-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:37:18] [03de7804-05c9-11ef-af4a-0242ac110002] [INFO] +------------------------------------------------------------+ | Cluster List | +--------+---------------------------------+-----------------+ | Name | Configuration Path | Status (Cached) | +--------+---------------------------------+-----------------+ | obdemo | /home/admin/.obd/cluster/obdemo | stopped | +--------+---------------------------------+-----------------+ [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] Open ssh connection [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] Cluster param config check [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:38:11] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] Check before start observer [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 1, error output: cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -0.67529296875 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] Check before start obproxy [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:38:12] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - import start [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] Start observer [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 1, error output: cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 01:38:13] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] observer program health check [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- local execute: ls /proc/759 [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 759] started [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - export start [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - import connect [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] Connect to observer [2024-04-29 01:38:16] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 01:38:19] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 01:38:19] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 01:38:19] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - export connect [2024-04-29 01:38:19] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:38:19] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - import display [2024-04-29 01:38:19] [23a7f174-05c9-11ef-a0e0-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 01:38:19] [23a7f174-05c9-11ef-a0e0-0242ac110002] [INFO] Wait for observer init [2024-04-29 01:40:01] [651fbbf0-05c9-11ef-84a2-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 01:40:01] [651fbbf0-05c9-11ef-84a2-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 01:40:37] [7a6f1c58-05c9-11ef-807a-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 01:40:37] [7a6f1c58-05c9-11ef-807a-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 01:40:42] [7dad9ee4-05c9-11ef-bf5e-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 01:40:42] [7dad9ee4-05c9-11ef-bf5e-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None} [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Check the deploy status [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 01:43:20] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Searching stop plugin for components ... [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Searching stop plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Searching stop plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Found for obproxy-py_script_stop-3.1.0 for obproxy-3.2.0 [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [INFO] Open ssh connection [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - import stop [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [INFO] Stop observer [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- local execute: ps uax | egrep " 759 " | grep -v grep [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 01:43:21] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- 127.0.0.1 observer is not running ... [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - export stop [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Call obproxy-py_script_stop-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - import stop [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [INFO] Stop obproxy [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:43:22] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:43:23] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 01:43:23] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - export stop [2024-04-29 01:43:23] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - Set obdemo deploy status to stopped [2024-04-29 01:43:23] [db9dc39e-05c9-11ef-9a23-0242ac110002] [DEBUG] - dump deploy info to /home/admin/.obd/cluster/obdemo/.data [2024-04-29 01:43:23] [db9dc39e-05c9-11ef-9a23-0242ac110002] [INFO] obdemo stopped [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] Open ssh connection [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] Cluster param config check [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:43:39] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] Check before start observer [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: ls /proc/759 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/759': No such file or directory [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -1.4990234375 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] Check before start obproxy [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:43:40] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - import start [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] Start observer [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: ls /proc/759 [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/759': No such file or directory [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 01:43:41] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] observer program health check [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- local execute: ls /proc/107 [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 107] started [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - export start [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - import connect [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] Connect to observer [2024-04-29 01:43:44] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 01:43:47] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 01:43:47] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 01:43:47] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - export connect [2024-04-29 01:43:47] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:43:47] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - import display [2024-04-29 01:43:47] [e72d0242-05c9-11ef-baea-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 01:43:47] [e72d0242-05c9-11ef-baea-0242ac110002] [INFO] Wait for observer init [2024-04-29 01:55:44] [e72d0242-05c9-11ef-baea-0242ac110002] [ERROR] [ERROR] Running Error. [2024-04-29 01:55:44] [e72d0242-05c9-11ef-baea-0242ac110002] [ERROR] Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 32, in display cursor.execute('select * from oceanbase.__all_server') File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/usr/obd/lib/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/usr/obd/lib/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist") During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 42, in display time.sleep(3) KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "obd.py", line 136, in do_command File "obd.py", line 381, in _do_command File "core.py", line 1025, in start_cluster File "_plugin.py", line 234, in __call__ File "_plugin.py", line 208, in _new_func File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 44, in display stdio.stop_loading('fail', 'observer need bootstarp') File "_stdio.py", line 313, in stop_loading File "_stdio.py", line 288, in _stop_sync_obj File "_stdio.py", line 294, in _stop_sync_obj File "_stdio.py", line 137, in fail File "_stdio.py", line 129, in stop_and_persist File "halo/halo.py", line 598, in stop_and_persist File "halo/halo.py", line 512, in stop File "threading.py", line 1011, in join File "threading.py", line 1027, in _wait_for_tstate_lock KeyboardInterrupt [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] Open ssh connection [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] Cluster param config check [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 01:58:22] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] Check before start observer [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: ls /proc/107 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/107': No such file or directory [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -1.483154296875 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - import start_check [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] Check before start obproxy [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - export start_check [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - import start [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] Start observer [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: ls /proc/107 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/107': No such file or directory [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 01:58:23] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] observer program health check [2024-04-29 01:58:26] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 01:58:26] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 01:58:26] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:26] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- local execute: ls /proc/93 [2024-04-29 01:58:26] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 01:58:26] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 93] started [2024-04-29 01:58:27] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 01:58:27] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - export start [2024-04-29 01:58:27] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:27] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - import connect [2024-04-29 01:58:27] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 01:58:27] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] Connect to observer [2024-04-29 01:58:27] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 01:58:30] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 01:58:30] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 01:58:30] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - export connect [2024-04-29 01:58:30] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 01:58:30] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - import display [2024-04-29 01:58:30] [f551d468-05cb-11ef-85c7-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 01:58:30] [f551d468-05cb-11ef-85c7-0242ac110002] [INFO] Wait for observer init [2024-04-29 02:13:46] [f551d468-05cb-11ef-85c7-0242ac110002] [ERROR] [ERROR] Running Error. [2024-04-29 02:13:46] [f551d468-05cb-11ef-85c7-0242ac110002] [ERROR] Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 32, in display cursor.execute('select * from oceanbase.__all_server') File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/usr/obd/lib/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/usr/obd/lib/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist") During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 42, in display time.sleep(3) KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "obd.py", line 136, in do_command File "obd.py", line 381, in _do_command File "core.py", line 1025, in start_cluster File "_plugin.py", line 234, in __call__ File "_plugin.py", line 208, in _new_func File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 44, in display stdio.stop_loading('fail', 'observer need bootstarp') File "_stdio.py", line 313, in stop_loading File "_stdio.py", line 288, in _stop_sync_obj File "_stdio.py", line 294, in _stop_sync_obj File "_stdio.py", line 137, in fail File "_stdio.py", line 129, in stop_and_persist File "halo/halo.py", line 598, in stop_and_persist File "halo/halo.py", line 512, in stop File "threading.py", line 1011, in join File "threading.py", line 1027, in _wait_for_tstate_lock KeyboardInterrupt [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None} [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Check the deploy status [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 02:17:46] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Searching stop plugin for components ... [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Searching stop plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Searching stop plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Found for obproxy-py_script_stop-3.1.0 for obproxy-3.2.0 [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [INFO] Open ssh connection [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - import stop [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [INFO] Stop observer [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- local execute: ps uax | egrep " 93 " | grep -v grep [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid:93] stopping ... [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- local execute: kill -9 93 [2024-04-29 02:17:47] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- 127.0.0.1 check whether the port is released [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- local execute: rm -f /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- 127.0.0.1 observer is stopped [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - export stop [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Call obproxy-py_script_stop-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - import stop [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [INFO] Stop obproxy [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 02:17:48] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 02:17:49] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 02:17:49] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - export stop [2024-04-29 02:17:49] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - Set obdemo deploy status to stopped [2024-04-29 02:17:49] [ab00391a-05ce-11ef-9109-0242ac110002] [DEBUG] - dump deploy info to /home/admin/.obd/cluster/obdemo/.data [2024-04-29 02:17:49] [ab00391a-05ce-11ef-9109-0242ac110002] [INFO] obdemo stopped [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] Open ssh connection [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] Cluster param config check [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:17:54] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - import start_check [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] Check before start observer [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 1, error output: cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -1.444580078125 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - export start_check [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - import start_check [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] Check before start obproxy [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - export start_check [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - import start [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] Start observer [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 1, error output: cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 02:17:55] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] observer program health check [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- local execute: ls /proc/787 [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 787] started [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - export start [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - import connect [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] Connect to observer [2024-04-29 02:17:58] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 02:18:02] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 02:18:02] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 02:18:02] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - export connect [2024-04-29 02:18:02] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:18:02] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - import display [2024-04-29 02:18:02] [afcfd37e-05ce-11ef-a42f-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 02:18:02] [afcfd37e-05ce-11ef-a42f-0242ac110002] [INFO] Wait for observer init [2024-04-29 02:29:45] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 02:29:45] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 02:29:45] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 02:29:45] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 02:29:45] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 02:29:45] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] Open ssh connection [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] Cluster param config check [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 02:29:46] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - import start_check [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] Check before start observer [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: ls /proc/787 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/787': No such file or directory [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -1.492431640625 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - export start_check [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - import start_check [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] Check before start obproxy [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - export start_check [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - import start [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] Start observer [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: ls /proc/787 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/787': No such file or directory [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 02:29:47] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] observer program health check [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- local execute: ls /proc/102 [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 102] started [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - export start [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - import connect [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] Connect to observer [2024-04-29 02:29:50] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 02:29:53] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 02:29:53] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 02:29:53] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - export connect [2024-04-29 02:29:53] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:29:53] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - import display [2024-04-29 02:29:53] [5804d674-05d0-11ef-94c8-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 02:29:53] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] Wait for observer init [2024-04-29 02:31:25] [934e7924-05d0-11ef-b583-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:31:25] [934e7924-05d0-11ef-b583-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:31:42] [9d8335ba-05d0-11ef-8345-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:31:42] [9d8335ba-05d0-11ef-8345-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:42:24] [1c1a8238-05d2-11ef-a83e-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:42:24] [1c1a8238-05d2-11ef-a83e-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:42:56] [2f7ffff6-05d2-11ef-8678-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:42:56] [2f7ffff6-05d2-11ef-8678-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:43:57] [5804d674-05d0-11ef-94c8-0242ac110002] [ERROR] [ERROR] Running Error. [2024-04-29 02:43:57] [5804d674-05d0-11ef-94c8-0242ac110002] [ERROR] Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 32, in display cursor.execute('select * from oceanbase.__all_server') File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/usr/obd/lib/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/usr/obd/lib/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist") During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 42, in display time.sleep(3) KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "obd.py", line 136, in do_command File "obd.py", line 381, in _do_command File "core.py", line 1025, in start_cluster File "_plugin.py", line 234, in __call__ File "_plugin.py", line 208, in _new_func File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 42, in display time.sleep(3) KeyboardInterrupt [2024-04-29 02:43:57] [5804d674-05d0-11ef-94c8-0242ac110002] [INFO] [ERROR] Running Error. [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] Open ssh connection [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] Cluster param config check [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:44:00] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - import start_check [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] Check before start observer [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: ls /proc/102 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -0.470947265625 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - export start_check [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - import start_check [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] Check before start obproxy [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - export start_check [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - import start [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] Start observer [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: ls /proc/102 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:01] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] observer program health check [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- local execute: ls /proc/102 [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 102] started [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - export start [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - import connect [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] Connect to observer [2024-04-29 02:44:04] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 02:44:07] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 02:44:08] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 02:44:08] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - export connect [2024-04-29 02:44:08] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:44:08] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - import display [2024-04-29 02:44:08] [552cf150-05d2-11ef-a073-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 02:44:08] [552cf150-05d2-11ef-a073-0242ac110002] [INFO] Wait for observer init [2024-04-29 02:44:10] [5b415b4e-05d2-11ef-b25b-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:44:10] [5b415b4e-05d2-11ef-b25b-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None} [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Check the deploy status [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 02:45:13] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Searching stop plugin for components ... [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Searching stop plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Searching stop plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Found for obproxy-py_script_stop-3.1.0 for obproxy-3.2.0 [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [INFO] Open ssh connection [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - import stop [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [INFO] Stop observer [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- local execute: ps uax | egrep " 102 " | grep -v grep [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 02:45:14] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- 127.0.0.1 observer is not running ... [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - export stop [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Call obproxy-py_script_stop-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - import stop [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [INFO] Stop obproxy [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 02:45:15] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 02:45:16] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 02:45:16] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - export stop [2024-04-29 02:45:16] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - Set obdemo deploy status to stopped [2024-04-29 02:45:16] [80b1325a-05d2-11ef-84ed-0242ac110002] [DEBUG] - dump deploy info to /home/admin/.obd/cluster/obdemo/.data [2024-04-29 02:45:16] [80b1325a-05d2-11ef-84ed-0242ac110002] [INFO] obdemo stopped [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] Open ssh connection [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] Cluster param config check [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 02:45:51] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - import start_check [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] Check before start observer [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: ls /proc/102 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/102': No such file or directory [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -1.5244140625 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - export start_check [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - import start_check [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] Check before start obproxy [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - export start_check [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - import start [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] Start observer [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: ls /proc/102 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/102': No such file or directory [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 02:45:52] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] observer program health check [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- local execute: ls /proc/108 [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 108] started [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - export start [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - import connect [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] Connect to observer [2024-04-29 02:45:55] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 02:45:58] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 02:45:58] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 02:45:58] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - export connect [2024-04-29 02:45:58] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 02:45:58] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - import display [2024-04-29 02:45:58] [9737cc78-05d2-11ef-b253-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 02:45:58] [9737cc78-05d2-11ef-b253-0242ac110002] [INFO] Wait for observer init [2024-04-29 02:46:49] [ba4507b2-05d2-11ef-b0a1-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:46:49] [ba4507b2-05d2-11ef-b0a1-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:47:24] [cf0799bc-05d2-11ef-b60e-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:47:24] [cf0799bc-05d2-11ef-b60e-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:49:32] [1b19e6fc-05d3-11ef-b263-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:49:32] [1b19e6fc-05d3-11ef-b263-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:50:19] [3767e17e-05d3-11ef-bf62-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:50:19] [3767e17e-05d3-11ef-bf62-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:50:25] [3aaf32c4-05d3-11ef-b901-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:50:25] [3aaf32c4-05d3-11ef-b901-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:50:39] [43290cea-05d3-11ef-95ce-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:50:39] [43290cea-05d3-11ef-95ce-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:51:13] [57ac96c8-05d3-11ef-aeb8-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:51:13] [57ac96c8-05d3-11ef-aeb8-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:52:01] [74367034-05d3-11ef-98c5-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:52:01] [74367034-05d3-11ef-98c5-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:56:26] [120f26ac-05d4-11ef-8862-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 02:56:26] [120f26ac-05d4-11ef-8862-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 02:58:09] [9737cc78-05d2-11ef-b253-0242ac110002] [ERROR] [ERROR] Running Error. [2024-04-29 02:58:09] [9737cc78-05d2-11ef-b253-0242ac110002] [ERROR] Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 32, in display cursor.execute('select * from oceanbase.__all_server') File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/usr/obd/lib/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/obd/lib/site-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/usr/obd/lib/site-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/usr/obd/lib/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/usr/obd/lib/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.ProgrammingError: (1146, "Table 'oceanbase.__all_server' doesn't exist") During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 42, in display time.sleep(3) KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "obd.py", line 136, in do_command File "obd.py", line 381, in _do_command File "core.py", line 1025, in start_cluster File "_plugin.py", line 234, in __call__ File "_plugin.py", line 208, in _new_func File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/display.py", line 44, in display stdio.stop_loading('fail', 'observer need bootstarp') File "_stdio.py", line 313, in stop_loading File "_stdio.py", line 288, in _stop_sync_obj File "_stdio.py", line 294, in _stop_sync_obj File "_stdio.py", line 137, in fail File "_stdio.py", line 129, in stop_and_persist File "halo/halo.py", line 598, in stop_and_persist File "halo/halo.py", line 512, in stop File "threading.py", line 1011, in join File "threading.py", line 1027, in _wait_for_tstate_lock KeyboardInterrupt [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None} [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Check the deploy status [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:00:09] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Searching stop plugin for components ... [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Searching stop plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Searching stop plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Found for obproxy-py_script_stop-3.1.0 for obproxy-3.2.0 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [INFO] Open ssh connection [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - import stop [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [INFO] Stop observer [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- local execute: ps uax | egrep " 108 " | grep -v grep [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid:108] stopping ... [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- local execute: kill -9 108 [2024-04-29 03:00:10] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- 127.0.0.1 check whether the port is released [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- local execute: rm -f /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- 127.0.0.1 observer is stopped [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - export stop [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Call obproxy-py_script_stop-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - import stop [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - add stop ref count to 1 [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [INFO] Stop obproxy [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 03:00:11] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 03:00:12] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - sub stop ref count to 0 [2024-04-29 03:00:12] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - export stop [2024-04-29 03:00:12] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - Set obdemo deploy status to stopped [2024-04-29 03:00:12] [96e1a468-05d4-11ef-8343-0242ac110002] [DEBUG] - dump deploy info to /home/admin/.obd/cluster/obdemo/.data [2024-04-29 03:00:12] [96e1a468-05d4-11ef-8343-0242ac110002] [INFO] obdemo stopped [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] Open ssh connection [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] Cluster param config check [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:00:50] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - import start_check [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] Check before start observer [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 1, error output: cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -0.54150390625 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - export start_check [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - import start_check [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] Check before start obproxy [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - export start_check [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - import start [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] Start observer [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 1, error output: cat: /home/admin/oceanbase-ce/run/observer.pid: No such file or directory [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 03:00:51] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] observer program health check [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- local execute: ls /proc/876 [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 876] started [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - export start [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - import connect [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] Connect to observer [2024-04-29 03:00:54] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 03:00:57] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 03:00:57] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 03:00:57] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - export connect [2024-04-29 03:00:57] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:00:57] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - import display [2024-04-29 03:00:57] [af1976aa-05d4-11ef-9eff-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 03:00:57] [af1976aa-05d4-11ef-9eff-0242ac110002] [INFO] Wait for observer init [2024-04-29 03:06:11] [6ec3aa98-05d5-11ef-80e3-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 03:06:11] [6ec3aa98-05d5-11ef-80e3-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 03:49:16] [73b3a426-05db-11ef-9ab3-0242ac110002] [ERROR] [ERROR] OBD is running. [2024-04-29 03:49:16] [73b3a426-05db-11ef-9ab3-0242ac110002] [ERROR] Traceback (most recent call last): File "obd.py", line 132, in do_command File "core.py", line 58, in __init__ File "core.py", line 64, in lock BlockingIOError: [Errno 11] Resource temporarily unavailable [2024-04-29 03:50:39] [a4ec441c-05db-11ef-8ef5-0242ac110002] [DEBUG] - cmd: [] [2024-04-29 03:50:39] [a4ec441c-05db-11ef-8ef5-0242ac110002] [DEBUG] - opts: {} [2024-04-29 03:50:39] [a4ec441c-05db-11ef-8ef5-0242ac110002] [DEBUG] - Get deploy list [2024-04-29 03:50:39] [a4ec441c-05db-11ef-8ef5-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 03:50:39] [a4ec441c-05db-11ef-8ef5-0242ac110002] [INFO] +------------------------------------------------------------+ | Cluster List | +--------+---------------------------------+-----------------+ | Name | Configuration Path | Status (Cached) | +--------+---------------------------------+-----------------+ | obdemo | /home/admin/.obd/cluster/obdemo | stopped | +--------+---------------------------------+-----------------+ [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - cmd: ['obdemo'] [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Get Deploy by name [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/cluster/ [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Deploy status judge [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Get deploy config [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] Get local repositories and plugins [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Get local repository oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/oceanbase-ce [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Get local repository obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/repository/obproxy [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching start_check plugin for components ... [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching start_check plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - mkdir /home/admin/.obd/plugins [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching start_check plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching start plugin for components ... [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching start plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching start plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for obproxy-py_script_start-3.1.0 for obproxy-3.2.0 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching connect plugin for components ... [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching connect plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching connect plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for obproxy-py_script_connect-3.1.0 for obproxy-3.2.0 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching bootstrap plugin for components ... [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching bootstrap plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for obproxy-py_script_bootstrap-3.1.0 for obproxy-3.2.0 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching display plugin for components ... [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching display plugin for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching display plugin for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for obproxy-py_script_display-3.1.0 for obproxy-3.2.0 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] Open ssh connection [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] Cluster param config check [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Searching param plugin for components ... [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Search param plugin for oceanbase-ce [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1 [2024-04-29 03:52:52] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Applying oceanbase-ce-param-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Search param plugin for obproxy [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Found for obproxy-param-3.1.0 for obproxy-3.2.0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Applying obproxy-param-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - 127.0.0.1 oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 param check [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - 127.0.0.1 obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd param check [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start_check-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - import start_check [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] Check before start observer [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: ls /proc/876 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/876': No such file or directory [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/sstable/block_file [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: ulimit -n [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [WARNING] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: cat /proc/meminfo [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: df --block-size=1024 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: date +%s%N [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- 127.0.0.1 time delta -1.504150390625 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] [WARN] (127.0.0.1) The recommended number of open files is 655350 (Current value: 65536) [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - export start_check [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Call obproxy-py_script_start_check-3.1.0 for obproxy-3.2.0-42a720977963f939282a470378ca383cc5824dfd [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - import start_check [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - add start_check ref count to 1 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] Check before start obproxy [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: cat /home/admin/obproxy/run/obproxy-127.0.0.1-2883.pid [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: ls /proc/30557 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/30557': No such file or directory [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- 127.0.0.1 port check [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B43' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp,udp}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B44' | awk -F' ' '{print $2}' | uniq [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - sub start_check ref count to 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - export start_check [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - import start [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - add start ref count to 1 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] Start observer [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: bash -c 'if [ -f /home/admin/oceanbase-ce/bin/observer ]; then exit 1; else exit 0; fi;' [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 1, error output: [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: ls /home/admin/oceanbase-ce/store/ilog/ [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: ls /proc/876 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 2, error output: ls: cannot access '/proc/876': No such file or directory [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- 127.0.0.1 start command construction [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- starting 127.0.0.1 observer [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '/home/admin/oceanbase-ce/lib:' [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -o __min_full_resource_pool_memory=268435456,memory_limit='10G',system_memory='4G',stack_size='512K',cpu_count=16,cache_wash_threshold='1G',workers_per_cpu_quota=10,schema_history_expire_time='1d',net_thread_count=4,major_freeze_duty_time='Disable',minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,max_user_connections=0,datafile_size='5G',enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,root_password='rootPWD123' -z 'zone1' -p 2881 -P 2882 -n 'obce-single' -c 1 -d '/home/admin/oceanbase-ce/store' -i 'lo' -l 'ERROR' [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- admin@127.0.0.1 set env LD_LIBRARY_PATH to '' [2024-04-29 03:52:53] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] observer program health check [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- 127.0.0.1 program health check [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: cat /home/admin/oceanbase-ce/run/observer.pid [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- local execute: ls /proc/95 [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- exited code 0 [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- 127.0.0.1 observer[pid: 95] started [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - sub start ref count to 0 [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - export start [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:52:56] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - import connect [2024-04-29 03:52:57] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - add connect ref count to 1 [2024-04-29 03:52:57] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] Connect to observer [2024-04-29 03:52:57] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -prootPWD123 [2024-04-29 03:53:00] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] -- connect 127.0.0.1 -P2881 -uroot -p [2024-04-29 03:53:00] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - sub connect ref count to 0 [2024-04-29 03:53:00] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - export connect [2024-04-29 03:53:00] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-3.1.1-c8d49673b8b01056ab1d6abc9d26dd6987d95a48 [2024-04-29 03:53:00] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - import display [2024-04-29 03:53:00] [f41d62b4-05db-11ef-91bc-0242ac110002] [DEBUG] - add display ref count to 1 [2024-04-29 03:53:00] [f41d62b4-05db-11ef-91bc-0242ac110002] [INFO] Wait for observer init