数据库断电后就无法重启了，麻烦大大看看

OceanBase 技术问题
#1

昨天机房断电后，服务器就无法重启了， 查询了相关资料，均无法解决。

[root@localhost store]# obd cluster start lo
Get local repositories ok
Search plugins ok
Load cluster param plugin ok
Open ssh connection ok
Check before start observer ok
[WARN] OBD-1007: (127.0.0.1) The recommended number of open files is 655350 (Current value: 20000)
[WARN] OBD-1007: (127.0.0.1) The recommended number of max user processes is 655350 (Current value: 120000)

Start observer ok
observer program health check ok
obshell program health check ok
Connect to observer x
[ERROR] OBD-1006: Failed to connect to oceanbase-ce
See 错误码及解决方案 - ob-deployer - OceanBase .
Trace ID: da9f8350-9ac9-11f1-920f-000c29fc4053
If you want to view detailed obd logs, please run: obd display-trace da9f8350-9ac9-11f1-920f-000c29fc4053
[2026-08-18 13:54:51.203911] ERROR issue_dba_error (ob_log.cpp:1875) [31104][observer][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_weak_read_service.cpp”, line_no=143, info=“invalid tg id”)
[2026-08-18 13:54:52.481237] ERROR issue_dba_error (ob_log.cpp:1875) [31104][observer][T500][Y0-0000000000000000-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_tenant.cpp”, line_no=769, info=“fail to create tenant module”)
[2026-08-18 13:54:52.481326] ERROR issue_dba_error (ob_log.cpp:1875) [31104][observer][T500][Y0-0000000000000000-0-0] [lt=48][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_multi_tenant.cpp”, line_no=959, info=“init tenant fail”)
[2026-08-18 13:54:52.587761] INFO [CLOG] abort_create_tenant (ob_server_log_block_mgr.cpp:341) [31104][observer][T500][Y0-0000000000000000-0-0] [lt=32] ObServerLogBlockMGR abort_create_tenant success(this={dir::"/root/observer/store/clog/log_pool", dir_fd:14, meta_fd:15, log_pool_meta:{curr_total_size:13958643712, next_total_size:13958643712, status:0}, min_block_id:46928, max_block_id:47111, min_log_disk_size_for_all_tenants_:0, is_inited:true}, log_disk_size=2147483648)
[2026-08-18 13:54:52.587903] ERROR issue_dba_error (ob_log.cpp:1875) [31104][observer][T500][Y0-0000000000000000-0-0] [lt=37][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_server_checkpoint_slog_handler.cpp”, line_no=831, info=“fail to replay create tenant”)
[2026-08-18 13:54:52.588105] ERROR issue_dba_error (ob_log.cpp:1875) [31104][observer][T500][Y0-0000000000000000-0-0] [lt=159][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_server_checkpoint_slog_handler.cpp”, line_no=784, info=“fail to handle tenant create commit”)
[2026-08-18 13:57:09.775887] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:1888) [1978][observer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4014] user tenant has been dropped(ret=-4014, ret=“OB_INNER_STAT_ERROR”, tenant_id=1)
[2026-08-18 13:57:09.970724] INFO [SHARE.SCHEMA] init_sys_table_name_map (ob_schema_struct.cpp:371) [1978][observer][T0][Y0-0000000000000000-0-0] [lt=27] set tenant space table name(key=14220930887511194059, table={database_id:201006, name_case_mode:2, table_name:“ALL_VIRTUAL_TENANT_ERROR_REAL_AGENT”}, strlen=35)
[2026-08-18 13:57:12.293549] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_tenant.cpp”, line_no=913, info=“init mtl module failed”)
[2026-08-18 13:57:12.293924] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_freeze_info_mgr.cpp”, line_no=130, info=“invalid tg id”)
[2026-08-18 13:57:12.294284] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_weak_read_service.cpp”, line_no=138, info=“invalid tg id”)
[2026-08-18 13:57:13.696217] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_freeze_info_mgr.cpp”, line_no=125, info=“invalid tg id”)
[2026-08-18 13:57:13.696492] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_weak_read_service.cpp”, line_no=143, info=“invalid tg id”)
[2026-08-18 13:57:14.628107] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T500][Y0-0000000000000000-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_tenant.cpp”, line_no=769, info=“fail to create tenant module”)
[2026-08-18 13:57:14.628147] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T500][Y0-0000000000000000-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_multi_tenant.cpp”, line_no=959, info=“init tenant fail”)
[2026-08-18 13:57:14.730723] INFO [CLOG] abort_create_tenant (ob_server_log_block_mgr.cpp:341) [1978][observer][T500][Y0-0000000000000000-0-0] [lt=5] ObServerLogBlockMGR abort_create_tenant success(this={dir::"/root/observer/store/clog/log_pool", dir_fd:14, meta_fd:15, log_pool_meta:{curr_total_size:13958643712, next_total_size:13958643712, status:0}, min_block_id:46928, max_block_id:47111, min_log_disk_size_for_all_tenants_:0, is_inited:true}, log_disk_size=2147483648)
[2026-08-18 13:57:14.730762] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T500][Y0-0000000000000000-0-0] [lt=9][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_server_checkpoint_slog_handler.cpp”, line_no=831, info=“fail to replay create tenant”)
[2026-08-18 13:57:14.730867] ERROR issue_dba_error (ob_log.cpp:1875) [1978][observer][T500][Y0-0000000000000000-0-0] [lt=92][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_server_checkpoint_slog_handler.cpp”, line_no=784, info=“fail to handle tenant create commit”)
[2026-08-18 13:58:46.564599] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:1888) [4250][observer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4014] user tenant has been dropped(ret=-4014, ret=“OB_INNER_STAT_ERROR”, tenant_id=1)
[2026-08-18 13:58:46.750388] INFO [SHARE.SCHEMA] init_sys_table_name_map (ob_schema_struct.cpp:371) [4250][observer][T0][Y0-0000000000000000-0-0] [lt=26] set tenant space table name(key=14220930887511194059, table={database_id:201006, name_case_mode:2, table_name:“ALL_VIRTUAL_TENANT_ERROR_REAL_AGENT”}, strlen=35)
[2026-08-18 13:58:49.791745] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_tenant.cpp”, line_no=913, info=“init mtl module failed”)
[2026-08-18 13:58:49.795227] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_freeze_info_mgr.cpp”, line_no=130, info=“invalid tg id”)
[2026-08-18 13:58:49.795951] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_weak_read_service.cpp”, line_no=138, info=“invalid tg id”)
[2026-08-18 13:58:51.203509] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_freeze_info_mgr.cpp”, line_no=125, info=“invalid tg id”)
[2026-08-18 13:58:51.204009] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_tenant_weak_read_service.cpp”, line_no=143, info=“invalid tg id”)
[2026-08-18 13:58:51.901102] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T500][Y0-0000000000000000-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_tenant.cpp”, line_no=769, info=“fail to create tenant module”)
[2026-08-18 13:58:51.901262] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T500][Y0-0000000000000000-0-0] [lt=124][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_multi_tenant.cpp”, line_no=959, info=“init tenant fail”)
[2026-08-18 13:58:52.007043] INFO [CLOG] abort_create_tenant (ob_server_log_block_mgr.cpp:341) [4250][observer][T500][Y0-0000000000000000-0-0] [lt=11] ObServerLogBlockMGR abort_create_tenant success(this={dir::"/root/observer/store/clog/log_pool", dir_fd:14, meta_fd:15, log_pool_meta:{curr_total_size:13958643712, next_total_size:13958643712, status:0}, min_block_id:46928, max_block_id:47111, min_log_disk_size_for_all_tenants_:0, is_inited:true}, log_disk_size=2147483648)
[2026-08-18 13:58:52.007129] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T500][Y0-0000000000000000-0-0] [lt=28][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_server_checkpoint_slog_handler.cpp”, line_no=831, info=“fail to replay create tenant”)
[2026-08-18 13:58:52.007415] ERROR issue_dba_error (ob_log.cpp:1875) [4250][observer][T500][Y0-0000000000000000-0-0] [lt=253][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4070, file=“ob_server_checkpoint_slog_handler.cpp”, line_no=784, info=“fail to handle tenant create commit”)

2 个赞
#3

看着像是文件损坏 导致的启动有问题
把启动的obd.log和启动时候的observer.log日志 完整的发一下 看看

#4

老师，你好！我无法上传文件，麻烦老师开一下权限。

#5

[2026-08-18 14:32:09.352] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - cmd: [‘lo’]
[2026-08-18 14:32:09.353] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2026-08-18 14:32:09.353] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - mkdir /root/.obd/lock/
[2026-08-18 14:32:09.354] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - unknown lock mode
[2026-08-18 14:32:09.355] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - try to get share lock /root/.obd/lock/global
[2026-08-18 14:32:09.355] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - share lock /root/.obd/lock/global, count 1
[2026-08-18 14:32:09.355] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Get Deploy by name
[2026-08-18 14:32:09.355] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - mkdir /root/.obd/cluster/
[2026-08-18 14:32:09.356] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - mkdir /root/.obd/config_parser/
[2026-08-18 14:32:09.357] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_lo
[2026-08-18 14:32:09.357] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - exclusive lock /root/.obd/lock/deploy_lo, count 1
[2026-08-18 14:32:09.373] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Deploy status judge
[2026-08-18 14:32:09.374] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] Get local repositories
[2026-08-18 14:32:09.376] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - mkdir /root/.obd/repository
[2026-08-18 14:32:09.376] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Get local repository oceanbase-ce-4.2.2.0-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.377] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Search repository oceanbase-ce version: 4.2.2.0, tag: aa3053da7370a6685a2ef457cd202d50e5ab75d3, release: None, package_hash: None
[2026-08-18 14:32:09.377] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2026-08-18 14:32:09.378] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 1
[2026-08-18 14:32:09.378] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2026-08-18 14:32:09.385] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Found repository oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.506] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Get deploy config
[2026-08-18 14:32:09.543] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] Search plugins
[2026-08-18 14:32:09.544] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching start_check plugin for components …
[2026-08-18 14:32:09.545] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.545] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - mkdir /root/.obd/plugins
[2026-08-18 14:32:09.548] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.2.0
[2026-08-18 14:32:09.548] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching create_tenant plugin for components …
[2026-08-18 14:32:09.548] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.550] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.2.0
[2026-08-18 14:32:09.550] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching start plugin for components …
[2026-08-18 14:32:09.550] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.552] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.2.0
[2026-08-18 14:32:09.552] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching connect plugin for components …
[2026-08-18 14:32:09.552] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.553] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.2.0 for oceanbase-ce-4.2.2.0
[2026-08-18 14:32:09.553] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching bootstrap plugin for components …
[2026-08-18 14:32:09.554] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.555] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.2.0 for oceanbase-ce-4.2.2.0
[2026-08-18 14:32:09.555] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching display plugin for components …
[2026-08-18 14:32:09.555] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.557] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.2.0
[2026-08-18 14:32:09.676] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] Load cluster param plugin
[2026-08-18 14:32:09.678] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Get local repository oceanbase-ce-4.2.2.0-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:09.678] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Searching param plugin for components …
[2026-08-18 14:32:09.678] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Search param plugin for oceanbase-ce
[2026-08-18 14:32:09.680] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Found for oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.2.0
[2026-08-18 14:32:09.680] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Applying oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:11.456] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] Open ssh connection
[2026-08-18 14:32:11.589] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:11.589] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - import start_check
[2026-08-18 14:32:11.611] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - add start_check ref count to 1
[2026-08-18 14:32:11.612] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] Check before start observer
[2026-08-18 14:32:11.623] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: ls /root/observer/store/clog/tenant_1/
[2026-08-18 14:32:11.630] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.630] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2026-08-18 14:32:11.637] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 1, error output:
[2026-08-18 14:32:11.637] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] cat: /root/observer/run/observer.pid: 没有那个文件或目录
[2026-08-18 14:32:11.638] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG]
[2026-08-18 14:32:11.638] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – 127.0.0.1 port check
[2026-08-18 14:32:11.638] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2026-08-18 14:32:11.650] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.650] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2026-08-18 14:32:11.664] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.664] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B46’ | awk -F’ ’ ‘{print $2}’ | uniq
[2026-08-18 14:32:11.679] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.680] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: ls /root/observer/store/sstable/block_file
[2026-08-18 14:32:11.689] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.690] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: [ -w /tmp/ ] || [ -w /tmp/obshell ]
[2026-08-18 14:32:11.695] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.696] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2026-08-18 14:32:11.707] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.707] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: ulimit -a
[2026-08-18 14:32:11.713] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.716] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [WARNING] OBD-1007: (127.0.0.1) The recommended number of open files is 655350 (Current value: 20000)
[2026-08-18 14:32:11.716] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [WARNING] OBD-1007: (127.0.0.1) The recommended number of max user processes is 655350 (Current value: 120000)
[2026-08-18 14:32:11.717] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: sysctl -a
[2026-08-18 14:32:11.749] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.754] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: cat /proc/meminfo
[2026-08-18 14:32:11.761] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.762] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: df --block-size=1024
[2026-08-18 14:32:11.772] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.774] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – get disk info for path /, total: 289856774144 avail: 108040785920
[2026-08-18 14:32:11.774] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – get disk info for path /dev, total: 16772116480 avail: 16772116480
[2026-08-18 14:32:11.774] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – get disk info for path /dev/shm, total: 16783454208 avail: 16783454208
[2026-08-18 14:32:11.774] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – get disk info for path /run, total: 16783454208 avail: 16774492160
[2026-08-18 14:32:11.775] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 16783454208 avail: 16783454208
[2026-08-18 14:32:11.775] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – get disk info for path /boot, total: 10726932480 avail: 10577571840
[2026-08-18 14:32:11.775] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – get disk info for path /run/user/0, total: 3356692480 avail: 3356692480
[2026-08-18 14:32:11.775] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – disk: {’/’: {‘total’: 289856774144, ‘avail’: 108040785920, ‘need’: 0}, ‘/dev’: {‘total’: 16772116480, ‘avail’: 16772116480, ‘need’: 0}, ‘/dev/shm’: {‘total’: 16783454208, ‘avail’: 16783454208, ‘need’: 0}, ‘/run’: {‘total’: 16783454208, ‘avail’: 16774492160, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 16783454208, ‘avail’: 16783454208, ‘need’: 0}, ‘/boot’: {‘total’: 10726932480, ‘avail’: 10577571840, ‘need’: 0}, ‘/run/user/0’: {‘total’: 3356692480, ‘avail’: 3356692480, ‘need’: 0}}
[2026-08-18 14:32:11.776] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: date +%s%N
[2026-08-18 14:32:11.783] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.784] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – 127.0.0.1 time delta -2.091552734375
[2026-08-18 14:32:11.888] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] e[33m[WARN]e[39m OBD-1007: (127.0.0.1) The recommended number of open files is 655350 (Current value: 20000)
[2026-08-18 14:32:11.888] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] e[33m[WARN]e[39m OBD-1007: (127.0.0.1) The recommended number of max user processes is 655350 (Current value: 120000)
[2026-08-18 14:32:11.889] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO]
[2026-08-18 14:32:11.890] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - sub start_check ref count to 0
[2026-08-18 14:32:11.891] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - export start_check
[2026-08-18 14:32:11.891] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:11.892] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - import start
[2026-08-18 14:32:11.903] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - add start ref count to 1
[2026-08-18 14:32:11.905] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] Start observer
[2026-08-18 14:32:11.906] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: ls /root/observer/store/clog/tenant_1/
[2026-08-18 14:32:11.913] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:11.913] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2026-08-18 14:32:11.921] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 1, error output:
[2026-08-18 14:32:11.921] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] cat: /root/observer/run/observer.pid: 没有那个文件或目录
[2026-08-18 14:32:11.921] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG]
[2026-08-18 14:32:11.921] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – 127.0.0.1 start command construction
[2026-08-18 14:32:11.922] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – starting 127.0.0.1 observer
[2026-08-18 14:32:11.923] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – root@127.0.0.1 set env LD_LIBRARY_PATH to ‘/root/observer/lib:’
[2026-08-18 14:32:11.923] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: cd /root/observer; /root/observer/bin/observer -r ‘127.0.0.1:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -c 1 -d ‘/root/observer/store’ -I ‘127.0.0.1’ -o __min_full_resource_pool_memory=2147483648,memory_limit=‘6G’,system_memory=‘1G’,datafile_size=‘20G’,datafile_next=‘20G’,datafile_maxsize=‘20G’,log_disk_size=‘13G’,cpu_count=16,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4
[2026-08-18 14:32:12.011] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:12.011] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – root@127.0.0.1 delete env LD_LIBRARY_PATH
[2026-08-18 14:32:12.037] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – start_obshell: True
[2026-08-18 14:32:12.037] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: cat /root/observer/run/obshell.pid
[2026-08-18 14:32:12.045] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:12.046] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: ls /proc/2047
[2026-08-18 14:32:12.054] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:12.054] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – 127.0.0.1 obshell[pid: 2047] started
[2026-08-18 14:32:12.055] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] observer program health check
[2026-08-18 14:32:15.059] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – 127.0.0.1 program health check
[2026-08-18 14:32:15.060] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: cat /root/observer/run/observer.pid
[2026-08-18 14:32:15.071] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:15.072] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: ls /proc/11908
[2026-08-18 14:32:15.084] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:15.084] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – 127.0.0.1 observer[pid: 11908] started
[2026-08-18 14:32:15.129] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] obshell program health check
[2026-08-18 14:32:15.131] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: cat /root/observer/run/obshell.pid
[2026-08-18 14:32:15.141] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:15.142] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – local execute: ls /proc/2047
[2026-08-18 14:32:15.152] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – exited code 0
[2026-08-18 14:32:15.153] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – 127.0.0.1 obshell[pid: 2047] started
[2026-08-18 14:32:15.394] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – need_bootstrap: False
[2026-08-18 14:32:15.395] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - sub start ref count to 0
[2026-08-18 14:32:15.395] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - export start
[2026-08-18 14:32:15.396] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - Call oceanbase-ce-py_script_connect-4.2.2.0 for oceanbase-ce-4.2.2.0-100000192024011915.el7-aa3053da7370a6685a2ef457cd202d50e5ab75d3
[2026-08-18 14:32:15.396] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - import connect
[2026-08-18 14:32:15.650] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] - add connect ref count to 1
[2026-08-18 14:32:15.651] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect obshell (127.0.0.1:2886)
[2026-08-18 14:32:15.652] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [INFO] Connect to observer
[2026-08-18 14:32:15.654] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:32:18.675] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:32:21.682] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:32:38.518] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:32:41.558] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:32:44.563] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:32:47.568] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:32:50.572] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:32:53.577] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:32:56.581] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:32:59.586] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:02.590] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:05.596] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:08.601] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:11.607] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:14.612] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:17.618] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:20.623] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:23.630] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:26.635] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:29.640] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:32.644] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:35.649] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:38.653] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:41.657] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:44.662] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:47.667] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:50.671] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:53.675] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:33:56.679] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:33:59.684] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:34:02.689] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:34:05.694] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:34:08.698] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p
[2026-08-18 14:34:11.703] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -pnRsicsp917Jc05RYltiR
[2026-08-18 14:34:14.709] [89f28b3c-9ace-11f1-92dc-000c29fc4053] [DEBUG] – connect 127.0.0.1 -P2881 -uroot -p

#6

基本上认证完 积分应该是够的 临时调整一下 你发个文件 后面自己发帖 赚积分 做活动

#7

Desktop.zip (2.4 MB)

#8

麻烦老师看看，感谢

#9

看着日志信息是sys 租户（tenant_id=1）的 clog 损坏了
the block has been corrupted!!!，路径：

/root/observer/store/clog/tenant_1/1/log
palf_id=1，block_id=46926，min_block_id=46903，max_block_id=46926

应该没有办法修复了 重新部署吧

#10

老师，数据还有方法恢复么？

#11

没有办法了 重新搭建吧 建议要做备份

1 个赞
#12

好的，谢谢

#13

感谢解答08问题！对于14场景，我的经验是重点关注18配置。