当前observer进程还活着但是连接报错:ERROR 8001 (08004): Server is initializing
部分日志:
[2024-10-22 13:43:05.456819] INFO init_config (ob_server.cpp:1829) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4] start tg(lib::TGDefIDs::ServerGTimer=37, tg_name=ServerGTimer)
[2024-10-22 13:43:05.456979] INFO create (ob_timer.cpp:71) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=8] ObTimer create success(this=0x1479e689fd50, thread_id=-1, lbt()=0x1039c10a 0xe80b4eb 0x93f3ef4 0x93ea3a5 0x66ffd04 0x1479e7423b27 0x4ba24af)
[2024-10-22 13:43:05.456991] INFO init_config (ob_server.cpp:1831) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=8] start tg(lib::TGDefIDs::FreezeTimer=38, tg_name=FreezeTimer)
[2024-10-22 13:43:05.457029] INFO create (ob_timer.cpp:71) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=5] ObTimer create success(this=0x1479e689feb0, thread_id=-1, lbt()=0x1039c10a 0xe80b4eb 0x93f3f73 0x93ea3a5 0x66ffd04 0x1479e7423b27 0x4ba24af)
[2024-10-22 13:43:05.457033] INFO init_config (ob_server.cpp:1833) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4] start tg(lib::TGDefIDs::SqlMemTimer=39, tg_name=SqlMemTimer)
[2024-10-22 13:43:05.457069] INFO create (ob_timer.cpp:71) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] ObTimer create success(this=0x1479e6916050, thread_id=-1, lbt()=0x1039c10a 0xe80b4eb 0x93f3ff2 0x93ea3a5 0x66ffd04 0x1479e7423b27 0x4ba24af)
[2024-10-22 13:43:05.457074] INFO init_config (ob_server.cpp:1835) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4] start tg(lib::TGDefIDs::ServerTracerTimer=40, tg_name=ServerTracerTimer)
[2024-10-22 13:43:05.457104] INFO create (ob_timer.cpp:71) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] ObTimer create success(this=0x1479e69161b0, thread_id=-1, lbt()=0x1039c10a 0xe80b4eb 0x93f4071 0x93ea3a5 0x66ffd04 0x1479e7423b27 0x4ba24af)
[2024-10-22 13:43:05.457108] INFO init_config (ob_server.cpp:1837) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] start tg(lib::TGDefIDs::CTASCleanUpTimer=102, tg_name=CTASCleanUpTimer)
[2024-10-22 13:43:05.457220] INFO create (ob_timer.cpp:71) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2] ObTimer create success(this=0x1479e691b6d0, thread_id=-1, lbt()=0x1039c10a 0xe80b4eb 0x93f40f0 0x93ea3a5 0x66ffd04 0x1479e7423b27 0x4ba24af)
[2024-10-22 13:43:05.457224] INFO init_config (ob_server.cpp:1839) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4] start tg(lib::TGDefIDs::MemDumpTimer=113, tg_name=MemDumpTimer)
[2024-10-22 13:43:05.457253] INFO create (ob_timer.cpp:71) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2] ObTimer create success(this=0x1479e6964470, thread_id=-1, lbt()=0x1039c10a 0xe80b4eb 0x93f4176 0x93ea3a5 0x66ffd04 0x1479e7423b27 0x4ba24af)
[2024-10-22 13:43:05.457260] INFO init (ob_config_manager.cpp:68) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4] start tg(lib::TGDefIDs::CONFIG_MGR=119, tg_name=ConfigMgr)
[2024-10-22 13:43:05.457295] INFO create (ob_timer.cpp:71) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] ObTimer create success(this=0x1479e6964ab0, thread_id=-1, lbt()=0x1039c10a 0xe80b4eb 0xe577795 0x93f41d7 0x93ea3a5 0x66ffd04 0x1479e7423b27 0x4ba24af)
[2024-10-22 13:43:05.457306] INFO [LIB] set_param (achunk_mgr.cpp:41) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4] set large page param(large_page_type_=0)
[2024-10-22 13:43:05.458575] INFO register_pm (ob_page_manager.cpp:40) [12970][][T0][Y0-0000000000000000-0-0] [lt=0] register pm finish(ret=0, &pm=0x1479d8057480, pm.get_tid()=12970, tenant_id=500)
[2024-10-22 13:43:05.458608] INFO register_pm (ob_page_manager.cpp:40) [12971][][T0][Y0-0000000000000000-0-0] [lt=0] register pm finish(ret=0, &pm=0x1479d80d5480, pm.get_tid()=12971, tenant_id=500)
[2024-10-22 13:43:05.459540] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12971][][T0][Y0-0000000000000000-0-0] [lt=912][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.459745] INFO run1 (ob_timer.cpp:290) [12971][][T0][Y0-0000000000000000-0-0] [lt=5] timer thread started(this=0x1479e689feb0, tid=12971, lbt()=0x107b3c5c 0x45f968a 0x10a46611 0x10a42c10 0x1479e75c0f1b 0x1479e74f61c0)
[2024-10-22 13:43:05.459752] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12971][FreezeTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.459755] INFO run1 (ob_timer.cpp:302) [12971][FreezeTimer][T0][Y0-0000000000000000-0-0] [lt=2] dump timer info(this=0x1479e689feb0, tasks_num=0, wakeup_time=0)
[2024-10-22 13:43:05.459079] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12970][][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.460346] INFO run1 (ob_timer.cpp:290) [12970][][T0][Y0-0000000000000000-0-0] [lt=917] timer thread started(this=0x1479e689fd50, tid=12970, lbt()=0x107b3c5c 0x45f968a 0x10a46611 0x10a42c10 0x1479e75c0f1b 0x1479e74f61c0)
[2024-10-22 13:43:05.460352] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12970][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.460458] INFO register_pm (ob_page_manager.cpp:40) [12973][][T0][Y0-0000000000000000-0-0] [lt=0] register pm finish(ret=0, &pm=0x1479d81d1480, pm.get_tid()=12973, tenant_id=500)
[2024-10-22 13:43:05.460695] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12973][][T0][Y0-0000000000000000-0-0] [lt=218][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.460827] INFO run1 (ob_timer.cpp:290) [12973][][T0][Y0-0000000000000000-0-0] [lt=4] timer thread started(this=0x1479e69161b0, tid=12973, lbt()=0x107b3c5c 0x45f968a 0x10a46611 0x10a42c10 0x1479e75c0f1b 0x1479e74f61c0)
[2024-10-22 13:43:05.460831] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12973][ServerTracerTim][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.459183] INFO [SERVER.OMT] init (ob_tenant_timezone.cpp:61) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] tenant timezone init(tz_info_map_=0x1479e6980d00, tenant_id_=1)
[2024-10-22 13:43:05.461124] INFO [SERVER.OMT] add_tenant_timezone (ob_tenant_timezone_mgr.cpp:169) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=1938] add tenant timezone success!(tenant_id=1)
[2024-10-22 13:43:05.461130] INFO init (ob_tenant_srs_mgr.cpp:87) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4] start tg(lib::TGDefIDs::SRS_MGR=128, tg_name=SrsMgr)
[2024-10-22 13:43:05.461214] INFO create (ob_timer.cpp:71) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] ObTimer create success(this=0x1479e6965530, thread_id=-1, lbt()=0x1039c10a 0xe80b4eb 0x8d04342 0x93ea50e 0x66ffd04 0x1479e7423b27 0x4ba24af)
[2024-10-22 13:43:05.461231] INFO [SQL] init_sql_factories (ob_sql_init.h:50) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] init sql factories
[2024-10-22 13:43:05.462378] INFO [SQL.ENG] create_hash_table (ob_serializable_function.cpp:147) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] function serialization hash table created(func_cnt=82580, bucket_size=262144, size=6237, conflicts=112)
[2024-10-22 13:43:05.467587] INFO [SHARE] change_initial_value (ob_system_variable.cpp:2929) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=43] succ to change_initial_value(version_comment=“OceanBase_CE 4.2.0.0 (r101000032023091319-600ea4557d9847784acfa3e6b7a7d24c595891c6) (Built Sep 13 2023 20:02:19)”, system_time_zone_str="+08:00", default_coll_int_str=“45”, server_uuid=“82872eff-9038-11ef-8317-000c29663fb3”)
[2024-10-22 13:43:05.467700] WDIAG [OCCAM] init (ob_vtable_event_recycle_buffer.h:58) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4002] invalid argument(ret=-4002, mem_tag=“MdsEventCache”, recycle_buffer_number=0, recycle_buffer_number=0, hash_idx_bkt_num_each=8192)
[2024-10-22 13:43:05.467706] WDIAG [OCCAM] init (ob_mds_event_buffer.h:234) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] init failed(ret=0, ret=“OB_SUCCESS”)
[2024-10-22 13:43:05.467731] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.467916] WDIAG [SERVER.OMT] get_tenant_config_with_lock (ob_tenant_config_mgr.cpp:357) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4201] failed to get fallback tenant config(fallback_tenant_id=1, ret=-4201)
[2024-10-22 13:43:05.467935] INFO set_str (ob_mem_leak_checker.h:131) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] leak mod to check: NONE
[2024-10-22 13:43:05.467953] INFO [SERVER] init_pre_setting (ob_server.cpp:1892) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] Whether record old log file(record_old_log_file=false)
[2024-10-22 13:43:05.467956] INFO [SERVER] init_pre_setting (ob_server.cpp:1894) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2] Whether log warn(log_warn=true)
[2024-10-22 13:43:05.467958] INFO [SERVER] init_pre_setting (ob_server.cpp:1896) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2] init log config(record_old_log_file=false, log_warn=true, enable_async_syslog=true)
[2024-10-22 13:43:05.467961] INFO [SERVER] init_pre_setting (ob_server.cpp:1898) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2] won’t recycle log file
[2024-10-22 13:43:05.468224] INFO [SERVER] init_pre_setting (ob_server.cpp:1932) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2] set limit memory(limit_memory=40802189312)
[2024-10-22 13:43:05.468227] INFO [SERVER] init_pre_setting (ob_server.cpp:1934) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4] REACH SYSLOG RATE LIMIT
[2024-10-22 13:43:05.468542] INFO [STORAGE] init_local_dirs (ob_file_system_router.cpp:161) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=0] succeed to construct local dir(data_dir_="/home/admin/oceanbase/store/ob4test3", slog_dir_="/home/admin/oceanbase/store/ob4test3/slog", clog_dir_="/home/admin/oceanbase/store/ob4test3/clog", sstable_dir_="/home/admin/oceanbase/store/ob4test3/sstable")
[2024-10-22 13:43:05.468564] INFO init (ob_io_struct.cpp:770) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=8] start tg(lib::TGDefIDs::IO_TUNING=120, tg_name=IO_TUNING)
[2024-10-22 13:43:05.468832] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.468986] WDIAG [SERVER.OMT] get_tenant_config_with_lock (ob_tenant_config_mgr.cpp:357) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4201] failed to get fallback tenant config(fallback_tenant_id=1, ret=-4201)
[2024-10-22 13:43:05.469036] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.469054] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.469074] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.469076] WDIAG [STORAGE.TRANS] getClock (ob_clock_generator.h:70) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] clock generator not inited
[2024-10-22 13:43:05.469079] WDIAG [SHARE.SCHEMA] check_inner_stat (ob_server_schema_service.cpp:285) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] inner stat error(schema_service_=NULL, sql_proxy_=NULL, config_=NULL)
[2024-10-22 13:43:05.469092] WDIAG [SHARE.SCHEMA] check_inner_stat (ob_multi_version_schema_service.cpp:1708) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] inner stat error(init_=false)
[2024-10-22 13:43:05.469097] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1981) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4014] inner stat error(ret=-4014)
[2024-10-22 13:43:05.469101] WDIAG [SERVER] nonblock_get_leader (ob_inner_sql_connection.cpp:1736) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4014] user tenant has been dropped(ret=-4014, ret=“OB_INNER_STAT_ERROR”, tenant_id=1)
[2024-10-22 13:43:05.469105] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4014] nonblock get leader failed(ret=-4014, tenant_id=1, ls_id={id:1}, cluster_id=20211224)
[2024-10-22 13:43:05.469113] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:932) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4014] retry_while_no_tenant_resource failed(ret=-4014, tenant_id=1)
[2024-10-22 13:43:05.469116] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1557) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4014] execute_read failed(ret=-4014, cluster_id=20211224, tenant_id=1)
[2024-10-22 13:43:05.469124] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:126) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4014] query failed(ret=-4014, conn=0x1479d5b9e050, start=1729575785469028, sql=select mode, size, latency, iops from all_disk_io_calibration where svr_ip = “192.168.1.130” and svr_port = 2882 and storage_name = “DATA”)
[2024-10-22 13:43:05.469131] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4014] read failed(ret=-4014)
[2024-10-22 13:43:05.469134] WDIAG [COMMON] parse_calibration_table (ob_io_calibration.cpp:824) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4014] query failed(ret=-4014, sql_string=select mode, size, latency, iops from all_disk_io_calibration where svr_ip = “192.168.1.130” and svr_port = 2882 and storage_name = “DATA”)
[2024-10-22 13:43:05.469179] WDIAG [COMMON] read_from_table (ob_io_calibration.cpp:690) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4014] parse calibration data failed(ret=-4014)
[2024-10-22 13:43:05.469193] ERROR issue_dba_error (ob_log.cpp:1804) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-9100, file=“ob_server_log_block_mgr.cpp”, line_no=501, info="prepare_dir_and_create_meta failed")
[2024-10-22 13:43:05.469200] EDIAG [CLOG] do_init (ob_server_log_block_mgr.cpp:501) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-9100] prepare_dir_and_create_meta failed(ret=-9100, log_pool_path="/home/admin/oceanbase/store/ob4test3/clog/log_pool", log_pool_tmp_path="/home/admin/oceanbase/store/ob4test3/clog/log_pool.tmp") BACKTRACE:0x107b3c5c 0x103ab308 0x67284df 0x6727f3f 0x6727e7e 0x6717fe6 0x7edb7d0 0x7ed3416 0x93ec101 0x66ffd04 0x1479e7423b27 0x4ba24af
[2024-10-22 13:43:05.469259] ERROR issue_dba_error (ob_log.cpp:1804) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=59][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-9100, file=“ob_server_log_block_mgr.cpp”, line_no=106, info="do_init failed")
[2024-10-22 13:43:05.469262] EDIAG [CLOG] init (ob_server_log_block_mgr.cpp:106) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-9100] do_init_ failed(ret=-9100, this={dir::"", dir_fd:-1, meta_fd:-1, log_pool_meta:{curr_total_size:0, next_total_size:0, status:0}, min_block_id:0, max_block_id:0, min_log_disk_size_for_all_tenants_:0, is_inited:false}, log_disk_base_path="/home/admin/oceanbase/store/ob4test3/clog") BACKTRACE:0x107b3c5c 0x103ab308 0x67284df 0x6727f3f 0x6727e7e 0x6717fe6 0x7ed3d44 0x7ed300d 0x93ec101 0x66ffd04 0x1479e7423b27 0x4ba24af
[2024-10-22 13:43:05.469287] WDIAG [CLOG] destroy (ob_server_log_block_mgr.cpp:125) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] ObServerLogBlockMgr destroy(this={dir::"", dir_fd:-1, meta_fd:-1, log_pool_meta:{curr_total_size:0, next_total_size:0, status:0}, min_block_id:0, max_block_id:0, min_log_disk_size_for_all_tenants_:0, is_inited:false})
[2024-10-22 13:43:05.469292] ERROR issue_dba_error (ob_log.cpp:1804) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-9100, file=“ob_server.cpp”, line_no=2040, info=“log block mgr init failed”)
[2024-10-22 13:43:05.469295] EDIAG [SERVER] init_io (ob_server.cpp:2040) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-9100] log block mgr init failed(ret=-9100, ret=“OB_NO_SUCH_FILE_OR_DIRECTORY”) BACKTRACE:0x107b3c5c 0x103ab308 0x6727a28 0x6727676 0x6727482 0x670b186 0x941fbb2 0x93ec930 0x66ffd04 0x1479e7423b27 0x4ba24af
[2024-10-22 13:43:05.469317] ERROR issue_dba_error (ob_log.cpp:1804) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-9100, file=“ob_server.cpp”, line_no=288, info=“init io failed”)
[2024-10-22 13:43:05.469319] EDIAG [SERVER] init (ob_server.cpp:288) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-9100] init io failed(ret=-9100, ret=“OB_NO_SUCH_FILE_OR_DIRECTORY”) BACKTRACE:0x107b3c5c 0x103ab308 0x6727a28 0x6727676 0x6727482 0x670b186 0x93f6b4c 0x93ed29e 0x66ffd04 0x1479e7423b27 0x4ba24af
[2024-10-22 13:43:05.469332] INFO [SERVER] set_stop (ob_service.cpp:295) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=9] [OBSERVICE_NOTICE] observice need stop now
[2024-10-22 13:43:05.469337] INFO [SERVER] set_stop (ob_server.cpp:1050) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=3] [OBSERVER_NOTICE] observer is setted to stop
[2024-10-22 13:43:05.469340] INFO [SERVER] destroy (ob_server.cpp:473) [12969][observer][T0][Y0-0000000000000000-0-0] [lt=2] [OBSERVER_NOTICE] destroy observer begin