【 使用环境 】生产环境
【 OB or 其他组件 】
【 使用版本 】OceanBase_CE 4.2.1.6
【问题描述】凌晨4点到5点数据库连不上
【复现路径】出现过一次，服务器正常，磁盘正常
【附件及日志】
[2024-05-24 01:05:54.136359] INFO New syslog file info: [address: “192.168.1.1:2882”, observer version: OceanBase_CE 4.2.1.6, revision: 106000012024042515-38166dc8c130f88929e5c8150eb4db3aed79b162, sysname: Linux, os release: 3.10.0-1160.71.1.el7.x86_64, machine: x86_64, tz GMT offset: 08:00]
[2024-05-24 04:15:18.863788] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:15:19.246852] ERROR issue_dba_error (ob_log.cpp:1875) [57881][RootBalance][T1][YB42C0A8294F-000618CE9A32170C-0-0] [lt=31][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:16:13.838233] ERROR issue_dba_error (ob_log.cpp:1875) [24849][T1_FrzInfoDet][T1][YB42C0A8294F-000618CE99206A39-0-0] [lt=23][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:16:14.457996] ERROR issue_dba_error (ob_log.cpp:1875) [24849][T1_FrzInfoDet][T1][YB42C0A8294F-000618CE99206A39-0-0] [lt=63][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:16:15.040286] ERROR issue_dba_error (ob_log.cpp:1875) [24849][T1_FrzInfoDet][T1][YB42C0A8294F-000618CE99206A39-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:16:29.053186] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:16:45.031263] ERROR issue_dba_error (ob_log.cpp:1875) [57881][RootBalance][T1][YB42C0A8294F-000618CE9A32176C-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:17:00.973622] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:17:16.213233] ERROR issue_dba_error (ob_log.cpp:1875) [57881][RootBalance][T1][YB42C0A8294F-000618CE9A321784-0-0] [lt=25][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:17:16.479880] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][YB42C0A8294F-000618CE9A10CD5C-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:19:40.942391] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:20:12.384701] ERROR issue_dba_error (ob_log.cpp:1875) [57881][RootBalance][T1][YB42C0A8294F-000618CE9A3217CD-0-0] [lt=33][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:20:12.491182] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:31:11.864003] ERROR issue_dba_error (ob_log.cpp:1875) [57881][RootBalance][T1][YB42C0A8294F-000618CE9A3219F6-0-0] [lt=15][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:31:12.010072] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][YB42C0A8294F-000618CE9A10CE0A-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:33:11.595638] ERROR issue_dba_error (ob_log.cpp:1875) [57881][RootBalance][T1][YB42C0A8294F-000618CE9A321A88-0-0] [lt=14][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:33:35.441904] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
[2024-05-24 04:33:52.034020] ERROR issue_dba_error (ob_log.cpp:1875) [57880][LostRepCheck][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file=“ob_tx_data_functor.cpp”, line_no=391, info=“unexpected io error”)
补充observer.log日志
[2024-05-24 04:23:28.767459] INFO New syslog file info: [address: “192.168.1.2:2882”, observer version: OceanBase_CE 4.2.1.6, revision: 106000012024042515-38166dc8c130f88929e5c8150eb4db3aed79b162, sysname: Linux, os release: 3.10.0-1160.71.1.el7.x86_64, machine: x86_64, tz GMT offset: 08:00]
[2024-05-24 04:23:31.224076] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:31.334967] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:31.456197] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:31.587433] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:31.841503] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:31.872659] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:31.990299] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.060719] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.060841] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.081201] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.333329] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.373717] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.565650] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.837132] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.837200] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.877275] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.917520] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.957686] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:32.998036] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:33.088498] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:33.118693] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:33.118805] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:33.118882] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:33.340091] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:33.340201] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:33.340274] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
[2024-05-24 04:23:33.360140] ERROR issue_dba_error (ob_log.cpp:1875) [18496][GEleTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4024, file=“ob_occam_timer.h”, line_no=224, info=“fail to register next timer task”)
合并时间是什么时候，可以先查一下昨晚合并是否正常
合并时间从2点开始，3点之前合并完成
肯定不正常的，取一份rootservice日志看看
看怎么像合并数据出错，是不是4点到5点合并数据，而且还没成功
合并数据库2点开始，每天合并数据都正常完成，15分钟合并能完成。
有没有可能是clog disk hang event引起的rootservice节点重选？