OB 由于时间戳问题导致无法重启

vvlzxl · 2025 年5 月 13 日 10:41

【使用环境】生产环境
【 OB or 其他组件】 OB
【使用版本】4.1.2
【问题描述】时钟同步服务器异常导致 OB 挂掉，现在时钟恢复，OB 无法重启

【附件及日志】
以下是分析日志
151.zip (837.4 KB)
以下是启动日志
observer.log.zip (22.2 MB)

辞霜 · 2025 年5 月 13 日 11:24

日志中并未存在启动集群失败相关信息。麻烦重新提供一份。可以使用
cat observer.log* |grep ‘NOTICE’ 看下启动期间日志在哪个时间段

vvlzxl · 2025 年5 月 13 日 16:29

log.zip (36.4 MB)

vvlzxl · 2025 年5 月 13 日 16:36

现在三个节点都无法启动，我怀疑有可能脑裂了，但是不知道怎么解决

辞霜 · 2025 年5 月 13 日 16:41

你参考这里，重启集群后，grep一下关键字NOTICE看看启动ob的相关线程都在哪个observer日志中。
麻烦提供一下相关日志。
上面提供的日志不包含启动集群的线程

vvlzxl · 2025 年5 月 13 日 16:43

我在 151 节点执行没有输出

执行有进程

辞霜 · 2025 年5 月 13 日 16:47

引号是中文的改成英文的

vvlzxl · 2025 年5 月 13 日 16:57

[2025-05-13 16:57:20.368196] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=23] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:57:30.369390] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=10] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:57:40.370352] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=22] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:57:50.371616] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=16] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:58:00.372736] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=18] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:58:10.373832] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=52] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:58:20.374812] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=24] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:58:30.375843] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=16] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:58:40.377111] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=64] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:58:50.378239] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=50] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:49:09.076550] INFO [SERVER] init (ob_server.cpp:256) [55224][observer][T0][Y0-0000000000000000-0-0] [lt=9] [OBSERVER_NOTICE] start to init observer
[2025-05-13 16:49:09.641409] INFO [SERVER] init (ob_service.cpp:214) [55224][observer][T0][Y0-0000000000000000-0-0] [lt=32] [OBSERVICE_NOTICE] init ob_service begin
[2025-05-13 16:49:09.651693] INFO [SERVER] init (ob_service.cpp:254) [55224][observer][T0][Y0-0000000000000000-0-0] [lt=12] [OBSERVICE_NOTICE] init ob_service finish(ret=0, ret=“OB_SUCCESS”, inited=true)
[2025-05-13 16:49:09.761292] INFO [SERVER] init (ob_server.cpp:510) [55224][observer][T0][Y0-0000000000000000-0-0] [lt=58] [OBSERVER_NOTICE] success to init observer(cluster_id=1703243595, lib::g_runtime_enabled=true)
[2025-05-13 16:49:09.761320] INFO [SERVER] start (ob_server.cpp:816) [55224][observer][T0][Y0-0000000000000000-0-0] [lt=14] [OBSERVER_NOTICE] start observer begin
[2025-05-13 16:49:48.223962] INFO [SERVER] start (ob_service.cpp:288) [55224][observer][T0][Y0-0000000000000000-0-0] [lt=75] [OBSERVICE_NOTICE] start ob_service begin
[2025-05-13 16:49:50.252216] INFO [SERVER] start (ob_service.cpp:305) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=38] [OBSERVICE_NOTICE] start ob_service end(ret=0, ret=“OB_SUCCESS”)
[2025-05-13 16:49:50.256750] INFO [SERVER] start (ob_server.cpp:994) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=27] [OBSERVER_NOTICE] server instance start succeed
[2025-05-13 16:51:29.327620] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2950) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=19] [OBSERVER_NOTICE] Refresh user tenant schema successfully (tenant_id=1)
[2025-05-13 16:51:29.327638] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2950) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=14] [OBSERVER_NOTICE] Refresh user tenant schema successfully (tenant_id=1001)
[2025-05-13 16:51:30.327741] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=11] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:51:40.328810] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=25] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:51:50.329936] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=20] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:52:00.331086] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=41] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:52:10.332200] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=24] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:52:20.333267] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=14] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:52:30.334331] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=20] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:52:40.335732] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=17] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:52:50.336574] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=9] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:53:00.337597] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=0] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:53:10.338731] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=121] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:53:20.339703] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=31] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:53:30.340852] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=36] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:53:40.343955] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=16] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:53:50.345098] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=7] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:54:00.346425] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=16] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:54:10.347655] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=38] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:54:20.348857] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=47] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:54:30.349941] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=6] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:54:40.351058] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=48] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:54:50.352219] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=27] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:55:00.353182] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=0] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:55:10.354263] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=38] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:55:20.355253] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=25] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:55:30.356166] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=0] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:55:40.357193] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=23] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:55:50.358197] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=25] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:56:00.359661] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=22] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:56:10.360650] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=0] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:56:20.361581] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=17] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:56:30.362792] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=19] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:56:40.363936] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=25] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:56:50.365257] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=18] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:57:00.366157] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=37] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
[2025-05-13 16:57:10.367280] INFO [SERVER] check_user_tenant_schema_refreshed (ob_server.cpp:2941) [55224][observer][T0][YB42C0A8C797-000635008BFE2979-0-0] [lt=34] [OBSERVER_NOTICE] Refreshing user tenant schema, need to wait (tenant_id=1002)
log.zip (22.8 MB)

vvlzxl · 2025 年5 月 13 日 17:02

这是 rootservice 日志
rootservice.log.zip (9.2 MB)

辞霜 · 2025 年5 月 13 日 17:19

排查下网络是否有问题

obpilot · 2025 年5 月 13 日 17:22

OB里时间是超前了还是落后了？

vvlzxl · 2025 年5 月 13 日 17:24

网络都相互通的

telnet 也能通

vvlzxl · 2025 年5 月 13 日 17:25

一开始时钟同步器吧时间给设置成 2038 年了后来恢复成当前时间，集群就起不来了，现在连不上数据库所有节点都是这样的

辞霜 · 2025 年5 月 13 日 17:25

所有节点都设置为2038年了么

辞霜 · 2025 年5 月 13 日 17:27

我看日志中也存在IO异常报错查询下是否存在坏盘情况看看/var/log/messages日志是否存在异常报错

vvlzxl · 2025 年5 月 13 日 17:29

是的整个机房时钟服务器错乱，把所有都设置成 2038 年，后来恢复了，就再也起不来了

辞霜 · 2025 年5 月 13 日 17:30

你的ob版本是多少，我记得这种情况低ob是不支持恢复了，不清楚新版是否修复这边去咨询下

vvlzxl · 2025 年5 月 13 日 17:32

4.2.1

obpilot · 2025 年5 月 13 日 17:35

如果没有别的原因导致集群起不来，，单就时间错误的话。把OB节点时间都改到2038年，业务停写。把集群拉起来，然后逻辑导出租户数据。

https://mp.weixin.qq.com/s/Neq09njIi_Fq9x63OL2LKQ

辞霜 · 2025 年5 月 13 日 17:38

425bp2版本支持当时钟向前调超过24小时时间后回调后启动ob。421不支持的当前只能参考楼上庆涛老师的方法了导出租户数据重建