observer 服务正常，但是任何查询都超时，包括sys租户的表

hongjack · 2024 年10 月 30 日 14:18

【使用环境】测试环境
【 OB or 其他组件】observer
【使用版本】4.3.1
【问题描述】1-1-1 架构，一个zone 硬盘故障后，登录sys租户，通过2881端口，发现一间巡检相关命令都是超时，理论上一个节点down后，应该不影响服务，这个如何能恢复下？
【复现路径】排查步骤参考：https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000001230231?back=kb
应该是一直在无主选举，但是排查网络通的，现在是能登录数据库服务，但是任何表的查询都是超时，包括系统表，表的数据量很小，可忽略大数据影响超时
【附件及日志】

注意：A.A.A.A:2882为坏的节点

grep 'T1002_.*{id:1002}' ./election.log | grep 'dump message count' | vim -

[2024-10-30 14:09:26.813293] INFO  [ELECT] operator() (election_proposer.cpp:252) [7     50][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=56] dump message count(ls_id="{     id:1002}", self_addr="X.X.X.X:2882", state=
match:[Prepare Request, Prepare Response, Accept Request, Accept Response, C     hange Leader]
"X.X.X.X:2882":send:[(5191), 2, 265291, 134, 1], rec:[(5189), 392, 134, 2     64466, 2], last_send:2024-10-30 14:09:16.896292, last_rec:2024-10-30 14:09:16.895077
"X.X.X.X:2882":send:[(5191), (5069), 265291, 265291, 0], rec:[(5190), (51     88), 265291, 265291, 0], last_send:2024-10-30 14:09:18.907356, last_rec:2024-10-30 1     4:09:18.907356
"A.A.A.A:2882":send:[8, 0, 172755, 0, 0], rec:[0, 0, 0, 0, 0], last_send:     2024-10-29 11:22:00.316997, last_rec:1970-01-01 08:00:00.0)
[2024-10-30 14:09:36.816412] INFO  [ELECT] operator() (election_proposer.cpp:252) [7     50][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=57] dump message count(ls_id="{     id:1002}", self_addr="X.X.X.X:2882", state=
match:[Prepare Request, Prepare Response, Accept Request, Accept Response, C     hange Leader]
"X.X.X.X:2882":send:[(5192), 2, 265291, 134, 1], rec:[(5190), 392, 134, 2     64466, 2], last_send:2024-10-30 14:09:26.906711, last_rec:2024-10-30 14:09:26.911132
"X.X.X.X:2882":send:[(5192), (5070), 265291, 265291, 0], rec:[(5191), (51     89), 265291, 265291, 0], last_send:2024-10-30 14:09:28.918764, last_rec:2024-10-30 1     4:09:28.918764
"A.A.A.A:2882":send:[8, 0, 172755, 0, 0], rec:[0, 0, 0, 0, 0], last_send:     2024-10-29 11:22:00.316997, last_rec:1970-01-01 08:00:00.0)
                                                                         
7716 [2024-10-30 14:09:47.314463] INFO  [ELECT] operator() (election_proposer.cpp:252) [7     50][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=83] dump message count(ls_id="{     id:1002}", self_addr="X.X.X.X:2882", state=
match:[Prepare Request, Prepare Response, Accept Request, Accept Response, C     hange Leader]
 "X.X.X.X:2882":send:[(5194), 2, 265291, 134, 1], rec:[(5192), 392, 134, 2     64466, 2], last_send:2024-10-30 14:09:46.901561, last_rec:2024-10-30 14:09:46.900145
"X.X.X.X:2882":send:[(5194), (5071), 265291, 265291, 0], rec:[(5193), (51     90), 265291, 265291, 0], last_send:2024-10-30 14:09:46.901561, last_rec:2024-10-30 1     4:09:46.901561
"A.A.A.A:2882":send:[8, 0, 172755, 0, 0], rec:[0, 0, 0, 0, 0], last_send:     2024-10-29 11:22:00.316997, last_rec:1970-01-01 08:00:00.0)
7721 [2024-10-30 14:09:57.314108] INFO  [ELECT] operator() (election_proposer.cpp:252) [7     50][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30] dump message count(ls_id="{     id:1002}", self_addr="X.X.X.X:2882", state=
match:[Prepare Request, Prepare Response, Accept Request, Accept Response, C     hange Leader]
"X.X.X.X:2882":send:[(5195), 2, 265291, 134, 1], rec:[(5193), 392, 134, 2     64466, 2], last_send:2024-10-30 14:09:56.919197, last_rec:2024-10-30 14:09:57.19282
"X.X.X.X:2882":send:[(5195), (5072), 265291, 265291, 0], rec:[(5194), (51     91), 265291, 265291, 0], last_send:2024-10-30 14:09:56.919197, last_rec:2024-10-30 1     4:09:56.919197
"A.A.A.A:2882":send:[8, 0, 172755, 0, 0], rec:[0, 0, 0, 0, 0], last_send:     2024-10-29 11:22:00.316997, last_rec:1970-01-01 08:00:00.0)





[**centos@localhost log]$ grep 'pcode.*538[4567]' ./observer.log**
[2024-10-30 14:05:17.100259] WDIAG [RPC] create (ob_poc_rpc_server.cpp:141) [93][pnio1][T0][XXXXXXXXXXXXXXXX] [lt=8][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5384, fly_ts=207372, send_timestamp=1730268316892886, peer="XXXXXXXX:38002", sz=828)

淇铭 · 2024 年10 月 30 日 14:21

可以继续按照这个文档继续排查先把三台机器的observe.log日志发一下还有election.log日志，rootservice.log的日志

hongjack · 2024 年10 月 30 日 14:41

新建文本文档.txt (46.8 KB)
这个是精简后的，第三台硬件坏了，无法获取。另外那个文档后续的步骤是查询系统表相关，这个已经无法查询了，都是超时

淇铭 · 2024 年10 月 30 日 16:20

你按照这个文档继续排查一下看看是什么原因造成的还在无主选举
https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000001230231?back=kb

hongjack · 2024 年10 月 30 日 17:09

有个错误：[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}] 这个是ob自己的schema 没有刷新会导致一直选不上吗？如果线上出现此问题能否手动指定一个leader，优先恢复业务？

如果无法收手动指定leader或者恢复服务，有什么工具可以将block_file 中的数据copy一份到本地么？

[2024-10-30 17:04:50.106731] WDIAG [ELECT] refresh_priority_ (election_impl.cpp:347) [1001][T1002_L0_G2][T1002][YB420A130327-000625ADBE5B0F5C-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={is_inited:true, is_running:true, proposer:{ls_id:{id:1003}, addr:"X.X.X.X8:2882", role:Follower, ballot_number:6202, prepare_success_ballot:6200, lease_interval:4.00s, memberlist_with_states:{member_list:{addr_list:["X.X.X.X8:2882", "X.X.X.X9:2882"], membership_version:{proposal_id:370, config_seq:189}, replica_num:3}, prepare_ok:False, accept_ok_promised_ts:invalid, follower_promise_membership_version:[{proposal_id:370, config_seq:189}, {proposal_id:370, config_seq:189}, {proposal_id:9223372036854775807, config_seq:-1}]}, priority_seed:0x1000, restart_counter:1, last_do_prepare_ts:2024-10-30 17:04:40.101984, self_priority:{priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}, p_election:0x7f15b1539230}, acceptor:{ls_id:{id:1003}, addr:"X.X.X.X8:2882", ballot_number:6201, ballot_of_time_window:6201, lease:{owner:"0.0.0.0:0", lease_end_ts:invalid, ballot_number:6200}, is_time_window_opened:False, vote_reason:, last_time_window_open_ts:2024-10-30 17:04:40.101984, p_election:0x7f15b1539230}, ls_biggest_min_cluster_version_ever_seen:4.3.1.0, priority:{priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}, temporarily_downgrade_priority_info:{downgrade_expire_ts:invalid, interval:0_us, reason:null}})

淇铭 · 2024 年10 月 30 日 17:36

1、你根据租户ID和ls_id日志查询一下这个信息
SELECT gmt_create,svr_ip,svr_port,event,name3,value3 FROM __all_server_event_history WHERE module=“ELECTION” AND value1=$tenant_id AND value2=1002 ORDER BY gmt_create;
2、这三个日志尽量在那两个好的zone上拿出来发一个全的具体看看
observe.log日志发一下还有election.log日志，rootservice.log的日志

淇铭 · 2024 年10 月 30 日 17:58

1、你的primary_zone设置了么？如果设置了可以手动切
查询一下 primary_zone的信息
obclient [oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;

2、暂时不支持这样的拷贝

淇铭 · 2024 年10 月 30 日 18:12

如果还能查询查一下这两个信息
select * from DBA_OB_SERVER_EVENT_HISTORY where module=‘FAILURE_DETECTOR’ limit 10;
select * from DBA_OB_SERVER_EVENT_HISTORY where module like ‘%ELECTION%’ limit 10;

hongjack · 2024 年10 月 30 日 19:08

坏的节点修好后启动加入集群就正常了，不过再坏的时间段内【持续大概8个小时】一直是无主选举，不确定什么原因


select * from DBA_OB_SERVER_EVENT_HISTORY where module='FAILURE_DETECTOR' limit 10;


obclient [oceanbase]> select * from DBA_OB_SERVER_EVENT_HISTORY where module='FAILURE_DETECTOR' limit 10;
+----------------------------+------------+----------+------------------+----------------+----------------+--------+--------------+----------------------+--------------+--------+-------+--------+-------+--------+-------+--------+------------+
| TIMESTAMP                  | SVR_IP     | SVR_PORT | MODULE           | EVENT          | NAME1          | VALUE1 | NAME2        | VALUE2               | NAME3        | VALUE3 | NAME4 | VALUE4 | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO |
+----------------------------+------------+----------+------------------+----------------+----------------+--------+--------------+----------------------+--------------+--------+-------+--------+-------+--------+-------+--------+------------+
| 2024-10-28 11:21:54.335344 | X.X.X.39 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-28 11:22:07.487953 | X.X.X.38 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-28 11:22:13.129623 | X.X.X.38 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-28 11:22:18.862247 | X.X.X.38 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-28 11:22:22.807247 | X.X.X.39 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-28 11:22:31.055770 | X.X.X.39 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-30 17:30:41.498981 | X.X.X.38 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-30 17:30:58.765095 | X.X.X.40 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-30 17:31:36.101141 | X.X.X.40 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
| 2024-10-30 17:33:54.470386 | X.X.X.40 |     2882 | FAILURE_DETECTOR | REMOVE FAILURE | FAILURE_MODULE | SCHEMA | FAILURE_TYPE | SCHEMA NOT REFRESHED | AUTO_RECOVER | False  |       |        |       |        |       |        |            |
+----------------------------+------------+----------+------------------+----------------+----------------+--------+--------------+----------------------+--------------+--------+-------+--------+-------+--------+-------+--------+------------+



select * from DBA_OB_SERVER_EVENT_HISTORY where module like %ELECTION% limit 10;

obclient [oceanbase]> select * from DBA_OB_SERVER_EVENT_HISTORY where module like '%ELECTION%' limit 10;
+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
| TIMESTAMP                  | SVR_IP     | SVR_PORT | MODULE   | EVENT                      | NAME1     | VALUE1 | NAME2 | VALUE2 | NAME3 | VALUE3                                                                                             | NAME4         | VALUE4                     | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO |
+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
| 2024-10-23 17:35:20.190169 | X.X.X.40 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1      | LS_ID | 1      | INFO  | leader lease expired, server|lease:[14:29:32.190|14:29:32.190|14:29:32.190]                        | HAPPENED_TIME | 2024-10-23 17:35:20.189699 |       |        |       |        |            |
| 2024-10-23 17:35:22.377483 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | leader lease expired, server|lease:[17:35:25.874|20:04:00.236|20:04:00.236]                        | HAPPENED_TIME | 2024-10-23 17:35:22.377253 |       |        |       |        |            |
| 2024-10-23 17:35:23.072564 | X.X.X.39 |     2882 | ELECTION | vote                       | TENANT_ID | 1002   | LS_ID | 1001   | INFO  | vote for "X.X.X.39:2882", reason:IP-PORT(priority equal)                                         | HAPPENED_TIME | 2024-10-23 17:35:23.72418  |       |        |       |        |            |
| 2024-10-23 17:35:23.670333 | X.X.X.40 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1      | LS_ID | 1      | INFO  | server|lease:["X.X.X.39:2882"|2024-10-23 17:35:25.106,"X.X.X.40:2882"|2024-10-23 17:35:25.106] | HAPPENED_TIME | 2024-10-23 17:35:21.146669 |       |        |       |        |            |
| 2024-10-23 17:35:24.040318 | X.X.X.40 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.40:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-23 17:35:21.401341 |       |        |       |        |            |
| 2024-10-23 17:35:26.430257 | X.X.X.39 |     2882 | ELECTION | vote                       | TENANT_ID | 1002   | LS_ID | 1      | INFO  | vote for "X.X.X.40:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-23 17:35:23.248208 |       |        |       |        |            |
| 2024-10-23 17:35:26.442465 | X.X.X.40 |     2882 | ELECTION | vote                       | TENANT_ID | 1002   | LS_ID | 1      | INFO  | vote for "X.X.X.40:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-23 17:35:24.657881 |       |        |       |        |            |
| 2024-10-23 17:35:28.384115 | X.X.X.40 |     2882 | ELECTION | lease expired              | TENANT_ID | 1002   | LS_ID | 1001   | INFO  | owner:"X.X.X.38:2882", expired time:17:35:27.372113, ballot number:593                           | HAPPENED_TIME | 2024-10-23 17:35:27.394292 |       |        |       |        |            |
| 2024-10-23 17:35:28.409816 | X.X.X.39 |     2882 | ELECTION | lease expired              | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:17:35:23.641515, ballot number:640                           | HAPPENED_TIME | 2024-10-23 17:35:23.653377 |       |        |       |        |            |
| 2024-10-23 17:35:28.433787 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1002   | LS_ID | 1001   | INFO  | leader lease expired, server|lease:[20:04:00.236|20:04:00.236|20:04:00.236]                        | HAPPENED_TIME | 2024-10-23 17:35:23.400357 |       |        |       |        |            |
+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+

淇铭 · 2024 年10 月 31 日 09:43

这个信息查一下我看看你的primary_zone怎么配置的
SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;

淇铭 · 2024 年10 月 31 日 10:07

把这个信息在查一下
select * from DBA_OB_SERVER_EVENT_HISTORY where VALUE1=‘1’ LS_ID=‘1’ and module like ‘%ELECTION%’ limit 20;

select * from DBA_OB_SERVER_EVENT_HISTORY where VALUE1=‘1001’ LS_ID=‘1’ and module like ‘%ELECTION%’ limit 20;

show parameters like '%rootservice_list%';

SELECT * FROM oceanbase.DBA_OB_SERVERS;

hongjack · 2024 年10 月 31 日 11:07

这个查询耗时不正常，把昨天回复的那个节点我down掉后，想模拟下，这个是结果

obclient [oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;

+-----------+-------------+-------------+-------------------+---------------------------------------------+

| TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE      | LOCALITY                                    |

+-----------+-------------+-------------+-------------------+---------------------------------------------+

|         1 | sys         | SYS         | RANDOM            | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |

|      1001 | META$1002   | META        | zone1,zone2,zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |

|      1002 | epp_tenant  | USER        | zone1,zone2,zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |

+-----------+-------------+-------------+-------------------+---------------------------------------------+

3 rows in set (51.101 sec)

淇铭 · 2024 年10 月 31 日 11:13

楼上的信息也能查询一下么这样可以分析看看

hongjack · 2024 年10 月 31 日 11:15

这个是leader一直没有连任成功，一直在重新选举？

obclient [oceanbase]> select * from DBA_OB_SERVER_EVENT_HISTORY where VALUE1='1' and  VALUE2='1' and module like '%ELECTION%' limit 20;

+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+

| TIMESTAMP                  | SVR_IP     | SVR_PORT | MODULE   | EVENT                      | NAME1     | VALUE1 | NAME2 | VALUE2 | NAME3 | VALUE3                                                                                             | NAME4         | VALUE4                     | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO |

+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+

| 2024-10-24 21:35:30.816399 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-24 21:35:30.816246 |       |        |       |        |            |

| 2024-10-24 21:35:40.499329 | X.X.X.40 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1      | LS_ID | 1      | INFO  | leader lease expired, server|lease:[14:29:32.190|14:29:32.190|21:35:37.685]                        | HAPPENED_TIME | 2024-10-24 21:35:34.187270 |       |        |       |        |            |

| 2024-10-24 21:35:43.119671 | X.X.X.40 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1      | LS_ID | 1      | INFO  | server|lease:["X.X.X.38:2882"|2024-10-24 21:35:37.302,"X.X.X.39:2882"|2024-10-24 21:35:37.302] | HAPPENED_TIME | 2024-10-24 21:35:35.485144 |       |        |       |        |            |

| 2024-10-25 02:03:58.408631 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.39:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-25 02:03:58.408551 |       |        |       |        |            |

| 2024-10-25 02:05:06.307963 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.40:2882", expired time:02:04:22.693794, ballot number:700                           | HAPPENED_TIME | 2024-10-25 02:04:22.712211 |       |        |       |        |            |

| 2024-10-25 02:05:10.963686 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-25 02:04:22.712374 |       |        |       |        |            |

| 2024-10-25 11:35:57.213392 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-25 11:35:54.924154 |       |        |       |        |            |

| 2024-10-25 12:21:59.376369 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.40:2882", expired time:12:21:59.356872, ballot number:703                           | HAPPENED_TIME | 2024-10-25 12:21:59.375797 |       |        |       |        |            |

| 2024-10-25 12:22:00.095275 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.39:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-25 12:21:59.375936 |       |        |       |        |            |

| 2024-10-25 18:35:50.431754 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.40:2882", expired time:18:35:50.398218, ballot number:704                           | HAPPENED_TIME | 2024-10-25 18:35:50.431314 |       |        |       |        |            |

| 2024-10-25 18:35:54.793278 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:IP-PORT(priority equal)                                         | HAPPENED_TIME | 2024-10-25 18:35:50.431533 |       |        |       |        |            |

| 2024-10-26 00:01:36.688838 | X.X.X.40 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1      | LS_ID | 1      | INFO  | leader lease expired, server|lease:[14:29:32.190|14:29:32.190|14:29:32.190]                        | HAPPENED_TIME | 2024-10-26 00:01:36.688702 |       |        |       |        |            |

| 2024-10-26 00:01:48.984860 | X.X.X.40 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.40:2882", expired time:00:01:38.684204, ballot number:707                           | HAPPENED_TIME | 2024-10-26 00:01:38.697818 |       |        |       |        |            |

| 2024-10-26 00:01:49.197314 | X.X.X.40 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.40:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-26 00:01:47.103168 |       |        |       |        |            |

| 2024-10-26 00:01:49.297562 | X.X.X.40 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1      | LS_ID | 1      | INFO  | server|lease:["X.X.X.38:2882"|2024-10-26 00:01:51.202,"X.X.X.40:2882"|2024-10-26 00:01:51.202] | HAPPENED_TIME | 2024-10-26 00:01:47.290928 |       |        |       |        |            |

| 2024-10-26 00:01:49.714876 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.40:2882", expired time:00:01:30.684322, ballot number:706                           | HAPPENED_TIME | 2024-10-26 00:01:30.703605 |       |        |       |        |            |

| 2024-10-26 00:01:50.068489 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-26 00:01:30.703706 |       |        |       |        |            |

| 2024-10-26 00:01:51.798677 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-26 00:01:37.83421  |       |        |       |        |            |

| 2024-10-26 00:01:54.143849 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.40:2882", expired time:00:01:40.959722, ballot number:707                           | HAPPENED_TIME | 2024-10-26 00:01:41.87561  |       |        |       |        |            |

| 2024-10-26 00:01:55.265094 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.40:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-26 00:01:47.133898 |       |        |       |        |            |

+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+

20 rows in set (0.028 sec)

obclient [oceanbase]> select * from DBA_OB_SERVER_EVENT_HISTORY where VALUE1='1001' and  VALUE2='1' and module like '%ELECTION%' limit 20;
+----------------------------+------------+----------+----------+-------------------------+-----------+--------+-------+--------+-------+-----------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
| TIMESTAMP                  | SVR_IP     | SVR_PORT | MODULE   | EVENT                   | NAME1     | VALUE1 | NAME2 | VALUE2 | NAME3 | VALUE3                                                                      | NAME4         | VALUE4                     | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO |
+----------------------------+------------+----------+----------+-------------------------+-----------+--------+-------+--------+-------+-----------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
| 2024-10-24 14:35:50.157507 | X.X.X.40 |     2882 | ELECTION | lease expired           | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:14:35:50.138114, ballot number:688    | HAPPENED_TIME | 2024-10-24 14:35:50.157415 |       |        |       |        |            |
| 2024-10-25 02:04:12.975364 | X.X.X.39 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.39:2882", reason:the only request                         | HAPPENED_TIME | 2024-10-25 02:04:11.109365 |       |        |       |        |            |
| 2024-10-25 02:04:14.337317 | X.X.X.39 |     2882 | ELECTION | lease expired           | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:02:04:14.25429, ballot number:693     | HAPPENED_TIME | 2024-10-25 02:04:14.108368 |       |        |       |        |            |
| 2024-10-25 02:04:14.440947 | X.X.X.40 |     2882 | ELECTION | lease expired           | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:02:04:05.296318, ballot number:693    | HAPPENED_TIME | 2024-10-25 02:04:05.320878 |       |        |       |        |            |
| 2024-10-25 02:04:16.965609 | X.X.X.40 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.40:2882", reason:the only request                         | HAPPENED_TIME | 2024-10-25 02:04:05.320984 |       |        |       |        |            |
| 2024-10-25 02:04:17.454128 | X.X.X.39 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                   | HAPPENED_TIME | 2024-10-25 02:04:16.382057 |       |        |       |        |            |
| 2024-10-25 02:04:19.440689 | X.X.X.40 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.39:2882", reason:IP-PORT(priority equal)                  | HAPPENED_TIME | 2024-10-25 02:04:16.315116 |       |        |       |        |            |
| 2024-10-25 02:04:21.532295 | X.X.X.40 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                   | HAPPENED_TIME | 2024-10-25 02:04:21.531806 |       |        |       |        |            |
| 2024-10-25 02:04:21.536072 | X.X.X.39 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                   | HAPPENED_TIME | 2024-10-25 02:04:21.535615 |       |        |       |        |            |
| 2024-10-25 02:04:26.370849 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke | TENANT_ID | 1001   | LS_ID | 1      | INFO  | leader lease expired, server|lease:[20:04:00.236|20:04:00.236|20:04:00.236] | HAPPENED_TIME | 2024-10-25 02:04:07.378937 |       |        |       |        |            |
| 2024-10-25 02:04:26.711989 | X.X.X.40 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                   | HAPPENED_TIME | 2024-10-25 02:04:26.711687 |       |        |       |        |            |
| 2024-10-25 02:04:26.718372 | X.X.X.39 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                   | HAPPENED_TIME | 2024-10-25 02:04:26.717939 |       |        |       |        |            |
| 2024-10-25 02:04:35.005163 | X.X.X.40 |     2882 | ELECTION | lease expired           | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:02:04:34.983766, ballot number:697    | HAPPENED_TIME | 2024-10-25 02:04:35.4677   |       |        |       |        |            |
| 2024-10-25 02:04:35.063446 | X.X.X.40 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.39:2882", reason:IP-PORT(priority equal)                  | HAPPENED_TIME | 2024-10-25 02:04:35.4808   |       |        |       |        |            |
| 2024-10-25 02:04:35.562381 | X.X.X.38 |     2882 | ELECTION | lease expired           | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:02:04:09.380016, ballot number:693    | HAPPENED_TIME | 2024-10-25 02:04:09.391390 |       |        |       |        |            |
| 2024-10-25 02:04:38.548485 | X.X.X.39 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.39:2882", reason:IP-PORT(priority equal)                  | HAPPENED_TIME | 2024-10-25 02:04:37.831741 |       |        |       |        |            |
| 2024-10-25 02:04:40.083304 | X.X.X.39 |     2882 | ELECTION | lease expired           | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:02:04:39.872731, ballot number:697    | HAPPENED_TIME | 2024-10-25 02:04:40.82672  |       |        |       |        |            |
| 2024-10-25 02:04:46.009839 | X.X.X.40 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.39:2882", reason:membership_version is higher             | HAPPENED_TIME | 2024-10-25 02:04:46.9537   |       |        |       |        |            |
| 2024-10-25 02:04:46.760679 | X.X.X.39 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                   | HAPPENED_TIME | 2024-10-25 02:04:46.760544 |       |        |       |        |            |
| 2024-10-25 02:04:49.639789 | X.X.X.38 |     2882 | ELECTION | vote                    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                   | HAPPENED_TIME | 2024-10-25 02:04:16.426370 |       |        |       |        |            |
+----------------------------+------------+----------+----------+-------------------------+-----------+--------+-------+--------+-------+-----------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
20 rows in set (1.934 sec)


obclient [oceanbase]> show parameters like '%rootservice_list%';
+-------+----------+------------+----------+------------------+-----------+-------------------------------------------+------------------------------------------------------------------------------+----------+---------+---------+-------------------+---------------+-----------+
| zone  | svr_type | svr_ip     | svr_port | name             | data_type | value                                     | info                                                                         | section  | scope   | source  | edit_level        | default_value | isdefault |
+-------+----------+------------+----------+------------------+-----------+-------------------------------------------+------------------------------------------------------------------------------+----------+---------+---------+-------------------+---------------+-----------+
| zone1 | observer | X.X.X.38 |     2882 | rootservice_list | STR_LIST  | X.X.X.38:2882:2881;X.X.X.39:2882:2881 | a list of servers against which election candidate is checked for validation | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |               |         0 |
| zone2 | observer | X.X.X.39 |     2882 | rootservice_list | STR_LIST  | X.X.X.38:2882:2881;X.X.X.39:2882:2881 | a list of servers against which election candidate is checked for validation | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |               |         0 |
+-------+----------+------------+----------+------------------+-----------+-------------------------------------------+------------------------------------------------------------------------------+----------+---------+---------+-------------------+---------------+-----------+
2 rows in set (5.653 sec)


obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_SERVERS;

+------------+----------+----+-------+----------+-----------------+----------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+----------------------------+
| SVR_IP     | SVR_PORT | ID | ZONE  | SQL_PORT | WITH_ROOTSERVER | STATUS   | START_SERVICE_TIME         | STOP_TIME | BLOCK_MIGRATE_IN_TIME | CREATE_TIME                | MODIFY_TIME                | BUILD_VERSION                                                                             | LAST_OFFLINE_TIME          |
+------------+----------+----+-------+----------+-----------------+----------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+----------------------------+
| X.X.X.38 |     2882 |  1 | zone1 |     2881 | YES             | ACTIVE   | 2024-10-30 15:09:46.645829 | NULL      | NULL                  | 2024-10-15 16:29:05.242650 | 2024-10-31 03:48:56.758247 | 4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 17:21:43) | NULL                       |
| X.X.X.39 |     2882 |  2 | zone2 |     2881 | NO              | ACTIVE   | 2024-10-30 17:45:52.110318 | NULL      | NULL                  | 2024-10-15 16:29:05.259115 | 2024-10-31 03:48:56.758247 | 4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 17:21:43) | NULL                       |
| X.X.X.40 |     2882 |  3 | zone3 |     2881 | NO              | INACTIVE | NULL                       | NULL      | NULL                  | 2024-10-15 16:29:07.513653 | 2024-10-30 19:29:18.536686 | 4.3.1.0_100000032024051615-bad90e897a7f0f56b8ce5c43e186aa8f4bab03c4(May 16 2024 17:21:43) | 2024-10-30 19:29:18.534228 |
+------------+----------+----+-------+----------+-----------------+----------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+----------------------------+
3 rows in set (17.245 sec)

hongjack · 2024 年10 月 31 日 11:19

这个是时钟diff

[root@localhost /]# clockdiff X.X.X.39
...
host=X.X.X.39 rtt=421(315)ms/0ms delta=1ms/1ms Thu Oct 31 11:17:55 2024
[root@localhost /]# clockdiff X.X.X.40
........
host=X.X.X.40 rtt=99(198)ms/0ms delta=3ms/3ms Thu Oct 31 11:18:13 2024
[root@localhost /]# clockdiff X.X.X.40
......
host=X.X.X.40 rtt=178(265)ms/0ms delta=4ms/4ms Thu Oct 31 11:18:22 2024
[root@localhost /]#

hongjack · 2024 年10 月 31 日 11:26

按照时间倒排的今天的日志

obclient [oceanbase]> select * from DBA_OB_SERVER_EVENT_HISTORY where VALUE1='1' and  VALUE2='1' and module like '%ELECTION%' order by timestamp  desc limit 20;


+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
| TIMESTAMP                  | SVR_IP     | SVR_PORT | MODULE   | EVENT                      | NAME1     | VALUE1 | NAME2 | VALUE2 | NAME3 | VALUE3                                                                                             | NAME4         | VALUE4                     | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO |
+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
| 2024-10-31 11:24:20.806439 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1      | LS_ID | 1      | INFO  | leader lease expired, server|lease:[11:23:58.349|11:23:54.843|11:19:22.433]                        | HAPPENED_TIME | 2024-10-31 11:23:54.848417 |       |        |       |        |            |
| 2024-10-31 11:23:51.084498 | X.X.X.39 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:18:51.873155, ballot number:9913                          | HAPPENED_TIME | 2024-10-31 11:18:51.887909 |       |        |       |        |            |
| 2024-10-31 11:23:47.776837 | X.X.X.39 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-31 11:18:36.376713 |       |        |       |        |            |
| 2024-10-31 11:23:47.700071 | X.X.X.39 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:18:36.350312, ballot number:9912                          | HAPPENED_TIME | 2024-10-31 11:18:36.376617 |       |        |       |        |            |
| 2024-10-31 11:23:46.987401 | X.X.X.38 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1      | LS_ID | 1      | INFO  | server|lease:["X.X.X.38:2882"|2024-10-31 11:23:48.365,"X.X.X.39:2882"|2024-10-31 11:23:48.365] | HAPPENED_TIME | 2024-10-31 11:23:44.475605 |       |        |       |        |            |
| 2024-10-31 11:23:46.232246 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-31 11:23:43.367391 |       |        |       |        |            |
| 2024-10-31 11:23:46.018677 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:23:43.343843, ballot number:9937                          | HAPPENED_TIME | 2024-10-31 11:23:43.366987 |       |        |       |        |            |
| 2024-10-31 11:23:45.719031 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1      | LS_ID | 1      | INFO  | leader lease expired, server|lease:[11:23:43.343|11:23:39.846|11:19:22.433]                        | HAPPENED_TIME | 2024-10-31 11:23:39.850673 |       |        |       |        |            |
| 2024-10-31 11:23:35.667354 | X.X.X.38 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1      | LS_ID | 1      | INFO  | server|lease:["X.X.X.38:2882"|2024-10-31 11:23:35.989,"X.X.X.39:2882"|2024-10-31 11:23:35.989] | HAPPENED_TIME | 2024-10-31 11:23:33.21315  |       |        |       |        |            |
| 2024-10-31 11:23:35.084848 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-31 11:23:27.861251 |       |        |       |        |            |
| 2024-10-31 11:23:35.058927 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:23:27.846860, ballot number:9936                          | HAPPENED_TIME | 2024-10-31 11:23:27.861159 |       |        |       |        |            |
| 2024-10-31 11:23:34.715493 | X.X.X.39 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-31 11:18:20.981266 |       |        |       |        |            |
| 2024-10-31 11:23:34.640692 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1      | LS_ID | 1      | INFO  | leader lease expired, server|lease:[11:23:27.846|11:23:24.343|11:19:22.433]                        | HAPPENED_TIME | 2024-10-31 11:23:24.345434 |       |        |       |        |            |
| 2024-10-31 11:23:34.439181 | X.X.X.38 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1      | LS_ID | 1      | INFO  | server|lease:["X.X.X.38:2882"|2024-10-31 11:23:21.869,"X.X.X.39:2882"|2024-10-31 11:23:21.869] | HAPPENED_TIME | 2024-10-31 11:23:17.973492 |       |        |       |        |            |
| 2024-10-31 11:23:34.433118 | X.X.X.39 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:18:20.971551, ballot number:9911                          | HAPPENED_TIME | 2024-10-31 11:18:20.980855 |       |        |       |        |            |
| 2024-10-31 11:23:34.433002 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-31 11:23:17.868176 |       |        |       |        |            |
| 2024-10-31 11:23:20.427730 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-31 11:23:07.868853 |       |        |       |        |            |
| 2024-10-31 11:23:20.134684 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1      | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-31 11:22:56.868550 |       |        |       |        |            |
| 2024-10-31 11:23:20.008965 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1      | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:22:56.849138, ballot number:9933                          | HAPPENED_TIME | 2024-10-31 11:22:56.868460 |       |        |       |        |            |
| 2024-10-31 11:23:19.878492 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1      | LS_ID | 1      | INFO  | leader lease expired, server|lease:[11:22:56.848|11:22:53.346|11:19:22.433]                        | HAPPENED_TIME | 2024-10-31 11:22:53.348019 |       |        |       |        |            |
+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
20 rows in set (18.720 sec)


obclient [oceanbase]> select * from DBA_OB_SERVER_EVENT_HISTORY where VALUE1='1001' and  VALUE2='1' and module like '%ELECTION%' order by timestamp  desc limit 20;
+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
| TIMESTAMP                  | SVR_IP     | SVR_PORT | MODULE   | EVENT                      | NAME1     | VALUE1 | NAME2 | VALUE2 | NAME3 | VALUE3                                                                                             | NAME4         | VALUE4                     | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO |
+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
| 2024-10-31 11:24:51.651593 | X.X.X.39 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-31 11:19:48.855391 |       |        |       |        |            |
| 2024-10-31 11:24:51.596614 | X.X.X.38 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1001   | LS_ID | 1      | INFO  | server|lease:["X.X.X.38:2882"|2024-10-31 11:24:53.194,"X.X.X.39:2882"|2024-10-31 11:24:53.194] | HAPPENED_TIME | 2024-10-31 11:24:49.284882 |       |        |       |        |            |
| 2024-10-31 11:24:51.550248 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-31 11:24:44.852565 |       |        |       |        |            |
| 2024-10-31 11:24:51.547265 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:24:44.834206, ballot number:9582                          | HAPPENED_TIME | 2024-10-31 11:24:44.852465 |       |        |       |        |            |
| 2024-10-31 11:24:51.019406 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | leader lease expired, server|lease:[11:24:44.833|11:24:40.838|11:19:22.433]                        | HAPPENED_TIME | 2024-10-31 11:24:41.332191 |       |        |       |        |            |
| 2024-10-31 11:24:50.859365 | X.X.X.39 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-31 11:19:38.55087  |       |        |       |        |            |
| 2024-10-31 11:24:50.804359 | X.X.X.39 |     2882 | ELECTION | lease expired              | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:19:38.39333, ballot number:9558                           | HAPPENED_TIME | 2024-10-31 11:19:38.54952  |       |        |       |        |            |
| 2024-10-31 11:24:35.020175 | X.X.X.39 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-31 11:19:18.359734 |       |        |       |        |            |
| 2024-10-31 11:24:34.580190 | X.X.X.38 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1001   | LS_ID | 1      | INFO  | server|lease:["X.X.X.38:2882"|2024-10-31 11:24:33.357,"X.X.X.39:2882"|2024-10-31 11:24:33.357] | HAPPENED_TIME | 2024-10-31 11:24:29.359092 |       |        |       |        |            |
| 2024-10-31 11:24:34.571559 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-31 11:24:29.357084 |       |        |       |        |            |
| 2024-10-31 11:24:34.558705 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:24:29.336561, ballot number:9581                          | HAPPENED_TIME | 2024-10-31 11:24:29.356963 |       |        |       |        |            |
| 2024-10-31 11:24:34.165948 | X.X.X.38 |     2882 | ELECTION | lease expired to revoke    | TENANT_ID | 1001   | LS_ID | 1      | INFO  | leader lease expired, server|lease:[11:24:29.336|11:24:25.338|11:19:22.433]                        | HAPPENED_TIME | 2024-10-31 11:24:25.833686 |       |        |       |        |            |
| 2024-10-31 11:24:34.081738 | X.X.X.39 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-31 11:19:07.360898 |       |        |       |        |            |
| 2024-10-31 11:24:33.842184 | X.X.X.39 |     2882 | ELECTION | lease expired              | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:19:07.337667, ballot number:9556                          | HAPPENED_TIME | 2024-10-31 11:19:07.360363 |       |        |       |        |            |
| 2024-10-31 11:24:33.730464 | X.X.X.38 |     2882 | ELECTION | decentralized to be leader | TENANT_ID | 1001   | LS_ID | 1      | INFO  | server|lease:["X.X.X.38:2882"|2024-10-31 11:24:23.366,"X.X.X.39:2882"|2024-10-31 11:24:23.366] | HAPPENED_TIME | 2024-10-31 11:24:19.787235 |       |        |       |        |            |
| 2024-10-31 11:24:33.127539 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:LOG TS                                                          | HAPPENED_TIME | 2024-10-31 11:24:19.356114 |       |        |       |        |            |
| 2024-10-31 11:24:32.521067 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-31 11:24:09.356929 |       |        |       |        |            |
| 2024-10-31 11:24:21.765826 | X.X.X.39 |     2882 | ELECTION | lease expired              | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:13:46.833540, ballot number:9532                          | HAPPENED_TIME | 2024-10-31 11:13:46.853286 |       |        |       |        |            |
| 2024-10-31 11:24:21.656872 | X.X.X.38 |     2882 | ELECTION | vote                       | TENANT_ID | 1001   | LS_ID | 1      | INFO  | vote for "X.X.X.38:2882", reason:the only request                                                | HAPPENED_TIME | 2024-10-31 11:23:58.353732 |       |        |       |        |            |
| 2024-10-31 11:24:21.609131 | X.X.X.38 |     2882 | ELECTION | lease expired              | TENANT_ID | 1001   | LS_ID | 1      | INFO  | owner:"X.X.X.38:2882", expired time:11:23:58.339533, ballot number:9578                          | HAPPENED_TIME | 2024-10-31 11:23:58.353624 |       |        |       |        |            |
+----------------------------+------------+----------+----------+----------------------------+-----------+--------+-------+--------+-------+----------------------------------------------------------------------------------------------------+---------------+----------------------------+-------+--------+-------+--------+------------+
20 rows in set (0.278 sec)```