obclient(ob-ce-4.0.0)长时间不活跃后,再次激活后报错退出

【 使用环境 】测试环境
【 OB or 其他组件 】
【 使用版本 】ob-ce-4.0.0
【问题描述】清晰明确描述问题
【复现路径】问题出现前后相关操作
【问题现象及影响】

【附件】

obclient建立两个连接,一个是通过obproxy,一个是直连observer。
长时间不活跃后,再次输入命令激活报错退出。

  1. 连接obproxy
$ obclient -h127.0.0.1 -P3308 -uroot@sys -A
...
fg 1
obclient -h127.0.0.1 -P3308 -uroot@sys -A
nopager;
PAGER set to stdout
obclient [test]>
7f7561d1c000-7f7561d1d000 rw-p 0000c000 fd:01 22403                      /usr/lib64/libnss_files-2.17.so
7f7561d1d000-7f7561d23000 rw-p 00000000 00:00 0
7f7561d23000-7f7561ee7000 r-xp 00000000 fd:01 3909                       /usr/lib64/libc-2.17.so
7f7561ee7000-7f75620e6000 ---p 001c4000 fd:01 3909                       /usr/lib64/libc-2.17.so
7f75620e6000-7f75620ea000 r--p 001c3000 fd:01 3909                       /usr/lib64/libc-2.17.so
7f75620ea000-7f75620ec000 rw-p 001c7000 fd:01 3909                       /usr/lib64/libc-2.17.so
7f75620ec000-7f75620f1000 rw-p 00000000 00:00 0
7f75620f1000-7f7562106000 r-xp 00000000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f7562106000-7f7562305000 ---p 00015000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f7562305000-7f7562306000 r--p 00014000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f7562306000-7f7562307000 rw-p 00015000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f7562307000-7f7562408000 r-xp 00000000 fd:01 22113                      /usr/lib64/libm-2.17.so
7f7562408000-7f7562607000 ---p 00101000 fd:01 22113                      /usr/lib64/libm-2.17.so
7f7562607000-7f7562608000 r--p 00100000 fd:01 22113                      /usr/lib64/libm-2.17.so
7f7562608000-7f7562609000 rw-p 00101000 fd:01 22113                      /usr/lib64/libm-2.17.so
7f7562609000-7f75626f2000 r-xp 00000000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7f75626f2000-7f75628f2000 ---p 000e9000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7f75628f2000-7f75628fa000 r--p 000e9000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7f75628fa000-7f75628fc000 rw-p 000f1000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7f75628fc000-7f7562911000 rw-p 00000000 00:00 0
7f7562911000-7f7562926000 r-xp 00000000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7f7562926000-7f7562b25000 ---p 00015000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7f7562b25000-7f7562b26000 r--p 00014000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7f7562b26000-7f7562b27000 rw-p 00015000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7f7562b27000-7f7562b29000 r-xp 00000000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7f7562b29000-7f7562d29000 ---p 00002000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7f7562d29000-7f7562d2a000 r--p 00002000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7f7562d2a000-7f7562d2b000 rw-p 00003000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7f7562d2b000-7f7562d42000 r-xp 00000000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7f7562d42000-7f7562f41000 ---p 00017000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7f7562f41000-7f7562f42000 r--p 00016000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7f7562f42000-7f7562f43000 rw-p 00017000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7f7562f43000-7f7562f47000 rw-p 00000000 00:00 0
7f7562f47000-7f7562f69000 r-xp 00000000 fd:01 3499                       /usr/lib64/ld-2.17.so
7f7563154000-7f756315a000 rw-p 00000000 00:00 0
7f756315e000-7f756315f000 rw-p 00000000 00:00 0
7f756315f000-7f7563166000 r--s 00000000 fd:01 4225                       /usr/lib64/gconv/gconv-modules.cache
7f7563166000-7f7563168000 rw-p 00000000 00:00 0
7f7563168000-7f7563169000 r--p 00021000 fd:01 3499                       /usr/lib64/ld-2.17.so
7f7563169000-7f756316a000 rw-p 00022000 fd:01 3499                       /usr/lib64/ld-2.17.so
7f756316a000-7f756316b000 rw-p 00000000 00:00 0
7ffcde210000-7ffcde231000 rw-p 00000000 00:00 0                          [stack]
7ffcde2a4000-7ffcde2a6000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted
  1. 连接observer
$ obclient -hmgr2 -P3306 -uroot@sys -A
...
fg 2
obclient -hmgr2 -P3306 -uroot@sys -A

obclient [test]> \s
*** Error in `obclient': double free or corruption (!prev): 0x000055d933bb4e20 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81299)[0x7f666eede299]
obclient(ob20_end+0x1c)[0x55d9318b9f0c]
obclient(ma_net_end+0x26)[0x55d9318ce826]
obclient(end_server+0x20)[0x55d9318a2860]
obclient(mthd_my_send_cmd+0x13e)[0x55d9318a2f4e]
obclient(mysql_real_query+0x166)[0x55d9318a1f06]
obclient(_Z25mysql_real_query_for_lazyPKcm+0x2a)[0x55d93189407a]
obclient(+0xdc358)[0x55d931895358]
obclient(+0xe10c6)[0x55d93189a0c6]
obclient(main+0x66e)[0x55d93188ec9e]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f666ee7f555]
obclient(+0xd6237)[0x55d93188f237]
======= Memory map: ========
55d9317b9000-55d931dfd000 r-xp 00000000 fd:01 397696                     /root/.oceanbase-all-in-one/obclient/u01/obclient/bin/obclient
55d931ffd000-55d93202c000 r--p 00644000 fd:01 397696                     /root/.oceanbase-all-in-one/obclient/u01/obclient/bin/obclient
55d93202c000-55d9320d5000 rw-p 00673000 fd:01 397696                     /root/.oceanbase-all-in-one/obclient/u01/obclient/bin/obclient
55d9320d5000-55d93210a000 rw-p 00000000 00:00 0
55d933b65000-55d933c37000 rw-p 00000000 00:00 0                          [heap]
7f6664000000-7f6664021000 rw-p 00000000 00:00 0
7f6664021000-7f6668000000 ---p 00000000 00:00 0
7f6668707000-7f666ec4a000 r--p 00000000 fd:01 22038                      /usr/lib/locale/locale-archive
7f666ec4a000-7f666ec56000 r-xp 00000000 fd:01 22403                      /usr/lib64/libnss_files-2.17.so
7f666ec56000-7f666ee55000 ---p 0000c000 fd:01 22403                      /usr/lib64/libnss_files-2.17.so
7f666ee55000-7f666ee56000 r--p 0000b000 fd:01 22403                      /usr/lib64/libnss_files-2.17.so
7f666ee56000-7f666ee57000 rw-p 0000c000 fd:01 22403                      /usr/lib64/libnss_files-2.17.so
7f666ee57000-7f666ee5d000 rw-p 00000000 00:00 0
7f666ee5d000-7f666f021000 r-xp 00000000 fd:01 3909                       /usr/lib64/libc-2.17.so
7f666f021000-7f666f220000 ---p 001c4000 fd:01 3909                       /usr/lib64/libc-2.17.so
7f666f220000-7f666f224000 r--p 001c3000 fd:01 3909                       /usr/lib64/libc-2.17.so
7f666f224000-7f666f226000 rw-p 001c7000 fd:01 3909                       /usr/lib64/libc-2.17.so
7f666f226000-7f666f22b000 rw-p 00000000 00:00 0
7f666f22b000-7f666f240000 r-xp 00000000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f666f240000-7f666f43f000 ---p 00015000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f666f43f000-7f666f440000 r--p 00014000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f666f440000-7f666f441000 rw-p 00015000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f666f441000-7f666f542000 r-xp 00000000 fd:01 22113                      /usr/lib64/libm-2.17.so
7f666f542000-7f666f741000 ---p 00101000 fd:01 22113                      /usr/lib64/libm-2.17.so
7f666f741000-7f666f742000 r--p 00100000 fd:01 22113                      /usr/lib64/libm-2.17.so
7f666f742000-7f666f743000 rw-p 00101000 fd:01 22113                      /usr/lib64/libm-2.17.so
7f666f743000-7f666f82c000 r-xp 00000000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7f666f82c000-7f666fa2c000 ---p 000e9000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7f666fa2c000-7f666fa34000 r--p 000e9000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7f666fa34000-7f666fa36000 rw-p 000f1000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7f666fa36000-7f666fa4b000 rw-p 00000000 00:00 0
7f666fa4b000-7f666fa60000 r-xp 00000000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7f666fa60000-7f666fc5f000 ---p 00015000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7f666fc5f000-7f666fc60000 r--p 00014000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7f666fc60000-7f666fc61000 rw-p 00015000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7f666fc61000-7f666fc63000 r-xp 00000000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7f666fc63000-7f666fe63000 ---p 00002000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7f666fe63000-7f666fe64000 r--p 00002000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7f666fe64000-7f666fe65000 rw-p 00003000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7f666fe65000-7f666fe7c000 r-xp 00000000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7f666fe7c000-7f667007b000 ---p 00017000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7f667007b000-7f667007c000 r--p 00016000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7f667007c000-7f667007d000 rw-p 00017000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7f667007d000-7f6670081000 rw-p 00000000 00:00 0
7f6670081000-7f66700a3000 r-xp 00000000 fd:01 3499                       /usr/lib64/ld-2.17.so
7f667028e000-7f6670294000 rw-p 00000000 00:00 0
7f6670298000-7f6670299000 rw-p 00000000 00:00 0
7f6670299000-7f66702a0000 r--s 00000000 fd:01 4225                       /usr/lib64/gconv/gconv-modules.cache
7f66702a0000-7f66702a2000 rw-p 00000000 00:00 0
7f66702a2000-7f66702a3000 r--p 00021000 fd:01 3499                       /usr/lib64/ld-2.17.so
7f66702a3000-7f66702a4000 rw-p 00022000 fd:01 3499                       /usr/lib64/ld-2.17.so
7f66702a4000-7f66702a5000 rw-p 00000000 00:00 0
7ffd0f5a8000-7ffd0f5c9000 rw-p 00000000 00:00 0                          [stack]
7ffd0f5eb000-7ffd0f5ed000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
7fa4164e7000-7fa4164e8000 rw-p 0000c000 fd:01 22403                      /usr/lib64/libnss_files-2.17.so
7fa4164e8000-7fa4164ee000 rw-p 00000000 00:00 0
7fa4164ee000-7fa4166b2000 r-xp 00000000 fd:01 3909                       /usr/lib64/libc-2.17.so
7fa4166b2000-7fa4168b1000 ---p 001c4000 fd:01 3909                       /usr/lib64/libc-2.17.so
7fa4168b1000-7fa4168b5000 r--p 001c3000 fd:01 3909                       /usr/lib64/libc-2.17.so
7fa4168b5000-7fa4168b7000 rw-p 001c7000 fd:01 3909                       /usr/lib64/libc-2.17.so
7fa4168b7000-7fa4168bc000 rw-p 00000000 00:00 0
7fa4168bc000-7fa4168d1000 r-xp 00000000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fa4168d1000-7fa416ad0000 ---p 00015000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fa416ad0000-7fa416ad1000 r--p 00014000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fa416ad1000-7fa416ad2000 rw-p 00015000 fd:01 19872                      /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fa416ad2000-7fa416bd3000 r-xp 00000000 fd:01 22113                      /usr/lib64/libm-2.17.so
7fa416bd3000-7fa416dd2000 ---p 00101000 fd:01 22113                      /usr/lib64/libm-2.17.so
7fa416dd2000-7fa416dd3000 r--p 00100000 fd:01 22113                      /usr/lib64/libm-2.17.so
7fa416dd3000-7fa416dd4000 rw-p 00101000 fd:01 22113                      /usr/lib64/libm-2.17.so
7fa416dd4000-7fa416ebd000 r-xp 00000000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7fa416ebd000-7fa4170bd000 ---p 000e9000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7fa4170bd000-7fa4170c5000 r--p 000e9000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7fa4170c5000-7fa4170c7000 rw-p 000f1000 fd:01 5035                       /usr/lib64/libstdc++.so.6.0.19
7fa4170c7000-7fa4170dc000 rw-p 00000000 00:00 0
7fa4170dc000-7fa4170f1000 r-xp 00000000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7fa4170f1000-7fa4172f0000 ---p 00015000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7fa4172f0000-7fa4172f1000 r--p 00014000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7fa4172f1000-7fa4172f2000 rw-p 00015000 fd:01 4230                       /usr/lib64/libz.so.1.2.7
7fa4172f2000-7fa4172f4000 r-xp 00000000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7fa4172f4000-7fa4174f4000 ---p 00002000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7fa4174f4000-7fa4174f5000 r--p 00002000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7fa4174f5000-7fa4174f6000 rw-p 00003000 fd:01 22046                      /usr/lib64/libdl-2.17.so
7fa4174f6000-7fa41750d000 r-xp 00000000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7fa41750d000-7fa41770c000 ---p 00017000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7fa41770c000-7fa41770d000 r--p 00016000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7fa41770d000-7fa41770e000 rw-p 00017000 fd:01 3935                       /usr/lib64/libpthread-2.17.so
7fa41770e000-7fa417712000 rw-p 00000000 00:00 0
7fa417712000-7fa417734000 r-xp 00000000 fd:01 3499                       /usr/lib64/ld-2.17.so
7fa41791f000-7fa417925000 rw-p 00000000 00:00 0
7fa417929000-7fa41792a000 rw-p 00000000 00:00 0
7fa41792a000-7fa417931000 r--s 00000000 fd:01 4225                       /usr/lib64/gconv/gconv-modules.cache
7fa417931000-7fa417933000 rw-p 00000000 00:00 0
7fa417933000-7fa417934000 r--p 00021000 fd:01 3499                       /usr/lib64/ld-2.17.so
7fa417934000-7fa417935000 rw-p 00022000 fd:01 3499                       /usr/lib64/ld-2.17.so
7fa417935000-7fa417936000 rw-p 00000000 00:00 0
7fff0e25d000-7fff0e27e000 rw-p 00000000 00:00 0                          [stack]
7fff0e34e000-7fff0e350000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

不活跃的时间大概是多久呢?我这边放几分钟之后再输入命令是没有问题的

具体不记得了,大概过了一个晚上吧

一个晚上那是正常的啊。mysql一样的机制,服务端有一个参数idle_timeout,默认就是8小时。数据库检查超过该时间不活跃的链接,主动断开。
ob的参数名好像不叫这个名。具体的文档查查

查到了,服务器关闭交互式连接前等待活动的秒数。
官方文档说明下下文:

会话空闲超时报错

https://www.oceanbase.com/docs/community-observer-cn-10000000000449523

超时退出没问题,但是退出时打印这一大堆莫名其妙的信息就令人很迷了

嗯嗯,这块确实是有问题的,我这边再尝试复现下

221版本已经修复该问题,建议升级到最新版本:Release V2.2.1 · oceanbase/obclient · GitHub