obd重启集群数十分钟_Over tenant memory limits

【产品名称】oceanbase-ce

【产品版本】3.1.2

【问题描述】两个问题:

1.执行查询报错:Over tenant memory limits

caused by: java.sql.SQLException: Over tenant memory limits
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1040)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1347)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchedInserts(ClientPreparedStatement.java:716)
    ... 46 more

2.随后重启集群,重启数十分钟后失败,再次重启OK,但数据库不能连接

重启集群是,在wait for observer ini \ 等待时间较长,数十分钟,且启动失败

重启日志:

[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] Traceback (most recent call last):
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "core.py", line 1439, in restart_cluster
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "core.py", line 1073, in start_cluster
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "_plugin.py", line 235, in __call__
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "_plugin.py", line 209, in _new_func
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 64, in connect
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]   db, cursor = _connect(server.ip, server_config['mysql_port'], password if password is not None else '')
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 41, in _connect
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]   db = mysql.connect(host=ip, user=user, port=int(port), password=str(password), cursorclass=mysql.cursors.DictCursor)
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "/usr/obd/lib/site-packages/pymysql/connections.py", line 353, in __init__
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]   self.connect()
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "/usr/obd/lib/site-packages/pymysql/connections.py", line 633, in connect
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]   self._request_authentication()
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "/usr/obd/lib/site-packages/pymysql/connections.py", line 907, in _request_authentication
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]   auth_packet = self._read_packet()
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "/usr/obd/lib/site-packages/pymysql/connections.py", line 725, in _read_packet
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]   packet.raise_for_error()
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "/usr/obd/lib/site-packages/pymysql/protocol.py", line 221, in raise_for_error
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]   err.raise_mysql_exception(self._data)
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]  File "/usr/obd/lib/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]   raise errorclass(errno, errval)
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] pymysql.err.OperationalError: (1045, "Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: YES)")
[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] 


目前数据库重启后obclient登录正常,但odc登录异常。

[admin@obp1 ~]$ obclient -hobp1 -P2883 -uroot@sys oceanbase -c -A -p"my_pwd"
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 1572867
Server version: 5.6.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [oceanbase]> 

odc一直在弹出这个界面,点击重新连接,还是这个界面;点击暂不连接,再次打开连接还会弹出这个界面

基本信息麻烦提供一下;版本号,集群规模;meomory_limit;systme_memory的值;报错租户的内存大小等;

版本号:oceanbase-ce 3.1.2

集群规模:单ob占用磁盘约1.3T,共3个observer

memory_limit: 50G

system_memory: 10G

报错租户的内存大小:30G (max_memory & min_memory均为30G)

租户内存满是内存爆导致转储做不下去导致的;重启后环境已经恢复;