obd重启集群数十分钟_Over tenant memory limits

【产品名称】oceanbase-ce

【产品版本】3.1.2

【问题描述】两个问题:

1.执行查询报错:Over tenant memory limits

caused by: java.sql.SQLException: Over tenant memory limits at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092) at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1040) at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1347) at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchedInserts(ClientPreparedStatement.java:716) … 46 more

2.随后重启集群,重启数十分钟后失败,再次重启OK,但数据库不能连接

重启集群是,在wait for observer ini \ 等待时间较长,数十分钟,且启动失败

重启日志:

[2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] Traceback (most recent call last): [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “core.py”, line 1439, in restart_cluster [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “core.py”, line 1073, in start_cluster [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “_plugin.py”, line 235, in __call__ [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “_plugin.py”, line 209, in _new_func [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 64, in connect [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] db, cursor = _connect(server.ip, server_config[‘mysql_port’], password if password is not None else ‘’) [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “/home/admin/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 41, in _connect [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] db = mysql.connect(host=ip, user=user, port=int(port), password=str(password), cursorclass=mysql.cursors.DictCursor) [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “/usr/obd/lib/site-packages/pymysql/connections.py”, line 353, in __init__ [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] self.connect() [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “/usr/obd/lib/site-packages/pymysql/connections.py”, line 633, in connect [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] self._request_authentication() [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “/usr/obd/lib/site-packages/pymysql/connections.py”, line 907, in _request_authentication [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] auth_packet = self._read_packet() [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “/usr/obd/lib/site-packages/pymysql/connections.py”, line 725, in _read_packet [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] packet.raise_for_error() [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “/usr/obd/lib/site-packages/pymysql/protocol.py”, line 221, in raise_for_error [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] err.raise_mysql_exception(self._data) [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] File “/usr/obd/lib/site-packages/pymysql/err.py”, line 143, in raise_mysql_exception [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] raise errorclass(errno, errval) [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR] pymysql.err.OperationalError: (1045, “Access denied for user ‘root’@‘xxx.xxx.xxx.xxx’ (using password: YES)”) [2022-01-14 10:41:52] [7cd38a0c-74e3-11ec-a20d-6c4b90ecdaef] [ERROR]

目前数据库重启后obclient登录正常,但odc登录异常。

[admin@obp1 ~]$ obclient -hobp1 -P2883 -uroot@sys oceanbase -c -A -p"my_pwd"
Welcome to the OceanBase. Commands end with ; or \g.
Your MySQL connection id is 1572867
Server version: 5.6.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [oceanbase]> 

odc一直在弹出这个界面,点击重新连接,还是这个界面;点击暂不连接,再次打开连接还会弹出这个界面

基本信息麻烦提供一下;版本号,集群规模;meomory_limit;systme_memory的值;报错租户的内存大小等;

版本号:oceanbase-ce 3.1.2

集群规模:单ob占用磁盘约1.3T,共3个observer

memory_limit: 50G

system_memory: 10G

报错租户的内存大小:30G (max_memory & min_memory均为30G)

租户内存满是内存爆导致转储做不下去导致的;重启后环境已经恢复;