obloader大数据导入报错

【产品名称】oceanbase-ce

【产品版本】3.1.2

【问题描述】我启动oceanbase后,执行优化语句:

alter system set memory_limit=‘12G’;

SET GLOBAL ob_trx_timeout=36000000000,GLOBAL ob_query_timeout=36000000000;

set global ob_sql_work_area_percentage=30;

alter system set freeze_trigger_percentage=30;

set global max_allowed_packet=1073741824;

再执行./obdumper -h 127.0.0.1 -P 2881 -u root -p abc123 --sys-password rootPWD123 -t obmysql -D test --csv --table data_pool -f outputs将500W数据导出

再执行./obloader -h 127.0.0.1 -P 2881 -u root -p abc123 --sys-password rootPWD123 -t obmysql -D test2 --csv --table data_pool --external-data -f outputs/data/test/TABLE,执行到400多W数据的时候报错:

2022-01-10 11:30:04 [ERROR] Error occurred when loading data into “test2”.“data_pool” java.sql.SQLNonTransientConnectionException: (conn=-1073479255) Connection is closed

at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:192) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.cmdPrologue(AbstractQueryProtocol.java:1945) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:303) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.ClientSidePreparedStatement.executeInternal(ClientSidePreparedStatement.java:220) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.ClientSidePreparedStatement.execute(ClientSidePreparedStatement.java:149) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.ClientSidePreparedStatement.executeUpdate(ClientSidePreparedStatement.java:181) ~[mariadb-java-client-2.7.4.jar:?]

at com.oceanbase.tools.loaddump.utils.JdbcUtils.executeUpdate(JdbcUtils.java:294) ~[ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.oceanbase.tools.loaddump.dumper.writer.rdb.DatabaseWriter$RetryCallback.call(DatabaseWriter.java:546) ~[ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.oceanbase.tools.loaddump.dumper.writer.rdb.DatabaseWriter$RetryCallback.call(DatabaseWriter.java:435) ~[ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-2.0.0.jar:?]

at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[guava-retrying-2.0.0.jar:?]

at com.oceanbase.tools.loaddump.dumper.writer.rdb.DatabaseWriter.onEvent(DatabaseWriter.java:305) [ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.oceanbase.tools.loaddump.dumper.writer.rdb.DatabaseWriter.onEvent(DatabaseWriter.java:71) [ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [disruptor-3.4.2.jar:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

请问应该如何解决?

日志如附件所示

ob-loader-dumper.zip (59901 KB)

建议:每次导出的时候,换个新的目录或者把原导出目录清空后再导出,因为OBDUMPER 导出的时候如果多次导出不同数据库或表,使用了同一个目录,导出文件会混在一起。同名的文件会被覆盖。在用 OBLOADER 导入的时候,可能会导入非预期内的数据。OBLOADER 导入是解析这个目录结构自动获取要导入的表。所以,如果多次导出的时候,建议每次导出时,导出目录都不要重复。或者导出之前,清空导出目录

所以麻烦根据以上导出的建议重新导出一下,然后再导入试试。

我只导出了一次,只导出了一张表

导出的数据按分区分成了10个文件,一个文件大约100M,我按顺序一个一个执行导入命令,执行到最后两个就会报错:

No memory or reach tenant memory limit

以及

java.sql.SQLNonTransientConnectionException: (conn=-1073478342) Connection is closed

at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:192) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.cmdPrologue(AbstractQueryProtocol.java:1945) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:303) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.ClientSidePreparedStatement.executeInternal(ClientSidePreparedStatement.java:220) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.ClientSidePreparedStatement.execute(ClientSidePreparedStatement.java:149) ~[mariadb-java-client-2.7.4.jar:?]

at org.mariadb.jdbc.ClientSidePreparedStatement.executeUpdate(ClientSidePreparedStatement.java:181) ~[mariadb-java-client-2.7.4.jar:?]

at com.oceanbase.tools.loaddump.utils.JdbcUtils.executeUpdate(JdbcUtils.java:294) ~[ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.oceanbase.tools.loaddump.dumper.writer.rdb.DatabaseWriter$RetryCallback.call(DatabaseWriter.java:546) ~[ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.oceanbase.tools.loaddump.dumper.writer.rdb.DatabaseWriter$RetryCallback.call(DatabaseWriter.java:435) ~[ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-2.0.0.jar:?]

at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[guava-retrying-2.0.0.jar:?]

at com.oceanbase.tools.loaddump.dumper.writer.rdb.DatabaseWriter.onEvent(DatabaseWriter.java:305) [ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.oceanbase.tools.loaddump.dumper.writer.rdb.DatabaseWriter.onEvent(DatabaseWriter.java:71) [ob-loader-dumper-3.0.0-RELEASE.jar:?]

at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [disruptor-3.4.2.jar:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

我感觉oceanbase在内存释放这块应该还是有点问题

内存不足,建议增大租户内存。