obdumper导出数据报错一个查询超出memory list 一个 RPC not found

OceanBase 技术问题
#1

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】
【 使用版本 】社区版4.0.0
【问题描述】清晰明确描述问题
【复现路径】问题出现前后相关操作
【问题现象及影响】
2023-03-27 18:32:02 [ERROR] Query data failed. Reason: exceed memory limit. SQL: SELECT l_orderkey,l_partkey,l_suppkey,l_linenumber,l_quantity,l_extendedprice,l_discount,l_tax,l_returnflag,l_linestatus,l_shipdate,l_commitdate,l_receiptdate,l_shipinstruct,l_shipmode,l_comment FROM lineitem partition(p51) WHERE (l_orderkey,l_linenumber)>(?,?) AND (l_orderkey,l_linenumber)<=(?,?). Range: 1536596194,1,1632184742,1.
2023-03-27 18:32:02 [ERROR] Dump test1.lineitem to “/data/ob_dump/data/test1/TABLE/lineitem.30.*.csv” failed java.lang.IllegalStateException: java.sql.SQLTransientConnectionException: exceed memory limit
at com.oceanbase.tools.loaddump.dumper.task.record.RecordDumpTask.executeUnsync(RecordDumpTask.java:511) ~[ob-loader-dumper-4.0.0-RELEASE.jar:?]
at com.oceanbase.tools.loaddump.dumper.task.record.RecordDumpTask.run(RecordDumpTask.java:355) ~[ob-loader-dumper-4.0.0-RELEASE.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_242]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_242]
Caused by: java.sql.SQLTransientConnectionException: exceed memory limit
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:79) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:192) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.readNextValue(SelectResultSet.java:383) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.addStreamingValue(SelectResultSet.java:360) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.nextStreamingValue(SelectResultSet.java:348) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.next(SelectResultSet.java:592) ~[mariadb-java-client-2.7.4.jar:?]
at com.oceanbase.tools.loaddump.dumper.task.record.RecordDumpTask.executeUnsync(RecordDumpTask.java:473) ~[ob-loader-dumper-4.0.0-RELEASE.jar:?]
… 6 more

2023-03-27 18:36:42 [ERROR] Query data failed. Reason: RPC session not found. SQL: SELECT l_orderkey,l_partkey,l_suppkey,l_linenumber,l_quantity,l_extendedprice,l_discount,l_tax,l_returnflag,l_linestatus,l_shipdate,l_commitdate,l_receiptdate,l_shipinstruct,l_shipmode,l_comment FROM lineitem partition(p41) WHERE (l_orderkey,l_linenumber)>(?,?) AND (l_orderkey,l_linenumber)<=(?,?). Range: 4415399717,2,4511627330,2.
2023-03-27 18:36:42 [ERROR] Dump test1.lineitem to “/data/ob_dump/data/test1/TABLE/lineitem.567.*.csv” failed java.lang.IllegalStateException: java.sql.SQLTransientConnectionException: RPC session not found
at com.oceanbase.tools.loaddump.dumper.task.record.RecordDumpTask.executeUnsync(RecordDumpTask.java:511) ~[ob-loader-dumper-4.0.0-RELEASE.jar:?]
at com.oceanbase.tools.loaddump.dumper.task.record.RecordDumpTask.run(RecordDumpTask.java:355) ~[ob-loader-dumper-4.0.0-RELEASE.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_242]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_242]
Caused by: java.sql.SQLTransientConnectionException: RPC session not found
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:79) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:192) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.readNextValue(SelectResultSet.java:383) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.addStreamingValue(SelectResultSet.java:360) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.nextStreamingValue(SelectResultSet.java:348) ~[mariadb-java-client-2.7.4.jar:?]
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.next(SelectResultSet.java:592) ~[mariadb-java-client-2.7.4.jar:?]
at com.oceanbase.tools.loaddump.dumper.task.record.RecordDumpTask.executeUnsync(RecordDumpTask.java:473) ~[ob-loader-dumper-4.0.0-RELEASE.jar:?]
… 6 more

【附件】

image
image1473×166 12.2 KB

#3

导出的数据量级大概多大 多久失败的?

#4

异常信息：RPC session not found. 属于内核错误；

  • 重试导出工具，是否正常导出；
  • 分析一下 observer.log 日志，看一下什么问题导致？
#5

部分导出成功了，部分失败，失败的部分还是报RPC 这个错
[2023-03-27 21:20:44.834280] ERROR [CLOG] pre_check_ (ob_log_replay_service.cpp:772) [27994][T1008_ReplaySrv][T1008][Y0-0000000000000000-0-0] [lt=39] replay has encount fatal error(replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:29516477}, err_ts_:1679912438021022, err_ret_:-4018}, ref_cnt_:34, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:15577, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1679923244831742, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:8863564, ret_code_:0}}, next_to_submit_lsn_:{lsn:32011938}, committed_end_lsn_:{lsn:32011938}, next_to_submit_log_ts_:1679772844356194142, base_lsn_:{lsn:0}, base_log_ts_:1679771280681160944}}, task={type_:2, enqueue_ts_:1679923244830757, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, ret=0) BACKTRACE:0xb553efb 0xb5459d6 0x3c3bfda 0x3c3bcf9 0x3c3bb00 0x3c3b952 0x44cdb75 0x44c6aea 0x44c5c8f 0x3a4f4a1 0xb5348ec 0xb538007 0xb53303a 0x7fa89a6cddf5 0x7fa89a3f748d
[2023-03-27 21:20:45.097925] WARN setup_next_scanner (ob_direct_receive_op.cpp:202) [26041][SerScheQueue1][T1008][YB424C4C233B-0005F7DF22469E22-0-0] [lt=11]
[2023-03-27 21:20:45.124079] WARN resolve_basic_table (ob_dml_resolver.cpp:1526) [27855][T1008_Occam][T1008][YB424C4C233B-0005F7DF1CA6A6EE-0-0] [lt=14] Table ‘oceanbase.__all_freeze_info’ doesn’t exist
[2023-03-27 21:20:45.516832] WARN setup_next_scanner (ob_direct_receive_op.cpp:202) [25813][SqlMemTimer][T1008][YB424C4C233B-0005F7DF20169A5D-0-0] [lt=12]
[2023-03-27 21:20:45.688296] WARN setup_next_scanner (ob_direct_receive_op.cpp:202) [25811][ServerGTimer][T1008][YB424C4C233B-0005F7DF19669E63-0-0] [lt=12]
[2023-03-27 21:20:45.800320] WARN resolve_basic_table (ob_dml_resolver.cpp:1526) [26087][DBMS_JOB_MASTER][T1008][YB424C4C233B-0005F7DF1BE691CF-0-0] [lt=10] Table ‘oceanbase.__all_job’ doesn’t exist
[2023-03-27 21:20:45.803088] WARN resolve_basic_table (ob_dml_resolver.cpp:1526) [26089][DBMS_SCHEDULER][T1008][YB424C4C233B-0005F7DF1BE691CF-0-0] [lt=10] Table ‘oceanbase.__all_tenant_scheduler_job’ doesn’t exist
[2023-03-27 21:20:46.114640] WARN resolve_basic_table (ob_dml_resolver.cpp:1526) [25818][TimezoneMgr][T1008][YB424C4C233B-0005F7DF2036BC05-0-0] [lt=10] Table ‘oceanbase.__all_sys_stat’ doesn’t exist
[2023-03-27 21:20:46.789442] INFO easy_connection_on_timeout_session (easy_connection.c:2128) [26153][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] Session has timed out, session(0x7f7b5c604f10), time(1.000790s), packet_id(1067459412268), pcode(1315), trace_id(YB424C4C233B-0005F7DF2196BD7F), conn(76.76.35.59:53810_76.76.35.67:2882_300_0x7fa6c4072a20 tp=1 t=0-1679923246777647 s=0 r=1 io=33123031312/341741058 sq=341688100).
[2023-03-27 21:20:47.041016] WARN setup_next_scanner (ob_direct_receive_op.cpp:202) [27903][T1008_FreInfoRe][T1008][YB424C4C233B-0005F7DF1C56BD35-0-0] [lt=9]
[2023-03-27 21:20:47.098604] WARN setup_next_scanner (ob_direct_receive_op.cpp:202) [26041][SerScheQueue1][T1008][YB424C4C233B-0005F7DF22469E23-0-0] [lt=13]
[2023-03-27 21:20:47.125142] WARN resolve_basic_table (ob_dml_resolver.cpp:1526) [27855][T1008_Occam][T1008][YB424C4C233B-0005F7DF1CA6A6EF-0-0] [lt=12] Table ‘oceanbase.__all_freeze_info’ doesn’t exist
[2023-03-27 21:20:47.279831] WARN setup_next_scanner (ob_direct_receive_op.cpp:202) [27853][T1008_ReqMemEvi][T1008][YB424C4C233B-0005F7DF2026A046-0-0] [lt=11]
[2023-03-27 21:20:47.279885] WARN easy_connection_do_response (easy_connection.c:1734) [26153][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] Not found session, packet_id(1067459412268), conn(76.76.35.59:53810_76.76.35.67:2882_300_0x7fa6c4072a20 tp=1 t=0-1679923247279873 s=0 r=0 io=33123033285/341744760 sq=341688100)
[2023-03-27 21:20:47.619150] WARN setup_next_scanner (ob_direct_receive_op.cpp:202) [25811][ServerGTimer][T1008][YB424C4C233B-0005F7DF19669E64-0-0] [lt=9]
[2023-03-27 21:20:47.889861] INFO easy_connection_on_timeout_session (easy_connection.c:2128) [26145][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] Session has timed out, session(0x7f7b5c604290), time(1.000111s), packet_id(1067624563070), pcode(1315), trace_id(YB424C4C233B-0005F7DF2196BD80), conn(76.76.35.59:53848_76.76.35.67:2882_382_0x7fa649805560 tp=1 t=0-1679923247872225 s=0 r=1 io=29368255522/342276654 sq=342271198).

#6

收到 有完整的日志吗 麻烦上传下