想使用 obloader 旁路导入功能提升导入性能，参考官网文档碰到问题，请教一下 OB 技术专家和各位用户有用过的吗。
obloader 官网文档 旁路导入-OceanBase 导数工具-OceanBase文档中心-分布式数据库使用文档 提到 旁路导入功能。
服务端 inner rpc port。获取方式：
* 连接 ODP 服务端时：
* 云数据库 OceanBase 环境下，ODP RPC Port 默认 3307。
* OceanBase 数据库环境下，默认端口 2885；如果需要自定义，可以在启动 ODP 时通过 `-s` 选项进行指定。
* 连接 OBServer 服务端时，sys 租户下查询系统视图 DBA_OB_SERVERS 即可获取 OBServer 的 RPC 端口，默认端口 2882。
这里社区版都是通过 ODP 连接 OB 集群，疑问就是文档里写的 2885 端口是什么意思？ODP 监听只有 2883（连接端口）和2884（监控获取信息的端口）
查看 ODP 的启动参数并没有 -s 这个选项
[admin@server063 obproxy]$ bin/obproxy -h
bin/obproxy -h
----------------------------------------------------------------------------------
obproxy [OPTIONS]
-h,--help print this help
-p,--listen_port LPORT obproxy listen port
-l,--promethues_listen_port PLPORT obproxy prometheus listen port
-o,--optstr OPTSTR extra options string
-n,--appname APPNAME application name
-r,--rs_list RS_LIST root server list(format ip:sql_prot)
-c,--cluster_name CLUSTER_NAME root server cluster name
-d,--dump_config_sql DSQL dump config sql to file
-e,--execute_config_sql ESQL exectue config sql(create tables, insert initial data)
-N,--nodaemon don't run in daemon
-V,--version VERSION current obproxy version
-R,--releaseid RELEASEID current obproxy kernel release id
-t,--regression_test TEST_NAME regression test
example:
run without config server:
./bin/obproxy -p6789 -r'10.125.224.11:26506;10.125.224.22:26577' -n test -o enable_cluster_checkout=false,syslog_level=INFO
OR ./bin/obproxy -p6789 -r'10.125.224.11:26506;10.125.224.22:26577' -c 'ob_test' -n test -o syslog_level=INFO
run with config server:
./bin/obproxy -p6789 -e -n test -o obproxy_config_server_url='your config url',syslog_level=INFO
Non-first start with local config file:
./bin/obproxy
dump config update sql:
./bin/obproxy -d
run regression tests:
./bin/obproxy -p6789 -r10.125.224.11:26506 -ntest -o obproxy_config_server_url='' -t ''
此外，obloader的旁路导入是不是依赖 observer 4.3 版本的 load data local 功能。如果是，文档描述的就不对（文档说是 observer 4.2.1 ）
1 个赞
大川
2024 年7 月 16 日 11:07
#3
您好，rpc是一个通用的协议，在分布式计算中应用较多，建议查看其RFC定义。在ob集群中，直连情况下使用的是默认的svr_port，通过obproxy连接，默认端口是2885。
2 个赞
谢谢回复。
另外，我就是想确认 obloader的这个旁路导入文件功能 是否依赖 observer 的 load data local 功能。我知道它们是两个概念，是想确认这两个功能之间是否有内在联系。
我就没测试成功这个旁路导入功能。
1 个赞
大川
2024 年7 月 16 日 15:28
#5
旁路导入是OB内核的一个能力不依赖于load data，而是load data接入了旁路导入的能力。
2 个赞
恩。 你说的这个比较严谨。 是我表述不当。谢谢。
我找到 OBProxy 新增 rpc-port 的用法了，是 4.3.0 新增功能。之前我提供的信息不对，我用错版本了。
[admin@server065 obproxy]$ netstat -ntlp |grep obproxy
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:2883 0.0.0.0:* LISTEN 41219/obproxy
tcp 0 0 0.0.0.0:2884 0.0.0.0:* LISTEN 41219/obproxy
tcp 0 0 0.0.0.0:2885 0.0.0.0:* LISTEN 41219/obproxy
其实我的主要问题是我想实现文档中 obloader 旁路导入功能用法。文档没有给出明确的示例只好自己尝试。
bin/obloader -h 10.0.0.65 -P 2883 -u tpch -t oboracle -p abcABC123 -c OB4216 --sys-password aaAA11__ -D tpch --table LINEITEM --external-data --csv -f /data/1/tpch/s4/bak/LINEITEM.1 --truncate-table --column-separator='|' --thread 16 --rpc-port=2885 --direct
但是一直报错。
2024-07-16 17:06:51 [ERROR] Fatal error occurred while loading data from "/data/1/tpch/s4/bak/LINEITEM.1" into table "TPCH"."LINEITEM". Reason: Write into
table "LINEITEM" failed. com.alipay.oceanbase.rpc.exception.ObTableServerConnectException: login failed after max 3 tries server [10.0.0.65:2885]
2024-07-16 17:06:51 [INFO] Halt all workers immediately at the end of their current cycle
2024-07-16 17:06:51 [INFO] Shutdown task context immediately finished
2024-07-16 17:06:51 [ERROR] Error: Write into table "LINEITEM" failed. com.alipay.oceanbase.rpc.exception.ObTableServerConnectException: login failed after max 3 tries server [10.0.0.65:2885]
2024-07-16 17:06:51 [ERROR] Error: Write into table "LINEITEM" failed. com.alipay.oceanbase.rpc.exception.ObTableServerConnectException: login failed after max 3 tries server [10.0.0.65:2885]
2024-07-16 17:06:51 [ERROR] Error: Write into table "LINEITEM" failed. com.alipay.oceanbase.rpc.exception.ObTableServerConnectException: login failed after max 3 tries server [10.0.0.65:2885]
2024-07-16 17:06:51 [ERROR] Error: Write into table "LINEITEM" failed. com.alipay.oceanbase.rpc.exception.ObTableServerConnectException: login failed after max 3 tries server [10.0.0.65:2885]
2024-07-16 17:06:51 [ERROR] Error: Write into table "LINEITEM" failed. com.alipay.oceanbase.rpc.exception.ObTableServerConnectException: login failed after max 3 tries server [10.0.0.65:2885]
2024-07-16 17:06:51 [ERROR] Error: Write into table "LINEITEM" failed. com.alipay.oceanbase.rpc.exception.ObTableServerConnectException: login failed after max 3 tries server [10.0.0.65:2885]
2024-07-16 17:06:51 [INFO] ---------- Finished Tasks: 6 Running Tasks: 0 Progress: 100.00% ----------
2024-07-16 17:06:51 [INFO]
obproxy 的 rpc-port 连接需要特别的账户吗？
1 个赞
大川
2024 年7 月 16 日 17:55
#7
这里应该是oracle租户吧，建议-u和-D改成大写后再试试
2 个赞
成功了，谢谢。
官网文档里如果能给出这个旁路导入的命令例子，也能减少初次使用时不必要的猜测。
[root@server063 ob-loader-dumper-4.3.1-RELEASE]# bin/obloader -h 10.0.0.63 -P 2881 -u TPCH -t oboracle -p abcABC123 --sys-password aaAA11__ -D TPCH --table LINEITEM --external-data --csv -f /data/1/tpch/s4/bak/LINEITEM.1 --truncate-
table --column-separator='|' --thread 16 --rpc-port=2882 --direct --parallel=8
2024-07-17 09:57:02 [WARN] The Java version (1.8.0_242) you are currently using is not recommended. Recommended range: [1.8.0_300, 9.0.0)
2024-07-17 09:57:02 [INFO] Parsed args:
[--csv] true
[--file-path] /data/1/tpch/s4/bak/LINEITEM.1
[--column-separator] |
[--thread] 16
[--parallel] 8
[--host] 10.0.0.63
[--port] 2881
[--user] TPCH
[--tenant] oboracle
[--password] ******
[--database] TPCH
[--sys-user] root
[--sys-password] ******
[--table] [LINEITEM]
[--truncate-table] true
[--external-data] true
[--direct] true
[--rpc-port] 2882
2024-07-17 09:57:03 [INFO] Log files will be written to /data/1/tpch/s4/bak/logs
2024-07-17 09:57:03 [INFO] Trying to establish JDBC connection to `TPCH@oboracle`...
2024-07-17 09:57:03 [INFO] Server Mode: OBORACLE-4.2.1.6
2024-07-17 09:57:03 [INFO] Querying table column metadata, this might take a while...
2024-07-17 09:57:03 [INFO] Filtering out empty tables...
2024-07-17 09:57:03 [INFO] Splitting data files into 64 MB logical chunks...
2024-07-17 09:57:04 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM.1" has been split into 6 chunks. Elapsed: 382.7 ms
2024-07-17 09:57:04 [INFO] Split 1 data files to 6 logical chunks success. Elapsed: 393.3 ms
2024-07-17 09:57:04 [INFO] Exec truncate table "TPCH"."LINEITEM" finished. Elapsed: 113.5 ms. Remain: 0
2024-07-17 09:57:09 [INFO] Wait 5000 ms for observer to refresh schema finished
2024-07-17 09:57:09 [INFO] Bootstrap with Max Heap: 3 GB, Safe Heap: 3.14 GB
2024-07-17 09:57:09,237 main WARN No Root logger was configured, creating default ERROR-level Root logger with Console appender
2024-07-17 09:57:09 [INFO] Create 1024 slots for ring buffer finished. [0.0.0.0]
2024-07-17 09:57:09 [INFO] Start 16 database writer threads finished. [0.0.0.0]
2024-07-17 09:57:09 [INFO] Start 16 record file reader threads success
2024-07-17 09:57:09 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-6.1" has been parsed finished
2024-07-17 09:57:14 [INFO]
<...>
2024-07-17 09:57:19 [INFO]
1. Enqueue Performance Monitor:
-------------------------------------------------------------------------------------------------------
Dimension \ Metric | Tps | Throughput | Buffer
-------------------------------------------------------------------------------------------------------
1.sec.avg | 162553.14 Records/sec | 22.99 MB/sec | 950 Slots
1.min.avg | 108491.12 Records/sec | 15.33 MB/sec | 950 Slots
Total | 1641718 Records | 232 MB | 950 Slots
-------------------------------------------------------------------------------------------------------
2. Dequeue Performance Monitor:
-------------------------------------------------------------------------------------------------------
Dimension \ Metric | Tps | Throughput | Buffer
-------------------------------------------------------------------------------------------------------
1.sec.avg | 134697.77 Records/sec | 19.05 MB/sec | 946 Slots
1.min.avg | 52635.28 Records/sec | 7.44 MB/sec | 946 Slots
Total | 1362118 Records | 192 MB | 946 Slots
-------------------------------------------------------------------------------------------------------
2024-07-17 09:57:21 [INFO] ---------- Finished Tasks: 1 Running Tasks: 5 Progress: 80.10% ----------
2024-07-17 09:57:21 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-1.1" has been parsed finished
2024-07-17 09:57:21 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-3.1" has been parsed finished
2024-07-17 09:57:23 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-2.1" has been parsed finished
2024-07-17 09:57:24 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-4.1" has been parsed finished
2024-07-17 09:57:24 [INFO] Commit load task on table "LINEITEM". This might take a while. Please wait...
2024-07-17 09:57:24 [INFO]
1. Enqueue Performance Monitor:
-------------------------------------------------------------------------------------------------------
Dimension \ Metric | Tps | Throughput | Buffer
-------------------------------------------------------------------------------------------------------
1.sec.avg | 177571.1 Records/sec | 25.13 MB/sec | 139 Slots
1.min.avg | 116339.23 Records/sec | 16.45 MB/sec | 139 Slots
Total | 2665953 Records | 377 MB | 139 Slots
-------------------------------------------------------------------------------------------------------
2. Dequeue Performance Monitor:
-------------------------------------------------------------------------------------------------------
Dimension \ Metric | Tps | Throughput | Buffer
-------------------------------------------------------------------------------------------------------
1.sec.avg | 177418.01 Records/sec | 25.11 MB/sec | 5 Slots
1.min.avg | 69290.97 Records/sec | 9.8 MB/sec | 5 Slots
Total | 2665953 Records | 377 MB | 5 Slots
-------------------------------------------------------------------------------------------------------
2024-07-17 09:57:26 [INFO] ---------- Finished Tasks: 5 Running Tasks: 1 Progress: 100.00% ----------
2024-07-17 09:57:32 [INFO] ---------- Finished Tasks: 5 Running Tasks: 1 Progress: 100.00% ----------
2024-07-17 09:57:35 [INFO] ---------- Finished Tasks: 5 Running Tasks: 1 Progress: 100.00% ----------
2024-07-17 09:57:38 [INFO] ---------- Finished Tasks: 5 Running Tasks: 1 Progress: 100.00% ----------
2024-07-17 09:57:38 [INFO] Load task on table "LINEITEM" is committed successfully! Elapsed: 14.54s
2024-07-17 09:57:39 [INFO] [Timer] Table: LINEITEM, Write Elapsed: 11.01s, Commit Elapsed: 14.54s, Total Elapsed: 25.55s
2024-07-17 09:57:39 [INFO] ---------- Finished Tasks: 6 Running Tasks: 0 Progress: 100.00% ----------
2024-07-17 09:57:39 [INFO]
All Load Tasks Finished:
----------------------------------------------------------------------------------------------------------------------------
No.# | Type | Name | Count | Status
----------------------------------------------------------------------------------------------------------------------------
1 | TABLE | LINEITEM | 2665953 -> 2665953 | SUCCESS
----------------------------------------------------------------------------------------------------------------------------
Total Count: 2665953 End Time: 2024-07-17 09:57:39
2024-07-17 09:57:39 [INFO] Load record finished. Total Elapsed: 35.88 s
2024-07-17 09:57:39 [INFO] System exit 0
[root@server063 ob-loader-dumper-4.3.1-RELEASE]#
[root@server063 ob-loader-dumper-4.3.1-RELEASE]# bin/obloader -h 10.0.0.65 -P 2883 -u TPCH -t oboracle -c OB4216 -p abcABC123 --sys-password aaAA11__ -D TPCH --table LINEITEM --external-data --csv -f /data/1/tpch/s4/bak/LINEITEM.1 -
-truncate-table --column-separator='|' --thread 16 --rpc-port=2885 --direct --parallel=8
2024-07-17 10:39:44 [WARN] The Java version (1.8.0_242) you are currently using is not recommended. Recommended range: [1.8.0_300, 9.0.0)
2024-07-17 10:39:44 [INFO] Parsed args:
[--csv] true
[--file-path] /data/1/tpch/s4/bak/LINEITEM.1
[--column-separator] |
[--thread] 16
[--parallel] 8
[--host] 10.0.0.65
[--port] 2883
[--user] TPCH
[--tenant] oboracle
[--cluster] OB4216
[--password] ******
[--database] TPCH
[--sys-user] root
[--sys-password] ******
[--table] [LINEITEM]
[--truncate-table] true
[--external-data] true
[--direct] true
[--rpc-port] 2885
2024-07-17 10:39:45 [INFO] Log files will be written to /data/1/tpch/s4/bak/logs
2024-07-17 10:39:45 [INFO] Trying to establish JDBC connection to `TPCH@oboracle#OB4216`...
2024-07-17 10:39:45 [INFO] Server Mode: OBORACLE-4.2.1.6
2024-07-17 10:39:46 [INFO] Querying table column metadata, this might take a while...
2024-07-17 10:39:46 [INFO] Filtering out empty tables...
2024-07-17 10:39:46 [INFO] Splitting data files into 64 MB logical chunks...
2024-07-17 10:39:46 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM.1" has been split into 6 chunks. Elapsed: 397.1 ms
2024-07-17 10:39:46 [INFO] Split 1 data files to 6 logical chunks success. Elapsed: 407.6 ms
2024-07-17 10:39:46 [INFO] Exec truncate table "TPCH"."LINEITEM" finished. Elapsed: 103.2 ms. Remain: 0
2024-07-17 10:39:51 [INFO] Wait 5000 ms for observer to refresh schema finished
2024-07-17 10:39:51 [INFO] Bootstrap with Max Heap: 3 GB, Safe Heap: 3.14 GB
2024-07-17 10:39:52,034 main WARN No Root logger was configured, creating default ERROR-level Root logger with Console appender
2024-07-17 10:39:51 [INFO] Bootstrap with Max Heap: 3 GB, Safe Heap: 3.14 GB 10:40:03 [48/1865]
2024-07-17 10:39:52,034 main WARN No Root logger was configured, creating default ERROR-level Root logger with Console appender
2024-07-17 10:39:52 [INFO] Create 1024 slots for ring buffer finished. [0.0.0.0]
2024-07-17 10:39:52 [INFO] Start 16 database writer threads finished. [0.0.0.0]
2024-07-17 10:39:52 [INFO] Start 16 record file reader threads success
2024-07-17 10:39:52 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-6.1" has been parsed finished
2024-07-17 10:39:57 [INFO]
<....>
2024-07-17 10:40:02 [INFO]
1. Enqueue Performance Monitor:
-------------------------------------------------------------------------------------------------------
Dimension \ Metric | Tps | Throughput | Buffer
-------------------------------------------------------------------------------------------------------
1.sec.avg | 196343.93 Records/sec | 27.78 MB/sec | 1024 Slots
1.min.avg | 134334.44 Records/sec | 19.0 MB/sec | 1024 Slots
Total | 1966218 Records | 278 MB | 1024 Slots
-------------------------------------------------------------------------------------------------------
2. Dequeue Performance Monitor:
-------------------------------------------------------------------------------------------------------
Dimension \ Metric | Tps | Throughput | Buffer
-------------------------------------------------------------------------------------------------------
1.sec.avg | 156094.56 Records/sec | 22.09 MB/sec | 1024 Slots
1.min.avg | 58999.54 Records/sec | 8.34 MB/sec | 1024 Slots
Total | 1565418 Records | 221 MB | 1024 Slots
-------------------------------------------------------------------------------------------------------
2024-07-17 10:40:03 [INFO] ---------- Finished Tasks: 1 Running Tasks: 5 Progress: 88.85% ----------
2024-07-17 10:40:04 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-5.1" has been parsed finished
2024-07-17 10:40:04 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-4.1" has been parsed finished
2024-07-17 10:40:04 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-1.1" has been parsed finished
2024-07-17 10:40:04 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-3.1" has been parsed finished
2024-07-17 10:40:04 [INFO] File: "/data/1/tpch/s4/bak/LINEITEM-2.1" has been parsed finished
2024-07-17 10:40:05 [INFO] Commit load task on table "LINEITEM". This might take a while. Please wait...
2024-07-17 10:40:07 [INFO]
1. Enqueue Performance Monitor:
-------------------------------------------------------------------------------------------------------
Dimension \ Metric | Tps | Throughput | Buffer
-------------------------------------------------------------------------------------------------------
1.sec.avg | 177577.95 Records/sec | 25.13 MB/sec | 944 Slots
1.min.avg | 134955.9 Records/sec | 19.09 MB/sec | 944 Slots
Total | 2665953 Records | 377 MB | 944 Slots
-------------------------------------------------------------------------------------------------------
2. Dequeue Performance Monitor:
-------------------------------------------------------------------------------------------------------
Dimension \ Metric | Tps | Throughput | Buffer
-------------------------------------------------------------------------------------------------------
1.sec.avg | 177436.85 Records/sec | 25.11 MB/sec | 2 Slots
1.min.avg | 72092.06 Records/sec | 10.2 MB/sec | 2 Slots
Total | 2665953 Records | 377 MB | 2 Slots
-------------------------------------------------------------------------------------------------------
2024-07-17 10:40:08 [INFO] ---------- Finished Tasks: 5 Running Tasks: 1 Progress: 100.00% ----------
2024-07-17 10:40:14 [INFO] ---------- Finished Tasks: 5 Running Tasks: 1 Progress: 100.00% ----------
2024-07-17 10:40:17 [INFO] ---------- Finished Tasks: 5 Running Tasks: 1 Progress: 100.00% ----------
2024-07-17 10:40:17 [INFO] Load task on table "LINEITEM" is committed successfully! Elapsed: 12.58s
2024-07-17 10:40:18 [INFO] [Timer] Table: LINEITEM, Write Elapsed: 9.31s, Commit Elapsed: 12.58s, Total Elapsed: 21.89s
2024-07-17 10:40:18 [INFO] ---------- Finished Tasks: 6 Running Tasks: 0 Progress: 100.00% ----------
2024-07-17 10:40:18 [INFO]
All Load Tasks Finished:
----------------------------------------------------------------------------------------------------------------------------
No.# | Type | Name | Count | Status
----------------------------------------------------------------------------------------------------------------------------
1 | TABLE | LINEITEM | 2665953 -> 2665953 | SUCCESS
----------------------------------------------------------------------------------------------------------------------------
Total Count: 2665953 End Time: 2024-07-17 10:40:18
2024-07-17 10:40:18 [INFO] Load record finished. Total Elapsed: 32.73 s
2024-07-17 10:40:18 [INFO] System exit 0
