java连接ob数据库不稳定,有时可以有时启动报错,运行过程中也可能报错

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】
【 使用版本 】4.2.0.0
【问题描述】通过连接池连接ob数据库不稳定,时而启动时正常启动,时而启动报错,以下是连接信息

连接信息如下:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: com.mysql.cj.jdbc.Driver
jdbcUrl: jdbc:mysql://10.1.12.47:2883/dev_global_db?useUnicode=true&characterEncoding=utf-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=falsie&serverTimezone=Asia/Shanghai&zeroDateTimeBehavior=CONVERT_TO_NULL&allowPublicKeyRetrieval=true&pinGlobalTxToPhysicalConnection=true&useSSL=false&nullCatalogMeansCurrent=true
username: hnwq@rems_tenant#rems
password: hnwq
minIdle: 10
maxActive: 50

报以下异常:
以下错误是下午运行过程报的一个异常,见图片

以下是mysql驱动版本:
com.mysql
mysql-connector-j
8.3.0
现在用的是8.3.0这个版本,也根据ob老师的建议试过8.0.21 和 5.1.47 一样的问题

ob集群配置:
10.1.12.47
10.1.12.48
10.1.12.49
三个observer节点, 其中10.1.12.47上还布署了obproxy

obd cluster display rems
Get local repositories and plugins ok
Open ssh connection ok
Cluster status check ok
Connect to observer ok
Wait for observer init ok
±---------------------------------------------+
| observer |
±-----------±--------±-----±------±-------+
| ip | version | port | zone | status |
±-----------±--------±-----±------±-------+
| 10.1.12.47 | 4.2.0.0 | 2881 | zone1 | ACTIVE |
| 10.1.12.48 | 4.2.0.0 | 2881 | zone2 | ACTIVE |
| 10.1.12.50 | 4.2.0.0 | 2881 | zone4 | ACTIVE |
±-----------±--------±-----±------±-------+

Connect to obproxy ok
±---------------------------------------------+
| obproxy |
±-----------±-----±----------------±-------+
| ip | port | prometheus_port | status |
±-----------±-----±----------------±-------+
| 10.1.12.47 | 2883 | 2884 | active |
±-----------±-----±----------------±-------+
Connect to Obagent ok
±--------------------------------------------------------------+
| obagent |
±-----------±-------------------±-------------------±-------+
| ip | mgragent_http_port | monagent_http_port | status |
±-----------±-------------------±-------------------±-------+
| 10.1.12.47 | 8089 | 8088 | active |
| 10.1.12.48 | 8089 | 8088 | active |
| 10.1.12.50 | 8089 | 8088 | active |
±-----------±-------------------±-------------------±-------+
Connect to ocp-express ok
±--------------------------------------------------------------+
| ocp-express |
±-----------------------±---------±-----------------±-------+
| url | username | initial password | status |
±-----------------------±---------±-----------------±-------+
| http://10.1.12.47:8180 | admin | ****** | active |
±-----------------------±---------±-----------------±-------+

以下中ob日志(observer.log是10.1.12.47的),报错的大概时间段在 15:52 - 15:54 间
observer-10.1.12.48.log.zip (603.7 KB)
obproxy.log.zip (2.8 MB)
observer.log.zip (1.3 MB)
observer-10.1.12.50.log.zip (9.0 MB)

17:25通过datagrip连接也报错了:
java.sql.SQLException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

部分日志如下:

[2024-04-17 17:25:01.439484] INFO [PROXY] ob_proxy_session_info_handler.cpp:746 [8130][Y0-00002B8C31380DB0] [lt=18] [dc=0] succ to set ob_capability_flag in negotiation(client_cap=0, server_cap=916303, orig_client_cap=0, orig_server_cap=916303, client_support_ob_v2=false, server_support_checksum=true, server_support_ob_v2=true, is_auth_request=false)
[2024-04-17 17:25:01.440419] INFO [PROXY.TXN] ob_mysql_transact.cpp:4252 [8130][Y0-00002B8C31380DB0] [lt=15] [dc=0] received reroute error, mayby need reroute((addr={10.1.12.48:2881}, pll_info={pl_attempts:1, lookup_success:true, force_renew_state:0, cached_dummy_entry_renew_state:0, te_name:{cluster_name:“rems”, tenant_name:“sys”, database_name:“oceanbase”, package_name:"", table_name:"_all_dummy"}, route:{consistency_level:“WEAK”, cur_chosen_route_type:“ROUTE_TYPE_NONPARTITION_UNMERGE_LOCAL”, cur_chosen_server:{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.48:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:true}, cur_chosen_pl:null, valid_count:3, leader_item:{idc_type:"", zone_type:“Invalid”, replica:NULL, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:true}, ldc_route:{policy:“MERGE_IDC_ORDER_OPTIMIZED”, disable_merge_status_check:false, curr_cursor_index:6, next_index_in_site:1, location:{use_ldc:false, idc_name:"", item_count:3, site_start_index_array:[[0]0, [1]3, [2]3, [3]3], item_array:[[0]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.48:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:true}, [1]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.50:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:false}, [2]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.47:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:false}], pl:null, ts:0x2b8c29d42060, readonly_exist_status:“READONLY_ZONE_UNKNOWN”}}, is_table_entry_from_remote:false, is_part_entry_from_remote:false, need_use_dup_replica:false, table_entry:0x2b8c31363a50, part_entry:null, dummy_entry:0x2b8c31363a50, skip_leader_item:false}}, state=“RESPONSE_ERROR”, error_type=“REQUEST_REROUTE_ERROR”, reroute_info={schema_version:1691735795295560, replica:{server:“10.1.12.47:2881”, is_dup_replica:false, role:“LEADER”, type:“FULL”}, table_name_buf:“all_zone"})
[2024-04-17 17:25:01.440475] INFO [PROXY.TXN] ob_mysql_transact.cpp:4322 [8130][Y0-00002B8C31380DB0] [lt=53] [dc=0] get err pkt from observer, not forward to client, will be consumed soon, and maybe need retry(send_action=“SERVER_SEND_REQUEST”, state=“RESPONSE_ERROR”, error_type=“REQUEST_REROUTE_ERROR”, resp={is_trans_completed:true, is_resp_completed:true, ending_type:2, is_partition_hit:true, has_new_sys_var:false, has_proxy_idc_name_user_var:false, is_server_db_reset:false, reserved_len:0, connection_id:0, scramble_buf:"", is_resultset_resp:false, server_capabilities_lower
.capability:0, ok_packet_action_type:2, last_ok_pkt_len:174, rewritten_last_ok_pkt_len:0, extra_info:{is_exist_sess_info:false, sess_info_count:0, extra_len:0}, error_pkt:field_count:255, errcode:5727, sqlstate:“HY000”, message:“SQL request should be rerouted”})
[2024-04-17 17:25:01.440493] INFO [PROXY.TXN] ob_mysql_transact.cpp:5146 [8130][Y0-00002B8C31380DB0] [lt=13] [dc=0] pre try failed, will retry((retry_status=FOUND_EXISTING_ADDR, attempts now=2, max_connect_attempts=7, old_target_server={10.1.12.48:2881}, retry server={10.1.12.47:2881}, is_first_request=true, is_auth_request=false, is_force_renew=false, force_retry_congested=false, send_action=“SERVER_SEND_REQUEST”, state=“RESPONSE_ERROR”, route={consistency_level:“WEAK”, cur_chosen_route_type:“ROUTE_TYPE_NONPARTITION_UNMERGE_LOCAL”, cur_chosen_server:{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.48:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:true}, cur_chosen_pl:null, valid_count:3, leader_item:{idc_type:"", zone_type:“Invalid”, replica:NULL, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:true}, ldc_route:{policy:“MERGE_IDC_ORDER_OPTIMIZED”, disable_merge_status_check:false, curr_cursor_index:6, next_index_in_site:1, location:{use_ldc:false, idc_name:"", item_count:3, site_start_index_array:[[0]0, [1]3, [2]3, [3]3], item_array:[[0]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.48:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:true}, [1]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.50:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:false}, [2]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.47:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:false}], pl:null, ts:0x2b8c29d42060, readonly_exist_status:“READONLY_ZONE_UNKNOWN”}}, is_table_entry_from_remote:false, is_part_entry_from_remote:false, need_use_dup_replica:false, table_entry:0x2b8c31363a50, part_entry:null, dummy_entry:0x2b8c31363a50, skip_leader_item:false}, s.congestion_entry={ref_count:18, this:0x2b8c33d59080, zone_state:{ref_count:2, this:0x2b8c33d4f090, zone_name:“zone2”, region_name:“sys_region”, state:“ACTIVE”, last_revalidate_time_us:1713167566417604}, server_ip:{10.1.12.48:2881}, server_state:“DETECT_ALIVE”, entry_state:“ENTRY_AVAIL”, alive_congested:0, last_alive_congested:0, last_revalidate_time_us:1713345628671791, dead_congested:0, detect_congested:0, stat_alive_failures:0, stat_conn_failures:0, cr_version:2, conn_last_fail_time=0, conn_failure_events=0, alive_last_fail_time=0, alive_failure_events=0})
[2024-04-17 17:25:01.440542] INFO [PROXY.SS] ob_mysql_server_session.cpp:153 [8130][Y0-00002B8C31380DB0] [lt=43] [dc=0] server session do_io_close((*this={ss_id:44581, server_sessid:3221993730, server_ip:{10.1.12.48:2881}, is_inited:true, magic:19136237, state:2, server_vc:0x2b8c339fb000, client_session:{this:0x2b8c3227aca0, is_proxy_mysql_client:true, is_waiting_trans_first_request:false, need_delete_cluster:false, is_first_dml_sql_got:true, vc_ready_killed:false, active:false, magic:19132429, conn_decrease:true, current_tid:8136, cs_id:2149318658, proxy_sessid:720870811547926559, session_info:{is_inited:true, priv_info:{has_all_privilege:false, cs_id:2149318658, user_priv_set:0, cluster_name:“rems”, tenant_name:“sys”, user_name:“proxyro”}, version:{common_hot_sys_var_version:0, common_sys_var_version:0, mysql_hot_sys_var_version:0, mysql_sys_var_version:0, hot_sys_var_version:0, sys_var_version:1, user_var_version:1, db_name_version:1, last_insert_id_version:0, sess_info_version:34785}, hash_version:{common_hot_sys_var_version:0, common_sys_var_version:0, mysql_hot_sys_var_version:0, mysql_sys_var_version:0, hot_sys_var_version:0, sys_var_version:0, user_var_version:0, db_name_version:0, last_insert_id_version:0, sess_info_version:0}, val_hash:{common_hot_sys_var_hash:0, common_cold_sys_var_hash:0, mysql_hot_sys_var_hash:0, mysql_cold_sys_var_hash:0, hot_sys_var_hash:0, cold_sys_var_hash:0, user_var_hash:0}, global_vars_version:0, is_global_vars_changed:false, is_trans_specified:false, is_user_idc_name_set:false, is_read_consistency_set:false, idc_name:"", cluster_id:0, real_meta_cluster_name:"", safe_read_snapshot:1, syncing_safe_read_snapshot:0, route_policy:1, proxy_route_policy:3, user_identity:4, global_vars_version:0, is_read_only_user:false, is_request_follower_user:false, obproxy_force_parallel_query_dop:1, ob20_request:{remain_payload_len:0, ob20_request_received_done:false, ob20_header:{ob 20 protocol header:{compressed_len:0, seq:0, non_compressed_len:0}, magic_num:0, header_checksum:0, connection_id:0, request_id:0, pkt_seq:0, payload_len:0, version:0, flag
.flags:0, reserved:0}}, client_cap:0, server_cap:916303, last_server_addr:{Not IP address [0]:0}, last_server_sess_id:0}, dummy_ldc:{use_ldc:false, idc_name:”", item_count:3, site_start_index_array:[[0]0, [1]3, [2]3, [3]3], item_array:[[0]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.48:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:false}, [1]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.50:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:false}, [2]{idc_type:“SAME_IDC”, zone_type:“ReadWrite”, replica:{server:“10.1.12.47:2881”, is_dup_replica:false, role:“FOLLOWER”, type:“FULL”}, is_merging:false, is_partition_server:false, is_force_congested:false, is_used:false}], pl:null, ts:0x2b8c29d42060, readonly_exist_status:“READONLY_ZONE_NOT_EXIST”}, dummy_entry:0x2b8c31363a50, server_state_version:2, cur_ss:null, bound_ss:0x2b8c33f5a4e0, lii_ss:null, cluster_resource:{this:0x2b8c33e04080, ref_count:46, is_inited:true, cluster_info_key:{cluster_name:{config_string:“rems”}, cluster_id:0}, cr_state:“CR_AVAIL”, version:2, last_access_time_ns:1713345901313222282, deleting_completed_thread_num:0, fetch_rslist_task_count:0, fetch_idc_list_task_count:0, last_idc_list_refresh_time_ns:1713167566390156287, last_rslist_refresh_time_ns:0, server_state_version:2}, client_vc:0x2b8c313bd830, using_ldg:false, trace_stats:NULL}, transact_count:1, server_trans_stat:0, session_info:{cap:916303, ob_server:, version:{common_hot_sys_var_version:0, common_sys_var_version:0, mysql_hot_sys_var_version:0, mysql_sys_var_version:0, hot_sys_var_version:0, sys_var_version:1, user_var_version:1, db_name_version:1, last_insert_id_version:0, sess_info_version:34785}, val_hash:{common_hot_sys_var_hash:0, common_cold_sys_var_hash:0, mysql_hot_sys_var_hash:0, mysql_cold_sys_var_hash:0, hot_sys_var_hash:0, cold_sys_var_hash:0, user_var_hash:0}, checksum_switch:1, server_type:1}}, server_vc
=0x2b8c339fb000, this=0x2b8c33f5a4e0)
[2024-04-17 17:25:01.440652] INFO [PROXY.SS] ob_mysql_server_session.cpp:190 [8130][Y0-00002B8C31380DB0] [lt=107] [dc=0] server session is closing(ss_id=44581, server_sessid=3221993730, server_ip={10.1.12.48:2881}, cs_id=2149318658, proxy_sessid=720870811547926559)
[2024-04-17 17:25:01.440723] INFO [PROXY.SS] ob_mysql_server_session.cpp:106 [8130][Y0-00002B8C31380DB0] [lt=4] [dc=0] server session born(ss_id=44582, server_ip={10.1.12.47:2881}, cs_id=2149318658, proxy_sessid=720870811547926559, server_type=1)
[2024-04-17 17:25:01.441263] INFO [PROXY] ob_proxy_session_info_handler.cpp:746 [8130][Y0-00002B8C31380DB0] [lt=11] [dc=0] succ to set ob_capability_flag in negotiation(client_cap=0, server_cap=916303, orig_client_cap=0, orig_server_cap=916303, client_support_ob_v2=false, server_support_checksum=true, server_support_ob_v2=true, is_auth_request=false)

详细日志见文件:

obproxy-10.1.12.47.log 2.zip (3.5 MB)


observer 的配置是怎样的,我看日志里有好多存储层报 cost too much time,是不是磁盘性能不行
用命令行连会出现连不上或者执行出错的情况吗

感谢老师,命令行我们用的少,主要是客户端和程序来连接,如何看硬盘的性能?但我们现在ob读写数据很少的,怎么会出现性能问题呢?

做了一个简单的测试:
dd if=/dev/zero of=testfile bs=1G count=1 oflag=dsync

记录了1+0 的读入

记录了1+0 的写出

1073741824字节(1.1 GB)已复制,6.38334 秒,168 MB/秒

OB配置:
user:
username: admin
key_file: /home/admin/.ssh/id_rsa
port: 22
oceanbase-ce:
version: 4.2.0.0
release: 100000152023080109.el7
package_hash: 5cc69b0ce9944adb57e36deb449bb70786d3ddc5
10.1.12.47:
zone: zone1
10.1.12.48:
zone: zone2

10.1.12.49:

zone: zone3

10.1.12.50:
zone: zone4
servers:

  • 10.1.12.47
  • 10.1.12.48
    #- 10.1.12.49
  • 10.1.12.50
    global:
    appname: rems
    mysql_port: 2881
    rpc_port: 2882
    home_path: /data/install/oceanbase//rems/oceanbase
    datafile_size: 500G
    log_disk_size: 100G
    ocp_meta_tenant_max_cpu: ‘1’
    ocp_meta_tenant_memory_size: 2G
    ocp_meta_tenant_log_disk_size: 7G
    enable_syslog_recycle: true
    enable_syslog_wf: false
    max_syslog_file_count: 4
    memory_limit: 50G
    system_memory: 8G
    cpu_count: 16
    obproxy-ce:
    version: 4.2.0.0
    package_hash: b3ead2f667d69fe3195fd22002de83df6eb19382
    release: 7.el7
    servers:
  • 10.1.12.47
    global:
    prometheus_listen_port: 2884
    listen_port: 2883
    home_path: /data/install/oceanbase//rems/obproxy
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    enable_cluster_checkout: false
    #rs_list: 10.1.12.47:2881;10.1.12.48:2881;10.1.12.49:2881
    rs_list: 10.1.12.47:2881;10.1.12.48:2881;10.1.12.50:2881
    cluster_name: rems
    depends:
  • oceanbase-ce
    obagent:
    version: 4.2.0
    package_hash: 30793df12dc6b8ec5ccdc93262e5e9f1d51ed50a
    release: 3.el7
    servers:
  • 10.1.12.47
  • 10.1.12.48
    #- 10.1.12.49
  • 10.1.12.50
    global:
    monagent_http_port: 8088
    mgragent_http_port: 8089
    home_path: /data/install/oceanbase//rems/obagent
    ob_monitor_status: active
    depends:
  • oceanbase-ce
    ocp-express:
    version: 4.2.0
    package_hash: ccec08112a29067633797d20685b6e6d70e890d9
    release: 100000042023073111.el7
    servers:
  • 10.1.12.47
    global:
    port: 8180
    home_path: /data/install/oceanbase//rems/ocpexpress
    memory_size: 872M
    depends:
  • obagent
  • oceanbase-ce
  • obproxy-ce

@chris-sun