obd部署obproxy不成功

OceanBase
#1

【产品名称】

【产品版本】

【问题描述】

3台节点集群，一台中控机，obproxy部署到其中一个节点，集群启动成功，但obproxy不成功。3台节点配置：4cpu+16g 虚拟机。

部署日志、obproxy启动日志、yaml如下：

===============================================deploy and start====

[root@oceanbase_x ~]# obd cluster deploy zkcluster -c rpm/proxy_mini-distributed-example.yaml

oceanbase-ce-3.1.0 already installed

obproxy-3.1.0 already installed

±------------------------------------------------------------------+

| Packages |

±-------------±--------±-----------------------------------------+

| Repository | Version | Md5 |

±-------------±--------±-----------------------------------------+

| oceanbase-ce | 3.1.0 | 56f57e9843e719d830ec03c206d914f4b3adc82b |

| obproxy | 3.1.0 | 0b17cf0459a3b53c5a2febb6572894d183154c64 |

±-------------±--------±-----------------------------------------+

Open ssh connection ok

Remote oceanbase-ce-3.1.0-56f57e9843e719d830ec03c206d914f4b3adc82b repository install ok

Remote oceanbase-ce-3.1.0-56f57e9843e719d830ec03c206d914f4b3adc82b repository lib check ok

Remote obproxy-3.1.0-0b17cf0459a3b53c5a2febb6572894d183154c64 repository install ok

Remote obproxy-3.1.0-0b17cf0459a3b53c5a2febb6572894d183154c64 repository lib check ok

z1(10.1.10.16) initializes cluster work home

z2(10.1.10.17) initializes cluster work home

z3(10.1.10.18) initializes cluster work home

10.1.10.18 init cluster work home

zkcluster deployed

[root@oceanbase_x ~]# obd cluster start zkcluster

Get local repositories and plugins ok

Start observer ok

observer program health check ok

Connect to observer ok

Initialize cluster

Cluster bootstrap ok

Wait for observer init ok

±---------------------------------------------+

| observer |

±-----------±--------±-----±------±-------+

| ip | version | port | zone | status |

±-----------±--------±-----±------±-------+

| 10.1.10.16 | 3.1.0 | 2883 | zone1 | active |

| 10.1.10.17 | 3.1.0 | 2883 | zone2 | active |

| 10.1.10.18 | 3.1.0 | 2881 | zone3 | active |

±-----------±--------±-----±------±-------+

Start obproxy ok

obproxy program health check x

[WARN] failed to start 10.1.10.18 obproxy

[ERROR] obproxy start failed

[root@oceanbase_x ~]#

#2

=========================================================yaml====================================================================

[root@oceanbase_x ~]# cat rpm/proxy_mini-distributed-example.yaml 

user:

 username: root

 key_file: /root/.ssh/id_rsa.pub

oceanbase-ce:

 servers:

  - name: z1

   # Please don't use hostname, only IP can be supported

   ip: 10.1.10.16

  - name: z2

   ip: 10.1.10.17

  - name: z3

   ip: 10.1.10.18

 global:

  # Please set devname as the network adaptor's name whose ip is in the setting of severs.

  # if set severs as "127.0.0.1", please set devname as "lo"

  # if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"

  devname: enp0s3

  cluster_id: 1

  datafile_size: 8G

  # please set memory limit to a suitable value which is matching resource. 

  memory_limit: 8G

  system_memory: 4G

  stack_size: 512K

  cpu_count: 16

  cache_wash_threshold: 1G

  __min_full_resource_pool_memory: 268435456

  workers_per_cpu_quota: 10

  schema_history_expire_time: 1d

  # The value of net_thread_count had better be same as cpu's core number. 

  net_thread_count: 4

  major_freeze_duty_time: Disable

  minor_freeze_times: 10

  enable_separate_sys_clog: 0

  enable_merge_by_turn: FALSE

  datafile_disk_percentage: 20

  max_syslog_file_count: 10

  # observer cluster name, consistent with obproxy's cluster_name

  appname: zkcluster

  # root user password, can be empty

  root_password: 123456

  # proxyro user password, consistent with obproxy's observer_sys_password,can be empty

  proxyro_password: 123456

 z1:

  mysql_port: 2883

  rpc_port: 2882

  home_path: /opt/observer

  zone: zone1

 z2:

  mysql_port: 2883

  rpc_port: 2882

  home_path: /opt/observer

  zone: zone2

 z3:

  mysql_port: 2881

  rpc_port: 2882

  home_path: /opt/observer

  zone: zone3


obproxy:

 servers:

  - 10.1.10.18

 global:

  listen_port: 2883

  #prometheus_listen_port: 2884

  home_path: /opt/obproxy

  # oceanbase root server list

  # format: ip:mysql_port;ip:mysql_port

  rs_list: 10.1.10.16:2883;10.1.10.17:2883;10.1.10.18:2881

  enable_cluster_checkout: false

  # observer cluster name, consistent with oceanbase-ce's appname

  cluster_name: zkcluster

  # obproxy sys user password, can be empty

  obproxy_sys_password: 123456

  # proxyro user password, consistent with oceanbase-ce's proxyro_password, can be empty

  observer_sys_password: 123456

[root@oceanbase_x ~]# 

#3

======================================obproxy.log==========================================================================

[2021-08-04 09:16:39.409628] INFO [PROXY] ob_proxy_main.cpp:560 [29836][Y0-0] [lt=0] [dc=0] succ to init logger(max_log_file_size=268435456, async_tid=140524778792704)

[2021-08-04 09:16:39.409940] INFO ob_file.cpp:59 [29836][Y0-0] [lt=90] [dc=0] open fname=[/dev/urandom] fd=14 flags=0 succ

[2021-08-04 09:16:39.410034] INFO [PROXY] ob_proxy_main.cpp:476 [29836][Y0-0] [lt=25] [dc=0] ObProxy-OceanBase 3.1.0-1-1-local-1a113db0fc2a2e4294b213c2c2556decfe8b4e65

[2021-08-04 09:16:39.410046] INFO [PROXY] ob_proxy_main.cpp:480 [29836][Y0-0] [lt=8] [dc=0] has no inherited sockets, start new obproxy((info={is_inherited:false, upgrade_version:-1, need_conn_accept:true, user_rejected:0, fd:-1, received_sig:-1, sub_pid:-1, graceful_exit_end_time:0, graceful_exit_start_time:0, active_client_vc_count:-1, local_addr:"0.0.0.0", rc_status:"", hu_cmd:"", state:"HU_STATE_WAIT_HU_CMD", hu_status:"", is_parent:true, sub_status:"", last_parent_status:"", last_sub_status:"", upgrade_version_buf:"", argc:11, argv[0]="/root/.obd/repository/obproxy/3.1.0/0b17cf0459a3b53c5a2febb6572894d183154c64/bin/obproxy", argv[1]="-o", argv[2]="enable_strict_kernel_release=False,enable_cluster_checkout=False,obproxy_sys_password=123456,observer_sys_password=123456", argv[3]="--listen_port", argv[4]="2883", argv[5]="--prometheus_listen_port", argv[6]="2884", argv[7]="--rs_list", argv[8]="10.1.10.16:2883;10.1.10.17:2883;10.1.10.18:2881", argv[9]="--cluster_name", argv[10]="zkcluster", inherited_argv[0]="/root/.obd/repository/obproxy/3.1.0/0b17cf0459a3b53c5a2febb6572894d183154c64/bin/obproxy", inherited_argv[1]="(null)", inherited_argv[2]="(null)", inherited_argv[3]="(null)"})

[2021-08-04 09:16:39.417760] INFO [PROXY] ob_proxy.cpp:596 [29836][Y0-0] [lt=99] [dc=0] fail to load config from file, but we can get the config from OCP by sql later(ret=0)

[2021-08-04 09:16:39.417841] INFO [SHARE] ob_common_config.cpp:97 [29836][Y0-0] [lt=32] [dc=0] Load config succ(name="enable_strict_kernel_release", value="False")

[2021-08-04 09:16:39.417861] INFO [SHARE] ob_common_config.cpp:97 [29836][Y0-0] [lt=17] [dc=0] Load config succ(name="enable_cluster_checkout", value="False")

[2021-08-04 09:16:39.417866] INFO [SHARE] ob_common_config.cpp:97 [29836][Y0-0] [lt=4] [dc=0] Load config succ(name="obproxy_sys_password", value="123456")

[2021-08-04 09:16:39.417878] INFO [SHARE] ob_common_config.cpp:97 [29836][Y0-0] [lt=11] [dc=0] Load config succ(name="observer_sys_password", value="123456")

[2021-08-04 09:16:39.417967] INFO [PROXY] ob_config_server_processor.cpp:1149 [29836][Y0-0] [lt=3] [dc=0] succ to init_proxy_kernel_release by unknown kernel(kernel_release_=RELEASE_UNKNOWN, enable_strict_kernel_release="False")

[2021-08-04 09:16:39.418051] INFO [PROXY] ob_proxy.cpp:567 [29836][Y0-0] [lt=50] [dc=0] use test mode, do not depend on config server(cluster_name="zkcluster", cluster_rs_list="10.1.10.16:2883;10.1.10.17:2883;10.1.10.18:2881")

[2021-08-04 09:16:39.418449] INFO [PROXY] ob_route_utils.cpp:905 [29836][Y0-0] [lt=7] [dc=0] update sys tenant __all_dummy succ((cluster_name=zkcluster, tmp_entry={{this:0x7fce78ee5610, ref_count:3, cr_version:0, cr_id:0, create_time_us:1628039799418376, last_valid_time_us:0, last_access_time_us:0, last_update_time_us:0, schema_version:0, tenant_version:0, time_for_expired:0, state:"AVAIL"}, this:0x7fce78ee5610, is_inited:true, is_dummy_entry:true, is_entry_from_rslist:true, is_empty_entry_allowed:false, is_need_force_flush:false, cr_id:0, name:{cluster_name:"zkcluster", tenant_name:"sys", database_name:"oceanbase", package_name:"", table_name:"__all_dummy"}, table_id:1099511627911, table_type:"UNKNOWN", part_num:1, replica_num:3, buf_len:32, buf_start:0x7fce78ee5710, tenant_servers:{this:0x7fce78eef7a0, is_inited:true, server_count:3, replica_count:3, partition_count:1, next_partition_idx:0, server_array:0x7fce78e5d7a0, server_array_:[[0]{server:"10.1.10.17:2883", role:"FOLLOWER", type:"FULL"}, [1]{server:"10.1.10.16:2883", role:"FOLLOWER", type:"FULL"}, [2]{server:"10.1.10.18:2881", role:"FOLLOWER", type:"FULL"}]}})

[2021-08-04 09:16:39.418586] INFO [PROXY] ob_proxy.cpp:618 [29836][Y0-0] [lt=130] [dc=0] use test mode, no need init remote config

[2021-08-04 09:16:39.418762] INFO ob_crc64.cpp:550 [29836][Y0-0] [lt=6] [dc=0] Use manual crc32 table lookup for crc64 calculate


#4

....

*end proxy config report* =======================

[2021-08-04 09:16:39.420423] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=3] [dc=0] load default system variable(var_name=autocommit, var_value=1)

[2021-08-04 09:16:39.420451] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=25] [dc=0] load default system variable(var_name=auto_increment_increment, var_value=1)

[2021-08-04 09:16:39.420458] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=5] [dc=0] load default system variable(var_name=auto_increment_offset, var_value=1)

[2021-08-04 09:16:39.420463] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=4] [dc=0] load default system variable(var_name=last_insert_id, var_value=0)

[2021-08-04 09:16:39.420469] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=5] [dc=0] load default system variable(var_name=interactive_timeout, var_value=0)

[2021-08-04 09:16:39.420475] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=4] [dc=0] load default system variable(var_name=ob_query_timeout, var_value=10000000)

[2021-08-04 09:16:39.420480] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=4] [dc=0] load default system variable(var_name=ob_trx_timeout, var_value=100000000)

[2021-08-04 09:16:39.420486] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=5] [dc=0] load default system variable(var_name=net_read_timeout, var_value=30)

[2021-08-04 09:16:39.420492] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=5] [dc=0] load default system variable(var_name=net_write_timeout, var_value=60)

[2021-08-04 09:16:39.420497] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=4] [dc=0] load default system variable(var_name=wait_timeout, var_value=28800)

[2021-08-04 09:16:39.420501] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=4] [dc=0] load default system variable(var_name=lower_case_table_names, var_value=2)

[2021-08-04 09:16:39.420509] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=3] [dc=0] load default system variable(var_name=tx_read_only, var_value=0)

[2021-08-04 09:16:39.420517] INFO [PROXY] ob_session_field_mgr.cpp:56 [29836][Y0-0] [lt=4] [dc=0] load default system variable(var_name=ob_read_consistency, var_value=3)

[2021-08-04 09:16:59.446222] INFO [PROXY.EVENT] ob_event_processor.h:396 [29836][Y0-0] [lt=8] [dc=0] allocate memory from thread local data buffer(size=544, cur_offset=544, ObEThread::MAX_THREAD_DATA_SIZE=8096)

[2021-08-04 09:16:59.446329] INFO [PROXY.API] ob_mysql_request_prepare_transform_plugin.h:114 [29836][Y0-0] [lt=42] [dc=0] init mysql request prepare transformation plugin


#5

[2021-08-04 09:16:59.446499] INFO [PROXY.API] ob_mysql_request_compress_transform_plugin.h:116 [29836][Y0-0] [lt=10] [dc=0] init mysql request compress transformation plugin

[2021-08-04 09:16:59.446551] INFO [PROXY.API] ob_mysql_response_compress_transform_plugin.h:112 [29836][Y0-0] [lt=11] [dc=0] init mysql response compress transformation plugin

[2021-08-04 09:16:59.446585] INFO [PROXY.API] ob_mysql_response_prepare_transform_plugin.h:118 [29836][Y0-0] [lt=8] [dc=0] init mysql response prepare transformation plugin

[2021-08-04 09:16:59.446619] INFO [PROXY.API] ob_mysql_response_cursor_transform_plugin.h:129 [29836][Y0-0] [lt=8] [dc=0] init mysql response cursor transformation plugin

[2021-08-04 09:16:59.446650] INFO [PROXY.API] ob_mysql_response_prepare_execute_transform_plugin.h:124 [29836][Y0-0] [lt=7] [dc=0] init mysql response prepare execute transformation plugin

[2021-08-04 09:16:59.446891] INFO [PROXY] ob_mysql_proxy_server_main.cpp:232 [29836][Y0-0] [lt=7] [dc=0] succ init mysql proxy port by config(proxy_port={0.0.0.0:0}, port=2883})

[2021-08-04 09:16:59.446926] INFO [PROXY.EVENT] ob_event_processor.h:396 [29836][Y0-0] [lt=30] [dc=0] allocate memory from thread local data buffer(size=3232, cur_offset=3776, ObEThread::MAX_THREAD_DATA_SIZE=8096)

[2021-08-04 09:16:59.447014] INFO [PROXY.EVENT] ob_event_processor.h:396 [29836][Y0-0] [lt=7] [dc=0] allocate memory from thread local data buffer(size=1536, cur_offset=5312, ObEThread::MAX_THREAD_DATA_SIZE=8096)

[2021-08-04 09:16:59.447112] INFO [PROXY.EVENT] ob_event_processor.h:396 [29836][Y0-0] [lt=67] [dc=0] allocate memory from thread local data buffer(size=64, cur_offset=5376, ObEThread::MAX_THREAD_DATA_SIZE=8096)

[2021-08-04 09:16:59.447125] INFO [PROXY.EVENT] ob_event_processor.h:396 [29836][Y0-0] [lt=12] [dc=0] allocate memory from thread local data buffer(size=416, cur_offset=5792, ObEThread::MAX_THREAD_DATA_SIZE=8096)

[2021-08-04 09:16:59.447134] INFO [PROXY.EVENT] ob_event_processor.h:396 [29836][Y0-0] [lt=5] [dc=0] allocate memory from thread local data buffer(size=416, cur_offset=6208, ObEThread::MAX_THREAD_DATA_SIZE=8096)

[2021-08-04 09:16:59.447153] INFO [PROXY.EVENT] ob_event_processor.h:396 [29836][Y0-0] [lt=3] [dc=0] allocate memory from thread local data buffer(size=832, cur_offset=7040, ObEThread::MAX_THREAD_DATA_SIZE=8096)

[2021-08-04 09:16:59.447179] INFO [PROXY] ob_proxy.cpp:223 [29836][Y0-0] [lt=16] [dc=0] obproxy init succeed(ret=0)

[2021-08-04 09:16:59.447320] INFO [PROXY.EVENT] ob_event_processor.cpp:136 [29836][Y0-0] [lt=8] [dc=0] get cpu num succeed(cpu_num=4, cpu_conf_num=4, cpu_onln_num=4, MAX_THREADS_IN_EACH_TYPE=256)


#6

[2021-08-04 09:16:59.448278] INFO [PROXY.EVENT] ob_event_processor.cpp:252 [29836][Y0-0] [lt=24] [dc=0] succ to start event thread group id(cpu_num=4, stacksize=1048576)

[2021-08-04 09:16:59.451957] INFO [PROXY.EVENT] ob_grpc_task.cpp:37 [29836][Y0-0] [lt=41] [dc=0] succ to start grpc threads(grpc_threads=8)

[2021-08-04 09:16:59.452280] INFO [PROXY.EVENT] ob_shard_watch_task.cpp:37 [29836][Y0-0] [lt=31] [dc=0] succ to start shard watch threads(shard_watch_threads=1)

[2021-08-04 09:16:59.459555] INFO [PROXY] ob_proxy_main.cpp:689 [29836][Y0-0] [lt=35] [dc=0] succ to schedule sig detect task(interval_us=100000, ret=0)

[2021-08-04 09:16:59.459578] INFO [PROXY] ob_proxy_main.cpp:701 [29836][Y0-0] [lt=19] [dc=0] succ to schedule mem monitor task(interval_us=2000000, ret=0)

[2021-08-04 09:16:59.459588] INFO [PROXY] ob_proxy_main.cpp:713 [29836][Y0-0] [lt=8] [dc=0] succ to schedule sqlaudit detect task(interval_us=1000000, ret=0)

[2021-08-04 09:16:59.459607] INFO [PROXY] ob_proxy.cpp:792 [29836][Y0-0] [lt=7] [dc=0] obproxy reload load new config(ret=0)

[2021-08-04 09:16:59.459734] INFO [PROXY] ob_cache_cleaner.cpp:871 [29836][Y0-0] [lt=8] [dc=0] succ schedule cache cleaners(target_ethread=0x11a50290, range={start_idx:0, end_idx:15, cur_idx:0}, i=0, part_num_per_thread=16, remain_num=0, net_thread_count=4, ret=0)

[2021-08-04 09:16:59.459787] INFO [PROXY] ob_cache_cleaner.cpp:871 [29836][Y0-0] [lt=50] [dc=0] succ schedule cache cleaners(target_ethread=0x11a534b0, range={start_idx:16, end_idx:31, cur_idx:16}, i=1, part_num_per_thread=16, remain_num=0, net_thread_count=4, ret=0)

[2021-08-04 09:16:59.459836] INFO [PROXY] ob_cache_cleaner.cpp:871 [29836][Y0-0] [lt=16] [dc=0] succ schedule cache cleaners(target_ethread=0x11a566d0, range={start_idx:32, end_idx:47, cur_idx:32}, i=2, part_num_per_thread=16, remain_num=0, net_thread_count=4, ret=0)

[2021-08-04 09:16:59.459852] INFO [PROXY] ob_cache_cleaner.cpp:871 [29836][Y0-0] [lt=13] [dc=0] succ schedule cache cleaners(target_ethread=0x11a598f0, range={start_idx:48, end_idx:63, cur_idx:48}, i=3, part_num_per_thread=16, remain_num=0, net_thread_count=4, ret=0)

[2021-08-04 09:16:59.459888] INFO [PROXY] ob_log_file_processor.cpp:95 [29836][Y0-0] [lt=22] [dc=0] succ to create and start log cleanup task(interval_us=600000000)

[2021-08-04 09:16:59.459931] INFO [PROXY] ob_prometheus_processor.cpp:56 [29836][Y0-0] [lt=8] [dc=0] succ to start prometheus sync task(interval_us=60040072)

[2021-08-04 09:16:59.459945] INFO [PROXY] ob_proxy.cpp:335 [29836][Y0-0] [lt=7] [dc=0] obproxy start up successful and can provide normal service(cost time(us)=20049777, is_service_started=true)

#8

ob节点的MySQL port都改为 2881。


后面rs_list里的端口 2883改为 2881。

#9

试了一样错误。看obproxy最后一条日志，启动耗时20多秒，是不是这里原因。“…load default system varible…”这块日志结束20秒后才有日志，每次都这样

#10

先确认一下， OB 集群直连是 OK 的。

mysql -h10.1.10.16 -uroot@sys -P2881 -p123456 -c -A oceanbase

select a.zone,concat(a.svr_ip,':',a.svr_port) observer, cpu_total, (cpu_total-cpu_assigned) cpu_free, round(mem_total/1024/1024/1024) mem_total_gb, round((mem_total-mem_assigned)/1024/1024/1024) mem_free_gb, usec_to_time(b.last_offline_time) last_offline_time, usec_to_time(b.start_service_time) start_service_time, b.status, usec_to_time(b.stop_time) stop_time, b.build_version 

from __all_virtual_server_stat a join __all_server b on (a.svr_ip=b.svr_ip and a.svr_port=b.svr_port)

order by a.zone, a.svr_ip

;
#11

用的obclient，直连可以（但参数带密码报错，不带密码可以），查询结果3条记录。不部署obproxy的3节点集群，前段时间测过租户、用户、表这些都可以的。

#12

mysql客户端版本是 5.5/5.6/5.7吗？

#13

装的obclient-2.0.0-1,libobclient-2.0.0-1,没有额外装mysql客户端