OBD部署三节点集群 bootstrap cluster 失败

liliang · 2021 年9 月 16 日 11:07

【产品名称】

ob社区版本

【产品版本】

3.1.0

【问题描述】

bootstrap cluster failed

[root@dev ~]# obd cluster start ob_cluster Get local repositories and plugins ok Open ssh connection ok Cluster param config check ok Check before start observer ok Check before start obproxy ok Start observer ok observer program health check ok Connect to observer okjInitialize clusteriCluster bootstrap xaCluster init failedn

检查日志如下

[2021-09-16 10:24:51] [13daa014-1695-11ec-ac0a-005056b7197c] [DEBUG] – execute sql: set session ob_query_timeout=1000000000 [2021-09-16 10:24:51] [13daa014-1695-11ec-ac0a-005056b7197c] [INFO] Cluster bootstrap [2021-09-16 10:24:51] [13daa014-1695-11ec-ac0a-005056b7197c] [DEBUG] – execute sql: alter system bootstrap REGION “sys_region” ZONE “zone-01” SERVER “172.31.60.181:2882”,REGION “sys_region” ZONE “zone-02” SERVER “172.31.60.182:2882”,REGION “sys_region” ZONE “zone-03” SERVER “172.31.60.183:2882” [2021-09-16 10:41:32] [13daa014-1695-11ec-ac0a-005056b7197c] [ERROR] Traceback (most recent call last): File “core.py”, line 989, in start_cluster File “_plugin.py”, line 234, in __call__ File “_plugin.py”, line 208, in _new_func File “/root/.obd/plugins/oceanbase-ce/3.1.0/bootstrap.py”, line 46, in bootstrap cursor.execute(sql) File “/usr/obd/lib/site-packages/pymysql/cursors.py”, line 148, in execute result = self._query(query) File “/usr/obd/lib/site-packages/pymysql/cursors.py”, line 310, in _query conn.query(q) File “/usr/obd/lib/site-packages/pymysql/connections.py”, line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File “/usr/obd/lib/site-packages/pymysql/connections.py”, line 775, in _read_query_result result.read() File “/usr/obd/lib/site-packages/pymysql/connections.py”, line 1156, in read first_packet = self.connection._read_packet() File “/usr/obd/lib/site-packages/pymysql/connections.py”, line 725, in _read_packet packet.raise_for_error() File “/usr/obd/lib/site-packages/pymysql/protocol.py”, line 221, in raise_for_error err.raise_mysql_exception(self._data) File “/usr/obd/lib/site-packages/pymysql/err.py”, line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.OperationalError: (4012, ‘Timeout’)

liliang · 2021 年9 月 16 日 11:10

配置文件如下

user:
  username: admin
  password: '123456'
oceanbase-ce:
  servers:
    - name: zone-01
      ip: 172.31.60.181
    - name: zone-02
      ip: 172.31.60.182
    - name: zone-03
      ip: 172.31.60.183
  global:
    devname: ens192
    cluster_id: 1 
    memory_limit: 10G
    system_memory: 4G
    stack_size: 512K
    cpu_count: 8
    cache_wash_threshold: 1G
    __min_full_resource_pool_memory: 268435456
    workers_per_cpu_quota: 5
    schema_history_expire_time: 1d
    net_thread_count: 4
    major_freeze_duty_time: Disable
    minor_freeze_times: 10
    enable_separate_sys_clog: 0
    enable_merge_by_turn: FALSE
    datafile_disk_percentage: 20
    syslog_level: INFO
    enable_syslog_wf: false
    enable_syslog_recycle: true
    max_syslog_file_count: 4
    appname: ob_cluster
    root_password: '123456'
    proxyro_password: '123456'
  zone-01:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase-ce
    data_dir: /data
    redo_dir: /redo
    zone: zone-01
  zone-02:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    home_path: /home/admin/oceanbase-ce
    data_dir: /data
    redo_dir: /redo
    zone: zone-02
  zone-03:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    home_path: /home/admin/oceanbase-ce
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data
    redo_dir: /redo
    zone: zone-03
obproxy:
  servers:
    - 172.31.60.180
  global:
    listen_port: 2883 # External port. The default value is 2883.
    prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
    home_path: /home/admin/oceanbase-ce
    rs_list: 172.31.60.181:2881;172.31.60.182:2881;172.31.60.183:2881
    enable_cluster_checkout: false
    # observer cluster name, consistent with oceanbase-ce's appname
    cluster_name: ob_cluster
    obproxy_sys_password: 
    observer_sys_password:

和顺 · 2021 年9 月 16 日 11:19

麻烦先调整obproxy_sys_password要和proxyro_password的密码一致再试一下。

obpilot · 2021 年9 月 16 日 11:32

| 配置类     | 配置项名                   | 配置值                           | 备注                                                                      |

|       | root_password              | 随机字符串                          | OB 集群的超级管理员 root@sys 的密码，默认是空，建议设置复杂的密码。                                                  |

|       | proxyro_password             | 随机字符串                          | OBPROXY 连接 OB集群使用的账户名（proxyro) 的密码                                                      |

obpilot · 2021 年9 月 16 日 11:33

|       | obproxysyspassword           | 随机字符串                          | OBPROXY 管理员账户（proxysys ）的密码。                                                         |

|       | observersyspassword          | 跟 proxyro_password 一致                  | OBPROXY 连接 OB集群使用的账户名（proxyro) 的密码                                                      |

和顺 · 2021 年9 月 16 日 11:40

不好意思，我这里记错了，请参考庆涛老师的，observer_sys_password要跟proxyro_password一致。

liliang · 2021 年9 月 16 日 15:34

已经重新修改配置部署启动，还是一样的错误

## Only need to configure when remote login is required
user:
  username: admin
  password: '123456'
oceanbase-ce:
  servers:
    - name: zone-01
      ip: 172.31.60.181
    - name: zone-02
      ip: 172.31.60.182
    - name: zone-03
      ip: 172.31.60.183
  global:
    devname: ens192
    cluster_id: 1
    memory_limit: 10G # The maximum running memory for an observer
    system_memory: 4G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
    stack_size: 512K
    cpu_count: 8
    cache_wash_threshold: 1G
    __min_full_resource_pool_memory: 268435456
    workers_per_cpu_quota: 5
    schema_history_expire_time: 1d
    # The value of net_thread_count had better be same as cpu's core number. 
    net_thread_count: 4
    major_freeze_duty_time: Disable
    minor_freeze_times: 10
    enable_separate_sys_clog: 0
    enable_merge_by_turn: FALSE
    datafile_disk_percentage: 20 # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90.
    syslog_level: INFO # System log level. The default value is INFO.
    enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
    enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
    max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
    # observer cluster name, consistent with obproxy's cluster_name
    appname: ob_cluster
    root_password: Qwerty1
    proxyro_password: Qwerty1
  zone-01:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    home_path: /home/admin/oceanbase-ce
    data_dir: /data
    redo_dir: /redo
    zone: zone-01
  zone-02:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    home_path: /home/admin/oceanbase-ce
    data_dir: /data
    redo_dir: /redo
    zone: zone-02
  zone-03:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    home_path: /home/admin/oceanbase-ce
    data_dir: /data
    redo_dir: /redo
    zone: zone-03
obproxy:
  servers:
    - 172.31.60.180
  global:
    listen_port: 2883 # External port. The default value is 2883.
    prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
    home_path: /home/admin/oceanbase-ce
    # oceanbase root server list
    # format: ip:mysql_port;ip:mysql_port
    rs_list: 172.31.60.181:2881;172.31.60.182:2881;172.31.60.183:2881
    enable_cluster_checkout: false
    # observer cluster name, consistent with oceanbase-ce's appname
    cluster_name: ob_cluster
    obproxy_sys_password: Qwerty1
    observer_sys_password: Qwerty1

obpilot · 2021 年9 月 17 日 09:55

看配置文件里是想使用 /home/admin/ 这个目录，以及使用 admin 这个用户。看帖子命令是运行在root 用户下。

请先清理所有相关目录。然后切换到 admin 用户下，重新创建部署配置文件试试。

此外，先检查三节点之间的时间是否同步、防火墙是否关闭等等。

如果还有问题，请在 OB社区钉钉群里联系我( OB庆涛）。