OCP 的 OBProxy 管理密码(obproxy_sys_password)修改问题

问题现象

OBProxy 有些自己的参数,需要通过用户 root@proxysys 和 OBProxy 的 IP 和 2883(默认 OBProxy 连接端口) 访问获取。

如果 OBProxy 是独立使用 RPM 包部署的时候,默认这个管理员密码为空,需要自己设置一个密码。 这个在以前的试验里介绍过(见文后参考)。但是如果是 OCP 部署的 OBProxy ,是会设置一个密码。 这个密码的密文会出现在 OBProxy 的配置文件 (obproxy_config.bin) 的配置(obproxy_sys_password) 中,并且是密文形式出现。反复部署 OB 集群和 OBProxy 观察发现,这个密码密文内容是一样的(1)。

OCP 提供了 OBProxy 的参数修改功能,可以改掉 obproxy_sys_password 的密码。

修改一下这个密码。

密码修改会生成一个任务,任何很快就会成功。
从任务日志里可以确认修改密码的方法,包括密码内容(2)。

密码修改成功后,可以在命令行下测试。进入 OBProxy 里可以分析 OBProxy 的连接信息,对理解 OBProxy 的原理非常有用。具体见文后参考。

但是,回到 OCP 的 OBProxy 界面开始有报错现象。

OBProxy 节点出现不可用提示。

查看 OBProxy 的参数页面报错。由此推断这些参数是 OCP 连接到 OBProxy 里去实时获取的。只不过 OCP 连接 OBProxy 管理用户报错了。

报错信息: 无法连接到 obproxy 10.0.0.61:2883,可能由于 obproxy 进程不存在、错误的 proxysys 密码或者 obproxy 参数 skip_proxy_sys_private_check 不为 true 引起

问题分析

这些 OBProxy 是可以用的,业务通过 OBProxy 连接 OB 集群依然是正常的。问题出在 OCP 连接 OBProxy 环节。

为此,分析一下 OCP 修改 OBProxy 管理密码的过程 。

在 OCP 元数据库里,能找到密码修改的历史。

SELECT obproxy_cluster_id ,version,name,value_before ,value_after ,update_time  FROM obproxy_parameter_change ORDER BY version,name DESC ;
obproxy_cluster_id version name value_before value_after update_time
1000015 2 obproxy_sys_password f0455d5edb4833fa2e91762e71b61bc35d45cf0b aaAA11__ 2023-06-10 19:26:08.000

查看这个日志可知 value_before 是密文,value_after 是明文(3) 。

SELECT * FROM obproxy_parameter WHERE obproxy_cluster_id =1000015;
id obproxy_cluster_id version name value create_time update_time
1000006 1000015 2 obproxy_sys_password aaAA11__ 2023-06-10 19:26:08.000 2023-06-10 19:26:08.000
1000005 1000015 1 syslog_level DEBUG 2023-06-10 19:19:17.000 2023-06-10 19:19:17.000

这里保存的也是明文(4)。

是否是 OCP BUG 导致修改密码后 OCP 连接不了 OBProxy。

尝试更新一下 OCP 元数据。

UPDATE obproxy_parameter_change SET value_after='ce77bdb3aad23ee556c638ea6ed86bad4949f676' where obproxy_cluster_id=1000015 and name='obproxy_sys_password';
UPDATE obproxy_parameter SET value='ce77bdb3aad23ee556c638ea6ed86bad4949f676' WHERE name='obproxy_sys_password' AND obproxy_cluster_id=1000015;
COMMIT;

刷新 OCP 页面查看 OBProxy 配置信息,依然报错。
重启 OCP 容器排除服务端缓存密码问题,再次查看 OBProxy 页面,依然报错。

查看 OCP 的运行日志,也看到连接失败的提示。

docker exec -it ocp bash
cd /home/admin/logs
vi ocp-server.0.out

2023-06-10 19:54:17.027  WARN 46 --- [pool-Check obproxy2,328de6b9191f4c81,ea147e756de8] c.a.ocp.core.odpsdk.OdpJdbcOperator      : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://10.0.0.61:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLSyntaxErrorException: Could not connect to address=(host=10.0.0.61)(port=2883)(type=master) : (conn=2006) Access denied for user 'root@proxysys'@'10.0.0.64' (using password: NO)
2023-06-10 19:54:17.027  WARN 46 --- [pool-Check obproxy2,328de6b9191f4c81,ea147e756de8] c.a.o.s.o.o.SystemOdpOperatorFactory     : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true

查看 OBProxy 的运行日志,也有连接失败的提示。

su - admin
cd /home/admin/logs/obproxy/log
vim obproxy.log

[2023-06-10 19:57:07.686198] DEBUG [PROXY.TXN] ob_mysql_transact.cpp:5353 [66787][Y0-00007F95319142E0] [lt=3] [dc=0] get config succ(addr={addr:"10.0.0.64:2883", vid:0}, cluster_name=obdem
o, tenant_name=proxysys, sm_->proxy_route_policy="", sm_->proxy_idc_name="", sm_->enable_cloud_full_username=false, sm_->enable_client_ssl=false, sm_->enable_server_ssl=false, ret=0)
[2023-06-10 19:57:07.686208] WARN  [PROXY.CS] is_authorised_proxysys (ob_mysql_client_session.cpp:1432) [66787][Y0-00007F95319142E0] [lt=8] [dc=0] root@proxysys check failed(ret=-4043)
[2023-06-10 19:57:07.686226] WARN  [PROXY.SM] analyze_login_request (ob_mysql_sm.cpp:2158) [66787][Y0-00007F95319142E0] [lt=17] [dc=0] fail to check user identity(sm_id=3211, ret=-4042)
[2023-06-10 19:57:07.686240] INFO  [PROXY.SM] ob_mysql_sm.cpp:2212 [66787][Y0-00007F95319142E0] [lt=4] [dc=0] client login audit(client_addr={10.0.0.64:47734}, hsr.cluster_name_=obdemo, hs
r.tenant_name_=proxysys, hsr.user_name_=root, status="failed")

查看 OBProxy 的连接,

(root@proxysys@127.1:2883) [(none)]> show processlist;
+---------+----------+------+------------------+-----------+-------------+-------------------+-------------------+-------+-------+
| Id      | Tenant   | User | Host             | db        | trans_count | svr_session_count | state             | tid   | pid   |
+---------+----------+------+------------------+-----------+-------------+-------------------+-------------------+-------+-------+
|  262235 | proxysys | root | 127.0.0.1:50473  | NULL      |           0 |                 0 | MCS_ACTIVE_READER | 66810 | 66787 |
|  262234 | sys      | root | 10.0.0.161:40498 | mysql     |           0 |                 1 | MCS_ACTIVE_READER | 66810 | 66787 |
|  524713 | obmysql  | tpcc | 10.0.0.46:23932  | tpccdb    |           0 |                 2 | MCS_ACTIVE_READER | 66811 | 66787 |
|  524712 | obmysql  | tpcc | 10.0.0.46:23928  | tpccdb    |           0 |                 2 | MCS_ACTIVE_READER | 66811 | 66787 |
|  786453 | obmysql  | tpcc | 10.0.0.46:23934  | tpccdb    |           0 |                 1 | MCS_ACTIVE_READER | 66812 | 66787 |
|  786452 | obmysql  | root | 10.0.0.107:53528 | oceanbase |           0 |                 1 | MCS_ACTIVE_READER | 66812 | 66787 |
| 1048608 | obmysql  | tpcc | 10.0.0.46:23936  | tpccdb    |           0 |                 1 | MCS_ACTIVE_READER | 66813 | 66787 |
| 1048607 | obmysql  | root | 10.0.0.107:58186 | oceanbase |           0 |                 1 | MCS_ACTIVE_READER | 66813 | 66787 |
| 1310722 | obmysql  | root | 10.0.0.107:58188 | oceanbase |           0 |                 1 | MCS_ACTIVE_READER | 66814 | 66787 |
| 1572967 | obmysql  | root | 10.0.0.113:10342 | mysql     |           0 |                 1 | MCS_ACTIVE_READER | 66815 | 66787 |
| 1835167 | obmysql  | root | 10.0.0.113:62159 | mysql     |           0 |                 1 | MCS_ACTIVE_READER | 66816 | 66787 |
| 2097173 | obmysql  | root | 10.0.0.113:10344 | mysql     |           0 |                 1 | MCS_ACTIVE_READER | 66817 | 66787 |
| 2359544 | sys      | root | 10.0.0.161:40494 | mysql     |           0 |                 1 | MCS_ACTIVE_READER | 66818 | 66787 |
|    2401 | obmysql  | tpcc | 10.0.0.46:23930  | tpccdb    |           0 |                 1 | MCS_ACTIVE_READER | 66787 | 66787 |
|    2400 | sys      | root | 10.0.0.161:40496 | mysql     |           0 |                 1 | MCS_ACTIVE_READER | 66787 | 66787 |
+---------+----------+------+------------------+-----------+-------------+-------------------+-------------------+-------+-------+
15 rows in set (0.02 sec)

(root@proxysys@127.1:2883) [(none)]> show proxysession attribute 524713 ;
+----------------------------------+--------------------+----------------+
| attribute_name                   | value              | info           |
+----------------------------------+--------------------+----------------+
| proxy_sessid                     | 720576202561225109 | cs common      |
| cs_id                            | 524713             | cs common      |
| cluster                          | obdemo             | cs common      |
| tenant                           | obmysql            | cs common      |
| user                             | tpcc               | cs common      |
| host_ip                          | 10.0.0.46          | cs common      |
| host_port                        | 23932              | cs common      |
| db                               | tpccdb             | cs common      |
| total_trans_cnt                  | 0                  | cs common      |
| svr_session_cnt                  | 2                  | cs common      |
| active                           | true               | cs common      |
| read_state                       | MCS_ACTIVE_READER  | cs common      |
| tid                              | 66811              | cs common      |
| pid                              | 66787              | cs common      |
| idc_name                         |                    | cs common      |
| modified_time                    | 0                  | cs stat        |
| reported_time                    | 0                  | cs stat        |
| hot_sys_var_version              | 1                  | cs var version |
| sys_var_version                  | 9                  | cs var version |
| user_var_version                 | 0                  | cs var version |
| last_insert_id_version           | 0                  | cs var version |
| db_name_version                  | 1                  | cs var version |
| server_ip                        | 10.0.0.62          | curr used ss   |
| server_port                      | 2881               | curr used ss   |
| server_sessid                    | 3221978748         | curr used ss   |
| ss_id                            | 1524               | curr used ss   |
| state                            | MSS_ACTIVE         | curr used ss   |
| transact_count                   | 160                | curr used ss   |
| server_trans_stat                | 1                  | curr used ss   |
| hot_sys_var_version              | 1                  | curr used ss   |
| sys_var_version                  | 9                  | curr used ss   |
| user_var_version                 | 0                  | curr used ss   |
| last_insert_id_version           | 0                  | curr used ss   |
| db_name_version                  | 1                  | curr used ss   |
| is_checksum_supported            | 1                  | curr used ss   |
| is_safe_read_weak_supported      | 0                  | curr used ss   |
| is_checksum_switch_supported     | 1                  | curr used ss   |
| checksum_switch                  | 1                  | curr used ss   |
| enable_extra_ok_packet_for_stats | 1                  | curr used ss   |
| server_ip                        | 10.0.0.61          | ss pool [0]    |
| server_port                      | 2881               | ss pool [0]    |
| server_sessid                    | 3221724216         | ss pool [0]    |
| ss_id                            | 1521               | ss pool [0]    |
| state                            | MSS_KA_SHARED      | ss pool [0]    |
| transact_count                   | 5                  | ss pool [0]    |
| server_trans_stat                | 0                  | ss pool [0]    |
| hot_sys_var_version              | 1                  | ss pool [0]    |
| sys_var_version                  | 9                  | curr used ss   |
| user_var_version                 | 0                  | curr used ss   |
| last_insert_id_version           | 0                  | curr used ss   |
| db_name_version                  | 1                  | curr used ss   |
| is_checksum_supported            | 1                  | curr used ss   |
| is_safe_read_weak_supported      | 0                  | curr used ss   |
| is_checksum_switch_supported     | 1                  | curr used ss   |
| checksum_switch                  | 1                  | curr used ss   |
| enable_extra_ok_packet_for_stats | 1                  | curr used ss   |
| server_ip                        | 10.0.0.61          | ss pool [0]    |
| server_port                      | 2881               | ss pool [0]    |
| server_sessid                    | 3221724216         | ss pool [0]    |
| ss_id                            | 1521               | ss pool [0]    |
| state                            | MSS_KA_SHARED      | ss pool [0]    |
| transact_count                   | 5                  | ss pool [0]    |
| server_trans_stat                | 0                  | ss pool [0]    |
| hot_sys_var_version              | 1                  | ss pool [0]    |
| sys_var_version                  | 9                  | ss pool [0]    |
| user_var_version                 | 0                  | ss pool [0]    |
| last_insert_id_version           | 0                  | ss pool [0]    |
| db_name_version                  | 1                  | ss pool [0]    |
| is_checksum_supported            | 1                  | ss pool [0]    |
| is_safe_read_weak_supported      | 0                  | ss pool [0]    |
| is_checksum_switch_supported     | 1                  | ss pool [0]    |
| checksum_switch                  | 1                  | ss pool [0]    |
| enable_extra_ok_packet_for_stats | 1                  | ss pool [0]    |
+----------------------------------+--------------------+----------------+
56 rows in set (0.01 sec)

问题结论

登录 OBProxy 管理用户,可以查看管理连接,管理机房属性,路由策略等。

官网提供下载的 OCP 版本可能太老(版本号: 3.3.4-20221201 发布日期: 2022年12月2日), 在 OBProxy 的管理密码修改这块功能估计有 BUG 。不知道最新的企业版是否修复了这个问题。

OBProxy 的管理员密码 obproxy_sys_password 对 OB 连接安全也很重要。必要的时候可以通过另外一个参数 skip_proxy_sys_private_check 关闭这个账户的远程登录。

(root@proxysys@127.1:2883) [(none)]> show proxyconfig like '%skip_proxy_sys_private_check%';
+------------------------------+-------+------------------------------+-------------+---------------+
| name                         | value | info                         | need_reboot | visible_level |
+------------------------------+-------+------------------------------+-------------+---------------+
| skip_proxy_sys_private_check | True  | skip_proxy_sys_private_check | false       | USER          |
+------------------------------+-------+------------------------------+-------------+---------------+
1 row in set (0.00 sec)

(root@proxysys@127.1:2883) [(none)]> alter proxyconfig set skip_proxy_sys_private_check=false;
Query OK, 0 rows affected (0.01 sec)

(root@proxysys@127.1:2883) [(none)]> show proxyconfig like '%skip_proxy_sys_private_check%';
+------------------------------+-------+------------------------------+-------------+---------------+
| name                         | value | info                         | need_reboot | visible_level |
+------------------------------+-------+------------------------------+-------------+---------------+
| skip_proxy_sys_private_check | False | skip_proxy_sys_private_check | false       | USER          |
+------------------------------+-------+------------------------------+-------------+---------------+
1 row in set (0.00 sec)
[admin@server061 obproxy]$ mysql -h127.1 -uroot@proxysys -P2883 -paaAA11__  -e "show proxyinfo idc"
+-----------------+--------------+-----------------+--------------+--------------------------------------+-------------+--------------+
| global_idc_name | cluster_name | match_type      | regions_name | same_idc                             | same_region | other_region |
+-----------------+--------------+-----------------+--------------+--------------------------------------+-------------+--------------+
|                 | obdemo       | MATCHED_BY_NONE | []           | [[0]"zone1", [1]"ZONE2", [2]"ZONE3"] | []          | []           |
+-----------------+--------------+-----------------+--------------+--------------------------------------+-------------+--------------+
[admin@server061 obproxy]$ mysql -h10.0.0.61 -uroot@proxysys -P2883 -paaAA11__  -e "show proxyinfo idc"
ERROR 1045 (42000): Access denied for user 'root@proxysys'@'10.0.0.61' (using password: YES)

其他参考