问题现象
OBProxy 有些自己的参数,需要通过用户 root@proxysys
和 OBProxy 的 IP 和 2883(默认 OBProxy 连接端口) 访问获取。
如果 OBProxy 是独立使用 RPM 包部署的时候,默认这个管理员密码为空,需要自己设置一个密码。 这个在以前的试验里介绍过(见文后参考)。但是如果是 OCP 部署的 OBProxy ,是会设置一个密码。 这个密码的密文会出现在 OBProxy 的配置文件 (obproxy_config.bin
) 的配置(obproxy_sys_password
) 中,并且是密文形式出现。反复部署 OB 集群和 OBProxy 观察发现,这个密码密文内容是一样的(1)。
OCP 提供了 OBProxy 的参数修改功能,可以改掉 obproxy_sys_password
的密码。
修改一下这个密码。
密码修改会生成一个任务,任何很快就会成功。
从任务日志里可以确认修改密码的方法,包括密码内容(2)。
密码修改成功后,可以在命令行下测试。进入 OBProxy 里可以分析 OBProxy 的连接信息,对理解 OBProxy 的原理非常有用。具体见文后参考。
但是,回到 OCP 的 OBProxy 界面开始有报错现象。
OBProxy 节点出现不可用提示。
查看 OBProxy 的参数页面报错。由此推断这些参数是 OCP 连接到 OBProxy 里去实时获取的。只不过 OCP 连接 OBProxy 管理用户报错了。
报错信息: 无法连接到 obproxy 10.0.0.61:2883,可能由于 obproxy 进程不存在、错误的 proxysys 密码或者 obproxy 参数 skip_proxy_sys_private_check 不为 true 引起
问题分析
这些 OBProxy 是可以用的,业务通过 OBProxy 连接 OB 集群依然是正常的。问题出在 OCP 连接 OBProxy 环节。
为此,分析一下 OCP 修改 OBProxy 管理密码的过程 。
在 OCP 元数据库里,能找到密码修改的历史。
SELECT obproxy_cluster_id ,version,name,value_before ,value_after ,update_time FROM obproxy_parameter_change ORDER BY version,name DESC ;
obproxy_cluster_id | version | name | value_before | value_after | update_time |
---|---|---|---|---|---|
1000015 | 2 | obproxy_sys_password | f0455d5edb4833fa2e91762e71b61bc35d45cf0b | aaAA11__ | 2023-06-10 19:26:08.000 |
查看这个日志可知 value_before
是密文,value_after
是明文(3) 。
SELECT * FROM obproxy_parameter WHERE obproxy_cluster_id =1000015;
id | obproxy_cluster_id | version | name | value | create_time | update_time |
---|---|---|---|---|---|---|
1000006 | 1000015 | 2 | obproxy_sys_password | aaAA11__ | 2023-06-10 19:26:08.000 | 2023-06-10 19:26:08.000 |
1000005 | 1000015 | 1 | syslog_level | DEBUG | 2023-06-10 19:19:17.000 | 2023-06-10 19:19:17.000 |
这里保存的也是明文(4)。
是否是 OCP BUG 导致修改密码后 OCP 连接不了 OBProxy。
尝试更新一下 OCP 元数据。
UPDATE obproxy_parameter_change SET value_after='ce77bdb3aad23ee556c638ea6ed86bad4949f676' where obproxy_cluster_id=1000015 and name='obproxy_sys_password';
UPDATE obproxy_parameter SET value='ce77bdb3aad23ee556c638ea6ed86bad4949f676' WHERE name='obproxy_sys_password' AND obproxy_cluster_id=1000015;
COMMIT;
刷新 OCP 页面查看 OBProxy 配置信息,依然报错。
重启 OCP 容器排除服务端缓存密码问题,再次查看 OBProxy 页面,依然报错。
查看 OCP 的运行日志,也看到连接失败的提示。
docker exec -it ocp bash
cd /home/admin/logs
vi ocp-server.0.out
2023-06-10 19:54:17.027 WARN 46 --- [pool-Check obproxy2,328de6b9191f4c81,ea147e756de8] c.a.ocp.core.odpsdk.OdpJdbcOperator : Failed to check odp connection Connection {driverClassName ='com.oceanbase.jdbc.Driver', connectionProperties ='[user=root@proxysys, url=jdbc:oceanbase://10.0.0.61:2883/?connectProxy=true]'}, error is Failed to obtain JDBC Connection; nested exception is java.sql.SQLSyntaxErrorException: Could not connect to address=(host=10.0.0.61)(port=2883)(type=master) : (conn=2006) Access denied for user 'root@proxysys'@'10.0.0.64' (using password: NO)
2023-06-10 19:54:17.027 WARN 46 --- [pool-Check obproxy2,328de6b9191f4c81,ea147e756de8] c.a.o.s.o.o.SystemOdpOperatorFactory : Failed to connect obproxy server, maybe caused by odp process down, wrong password, or ocp address not in private ip list and odp parameter skip_proxy_sys_private_check is not true
查看 OBProxy 的运行日志,也有连接失败的提示。
su - admin
cd /home/admin/logs/obproxy/log
vim obproxy.log
[2023-06-10 19:57:07.686198] DEBUG [PROXY.TXN] ob_mysql_transact.cpp:5353 [66787][Y0-00007F95319142E0] [lt=3] [dc=0] get config succ(addr={addr:"10.0.0.64:2883", vid:0}, cluster_name=obdem
o, tenant_name=proxysys, sm_->proxy_route_policy="", sm_->proxy_idc_name="", sm_->enable_cloud_full_username=false, sm_->enable_client_ssl=false, sm_->enable_server_ssl=false, ret=0)
[2023-06-10 19:57:07.686208] WARN [PROXY.CS] is_authorised_proxysys (ob_mysql_client_session.cpp:1432) [66787][Y0-00007F95319142E0] [lt=8] [dc=0] root@proxysys check failed(ret=-4043)
[2023-06-10 19:57:07.686226] WARN [PROXY.SM] analyze_login_request (ob_mysql_sm.cpp:2158) [66787][Y0-00007F95319142E0] [lt=17] [dc=0] fail to check user identity(sm_id=3211, ret=-4042)
[2023-06-10 19:57:07.686240] INFO [PROXY.SM] ob_mysql_sm.cpp:2212 [66787][Y0-00007F95319142E0] [lt=4] [dc=0] client login audit(client_addr={10.0.0.64:47734}, hsr.cluster_name_=obdemo, hs
r.tenant_name_=proxysys, hsr.user_name_=root, status="failed")
查看 OBProxy 的连接,
(root@proxysys@127.1:2883) [(none)]> show processlist;
+---------+----------+------+------------------+-----------+-------------+-------------------+-------------------+-------+-------+
| Id | Tenant | User | Host | db | trans_count | svr_session_count | state | tid | pid |
+---------+----------+------+------------------+-----------+-------------+-------------------+-------------------+-------+-------+
| 262235 | proxysys | root | 127.0.0.1:50473 | NULL | 0 | 0 | MCS_ACTIVE_READER | 66810 | 66787 |
| 262234 | sys | root | 10.0.0.161:40498 | mysql | 0 | 1 | MCS_ACTIVE_READER | 66810 | 66787 |
| 524713 | obmysql | tpcc | 10.0.0.46:23932 | tpccdb | 0 | 2 | MCS_ACTIVE_READER | 66811 | 66787 |
| 524712 | obmysql | tpcc | 10.0.0.46:23928 | tpccdb | 0 | 2 | MCS_ACTIVE_READER | 66811 | 66787 |
| 786453 | obmysql | tpcc | 10.0.0.46:23934 | tpccdb | 0 | 1 | MCS_ACTIVE_READER | 66812 | 66787 |
| 786452 | obmysql | root | 10.0.0.107:53528 | oceanbase | 0 | 1 | MCS_ACTIVE_READER | 66812 | 66787 |
| 1048608 | obmysql | tpcc | 10.0.0.46:23936 | tpccdb | 0 | 1 | MCS_ACTIVE_READER | 66813 | 66787 |
| 1048607 | obmysql | root | 10.0.0.107:58186 | oceanbase | 0 | 1 | MCS_ACTIVE_READER | 66813 | 66787 |
| 1310722 | obmysql | root | 10.0.0.107:58188 | oceanbase | 0 | 1 | MCS_ACTIVE_READER | 66814 | 66787 |
| 1572967 | obmysql | root | 10.0.0.113:10342 | mysql | 0 | 1 | MCS_ACTIVE_READER | 66815 | 66787 |
| 1835167 | obmysql | root | 10.0.0.113:62159 | mysql | 0 | 1 | MCS_ACTIVE_READER | 66816 | 66787 |
| 2097173 | obmysql | root | 10.0.0.113:10344 | mysql | 0 | 1 | MCS_ACTIVE_READER | 66817 | 66787 |
| 2359544 | sys | root | 10.0.0.161:40494 | mysql | 0 | 1 | MCS_ACTIVE_READER | 66818 | 66787 |
| 2401 | obmysql | tpcc | 10.0.0.46:23930 | tpccdb | 0 | 1 | MCS_ACTIVE_READER | 66787 | 66787 |
| 2400 | sys | root | 10.0.0.161:40496 | mysql | 0 | 1 | MCS_ACTIVE_READER | 66787 | 66787 |
+---------+----------+------+------------------+-----------+-------------+-------------------+-------------------+-------+-------+
15 rows in set (0.02 sec)
(root@proxysys@127.1:2883) [(none)]> show proxysession attribute 524713 ;
+----------------------------------+--------------------+----------------+
| attribute_name | value | info |
+----------------------------------+--------------------+----------------+
| proxy_sessid | 720576202561225109 | cs common |
| cs_id | 524713 | cs common |
| cluster | obdemo | cs common |
| tenant | obmysql | cs common |
| user | tpcc | cs common |
| host_ip | 10.0.0.46 | cs common |
| host_port | 23932 | cs common |
| db | tpccdb | cs common |
| total_trans_cnt | 0 | cs common |
| svr_session_cnt | 2 | cs common |
| active | true | cs common |
| read_state | MCS_ACTIVE_READER | cs common |
| tid | 66811 | cs common |
| pid | 66787 | cs common |
| idc_name | | cs common |
| modified_time | 0 | cs stat |
| reported_time | 0 | cs stat |
| hot_sys_var_version | 1 | cs var version |
| sys_var_version | 9 | cs var version |
| user_var_version | 0 | cs var version |
| last_insert_id_version | 0 | cs var version |
| db_name_version | 1 | cs var version |
| server_ip | 10.0.0.62 | curr used ss |
| server_port | 2881 | curr used ss |
| server_sessid | 3221978748 | curr used ss |
| ss_id | 1524 | curr used ss |
| state | MSS_ACTIVE | curr used ss |
| transact_count | 160 | curr used ss |
| server_trans_stat | 1 | curr used ss |
| hot_sys_var_version | 1 | curr used ss |
| sys_var_version | 9 | curr used ss |
| user_var_version | 0 | curr used ss |
| last_insert_id_version | 0 | curr used ss |
| db_name_version | 1 | curr used ss |
| is_checksum_supported | 1 | curr used ss |
| is_safe_read_weak_supported | 0 | curr used ss |
| is_checksum_switch_supported | 1 | curr used ss |
| checksum_switch | 1 | curr used ss |
| enable_extra_ok_packet_for_stats | 1 | curr used ss |
| server_ip | 10.0.0.61 | ss pool [0] |
| server_port | 2881 | ss pool [0] |
| server_sessid | 3221724216 | ss pool [0] |
| ss_id | 1521 | ss pool [0] |
| state | MSS_KA_SHARED | ss pool [0] |
| transact_count | 5 | ss pool [0] |
| server_trans_stat | 0 | ss pool [0] |
| hot_sys_var_version | 1 | ss pool [0] |
| sys_var_version | 9 | curr used ss |
| user_var_version | 0 | curr used ss |
| last_insert_id_version | 0 | curr used ss |
| db_name_version | 1 | curr used ss |
| is_checksum_supported | 1 | curr used ss |
| is_safe_read_weak_supported | 0 | curr used ss |
| is_checksum_switch_supported | 1 | curr used ss |
| checksum_switch | 1 | curr used ss |
| enable_extra_ok_packet_for_stats | 1 | curr used ss |
| server_ip | 10.0.0.61 | ss pool [0] |
| server_port | 2881 | ss pool [0] |
| server_sessid | 3221724216 | ss pool [0] |
| ss_id | 1521 | ss pool [0] |
| state | MSS_KA_SHARED | ss pool [0] |
| transact_count | 5 | ss pool [0] |
| server_trans_stat | 0 | ss pool [0] |
| hot_sys_var_version | 1 | ss pool [0] |
| sys_var_version | 9 | ss pool [0] |
| user_var_version | 0 | ss pool [0] |
| last_insert_id_version | 0 | ss pool [0] |
| db_name_version | 1 | ss pool [0] |
| is_checksum_supported | 1 | ss pool [0] |
| is_safe_read_weak_supported | 0 | ss pool [0] |
| is_checksum_switch_supported | 1 | ss pool [0] |
| checksum_switch | 1 | ss pool [0] |
| enable_extra_ok_packet_for_stats | 1 | ss pool [0] |
+----------------------------------+--------------------+----------------+
56 rows in set (0.01 sec)
问题结论
登录 OBProxy 管理用户,可以查看管理连接,管理机房属性,路由策略等。
官网提供下载的 OCP 版本可能太老(版本号: 3.3.4-20221201 发布日期: 2022年12月2日), 在 OBProxy 的管理密码修改这块功能估计有 BUG 。不知道最新的企业版是否修复了这个问题。
OBProxy 的管理员密码 obproxy_sys_password
对 OB 连接安全也很重要。必要的时候可以通过另外一个参数 skip_proxy_sys_private_check
关闭这个账户的远程登录。
(root@proxysys@127.1:2883) [(none)]> show proxyconfig like '%skip_proxy_sys_private_check%';
+------------------------------+-------+------------------------------+-------------+---------------+
| name | value | info | need_reboot | visible_level |
+------------------------------+-------+------------------------------+-------------+---------------+
| skip_proxy_sys_private_check | True | skip_proxy_sys_private_check | false | USER |
+------------------------------+-------+------------------------------+-------------+---------------+
1 row in set (0.00 sec)
(root@proxysys@127.1:2883) [(none)]> alter proxyconfig set skip_proxy_sys_private_check=false;
Query OK, 0 rows affected (0.01 sec)
(root@proxysys@127.1:2883) [(none)]> show proxyconfig like '%skip_proxy_sys_private_check%';
+------------------------------+-------+------------------------------+-------------+---------------+
| name | value | info | need_reboot | visible_level |
+------------------------------+-------+------------------------------+-------------+---------------+
| skip_proxy_sys_private_check | False | skip_proxy_sys_private_check | false | USER |
+------------------------------+-------+------------------------------+-------------+---------------+
1 row in set (0.00 sec)
[admin@server061 obproxy]$ mysql -h127.1 -uroot@proxysys -P2883 -paaAA11__ -e "show proxyinfo idc"
+-----------------+--------------+-----------------+--------------+--------------------------------------+-------------+--------------+
| global_idc_name | cluster_name | match_type | regions_name | same_idc | same_region | other_region |
+-----------------+--------------+-----------------+--------------+--------------------------------------+-------------+--------------+
| | obdemo | MATCHED_BY_NONE | [] | [[0]"zone1", [1]"ZONE2", [2]"ZONE3"] | [] | [] |
+-----------------+--------------+-----------------+--------------+--------------------------------------+-------------+--------------+
[admin@server061 obproxy]$ mysql -h10.0.0.61 -uroot@proxysys -P2883 -paaAA11__ -e "show proxyinfo idc"
ERROR 1045 (42000): Access denied for user 'root@proxysys'@'10.0.0.61' (using password: YES)