obdiag version: 3.6.0 report time: 2026-03-13 17:08:37 observer version: 4.3.5.4 observer commit id: 4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42) +------------------------------------------------------------------------------------------------------+ | fail-tasks-report | +-----------------------------------+------------------------------------------------------------------+ | task | task_report | +-----------------------------------+------------------------------------------------------------------+ | network.log_easy_slow | [fail] Failed to parse EASY SLOW count on remote_192_168_2_5: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_6: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_8: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_9: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_7: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_10: 0 | | | 0 | | network.network_write_cond_wakeup | [fail] Failed to parse wakeup count on remote_192_168_2_9: 0 | | | 0 | | | [fail] Failed to parse wakeup count on remote_192_168_2_5: 0 | | | 0 | | | [fail] Failed to parse wakeup count on remote_192_168_2_7: 0 | | | 0 | | | [fail] Failed to parse wakeup count on remote_192_168_2_6: 0 | | | 0 | +-----------------------------------+------------------------------------------------------------------+ +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | critical-tasks-report | +-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | task | task_report | +-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | bugs.bug_469 | [critical] [remote_192_168_2_9] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_8] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_7] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_5] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_10] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_6] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | cluster.session_limit | [critical] tenant: epean. The number of connections for the tenant is 5637 | | disk.mount_disk_full | [critical] node: remote_192_168_2_8 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_6 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_6 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_7 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_7 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_5 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_10 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_5 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_9 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_10 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_9 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_8 mount point: /data disk usage: 92%, over 90% | | disk.xfs_repair | [critical] [remote_192_168_2_10] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_9] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_7] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_8] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_6] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_5] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | network.TCP-retransmission | [critical] [remote_192_168_2_10] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_9] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_5] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_7] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_8] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_6] tsar is not installed. we can not check tcp retransmission. | | network.network_drop | [critical] [remote_192_168_2_5] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_9] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_8] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_6] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_7] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_10] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | network.network_write_cond_wakeup | [critical] Found 788 'write cond wakeup' occurrences in observer.log on remote_192_168_2_10. This indicates potential network issues between client and OBServer. Please check network connectivity and performance. | | | [critical] Found 73 'write cond wakeup' occurrences in observer.log on remote_192_168_2_8. This indicates potential network issues between client and OBServer. Please check network connectivity and performance. | | system.instruction_set_avx | [critical] CPU on remote_192_168_2_10 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_6 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_9 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_7 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_8 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_5 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | system.mount_options | [critical] node: remote_192_168_2_5 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_7 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_8 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_8 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_10 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_10 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_6 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_7 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_9 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_5 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_9 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_6 /obbackup mount option lookupcache=positive is not exist | | ls.paxo_members | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525077584,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525061332,192.168.2.10:2882:1 | | | [critical] tenant_id:1005 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740514856948,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525077584,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525077584,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525052044,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525070775,192.168.2.10:2882:1 | | | [critical] tenant_id:1005 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740514856948,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525061332,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525052044,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525042879,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525042879,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525052044,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525070775,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525042879,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525061332,192.168.2.10:2882:1 | | | [critical] tenant_id:1005 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740514856948,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525070775,192.168.2.10:2882:1 | +-----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | warning-tasks-report | +-------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ | task | task_report | +-------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ | cluster.cgroup | [warning] svr_ip: None enable_cgroup is False, recommended is True | | cluster.freeze_trigger_percentage | [warning] Found incorrect freeze_trigger_percentage(!=20): | | | Tenant 1004 has incorrect freeze_trigger_percentage on nodes: 192.168.2.5, 192.168.2.6, 192.168.2.7 | | | Tenant 1006 has incorrect freeze_trigger_percentage on nodes: 192.168.2.10, 192.168.2.8, 192.168.2.9 | | cluster.large_query_threshold | [warning] svr_ip: 192.168.2.10. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.8. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.7. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.6. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.5. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.9. large_query_threshold is 10s, recommended value is 5s. | | cluster.memory_limit_percentage | [warning] svr_ip: 192.168.2.10 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.6 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.9 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.7 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.8 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.5 memory_limit_percentage is 90, default value is 80 | | cluster.memstore_limit_percentage | [warning] svr_ip: 192.168.2.6. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.5. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.7. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.8. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.9. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.10. memstore_limit_percentage is 20, recommended value is 0 or 50. | | cluster.mod_too_large | [warning] [cluster:obcluster] mod max memory over 10G，Please check on oceanbase.__all_virtual_memory_info to find some large mod | | cluster.sys_log_level | [warning] [cluster:obcluster] sys_log_level: WARN. on 4.x, the recommended value for sys_log_level is WDIAG | | cluster.trace_log_slow_query_watermark | [warning] ip: 192.168.2.5, 192.168.2.7, 192.168.2.6, 192.168.2.10, 192.168.2.9, 192.168.2.8 trace_log_slow_query_watermark not in [1,2),please check | | disk.disk_full | [warning] [cluster:obcluster] log disk usage exceeds 80% of capacity | | system.check_command | [warning] node: remote_192_168_2_8 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_10 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_5 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_6 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_9 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_7 mtr is not existed. Network testing tools. Suggested installation | | system.clock_source_check | [warning] node:remote_192_168_2_5. chronyc is not running. | | | [warning] node:remote_192_168_2_7. chronyc is not running. | | | [warning] node:remote_192_168_2_10. chronyc is not running. | | | [warning] node:remote_192_168_2_6. chronyc is not running. | | | [warning] node:remote_192_168_2_9. chronyc is not running. | | | [warning] node:remote_192_168_2_8. chronyc is not running. | | | [warning] No clock source configuration found. | | system.dmesg_log | [warning] node:remote_192_168_2_7. dmesg log is empty. | | | [warning] node:remote_192_168_2_8. dmesg log is empty. | | | [warning] node:remote_192_168_2_10. dmesg log is empty. | | | [warning] node:remote_192_168_2_5. dmesg log is empty. | | | [warning] node:remote_192_168_2_6. dmesg log is empty. | | | [warning] node:remote_192_168_2_9. dmesg log is empty. | | system.parameter | [warning] [remote_192_168_2_6] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_8] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_10] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_5] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_9] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_7] kernel.numa_balancing : 1. recommended: 0. | | system.parameter_ip_local_port_range | [warning] [remote_192_168_2_9] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_5] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_10] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_8] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_10] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_8] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_5] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_9] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_6] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_7] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_6] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_7] ip_local_port_range_max : 60999. recommended: 65535 | | system.tcp_tw_reuse | [warning] node: remote_192_168_2_9 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_6 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_5 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_10 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_8 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_7 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | tenant.macroblock_utilization_rate_tenant | [warning] tenant: META$1004 ratio: 0.86, dataSize: 5.18G, requiredSize: 6.04G. need major | | | [warning] tenant: META$1006 ratio: 0.88, dataSize: 12.12G, requiredSize: 13.82G. need major | | | [warning] tenant: epean ratio: 1.07, dataSize: 4272.34G, requiredSize: 4011.04G. need major | | | [warning] tenant: mengxuan ratio: 0.99, dataSize: 2712.31G, requiredSize: 2743.01G. need major | | tenant.parameters_default | [warning] the observer_id value: 6, default_value: 0 | | | [warning] the cluster value: oa, default_value: obcluster | | | [warning] the enable_cgroup value: False, default_value: True | | | [warning] the log_storage_compress_all tenant_ids: 1,1004, value: True, default_value: False | | | [warning] the devname value: bond0.20, default_value: bond0 | | | [warning] the max_syslog_file_count value: 1000, default_value: 0 | | | [warning] the observer_id value: 5, default_value: 0 | | | [warning] the ob_enable_batched_multi_statement tenant_ids: 1004,1006, value: True, default_value: False | | | [warning] the data_dir value: /home/admin/oceanbase/store/oa, default_value: store | | | [warning] the syslog_level value: WARN, default_value: WDIAG | | | [warning] the enable_record_trace_log value: False, default_value: True | | | [warning] the log_archive_concurrency tenant_ids: 1004, value: 10, default_value: 0 | | | [warning] the trace_log_slow_query_watermark value: 15s, default_value: 1s | | | [warning] the cpu_quota_concurrency tenant_ids: 1, value: 10, default_value: 4 | | | [warning] the writing_throttling_trigger_percentage tenant_ids: 1004,1006, value: 90, default_value: 60 | | | [warning] the memory_limit_percentage value: 90, default_value: 80 | | | [warning] the alert_log_level value: WARN, default_value: INFO | | | [warning] the observer_id value: 1, default_value: 0 | | | [warning] the observer_id value: 3, default_value: 0 | | | [warning] the enable_syslog_recycle value: True, default_value: False | | | [warning] the memstore_limit_percentage value: 20, default_value: 0 | | | [warning] the enable_rebalance tenant_ids: 1, value: False, default_value: True | | | [warning] the _trace_control_info tenant_ids: 1006, value: { | | | "type_t": { | | | "level":1, | | | "sample_pct":"0.10", | | | "record_policy":"SAMPLE_AND_SLOW_QUERY" | | | } | | | }, default_value: {"type_t":{"level":1,"sample_pct":"0.10","record_policy":"SAMPLE_AND_SLOW_QUERY"}} | | | [warning] the observer_id value: 2, default_value: 0 | | | [warning] the internal_sql_execute_timeout value: 5m, default_value: 30s | | | [warning] the cluster_id value: 1749632252, default_value: 0 | | | [warning] the large_query_threshold value: 10s, default_value: 5s | | | [warning] the observer_id value: 4, default_value: 0 | | | [warning] the sys_bkgd_net_percentage value: 30, default_value: 60 | | | [warning] the sql_plan_management_mode tenant_ids: 1004,1006, value: OnlineEvolve, default_value: Disable | | | [warning] the _enable_dbms_job_package value: False, default_value: True | | | [warning] the rpc_timeout value: 5m, default_value: 2s | | | [warning] the major_freeze_duty_time tenant_ids: 1004,1006, value: 22:00, default_value: 02:00 | | | [warning] the _bloom_filter_ratio value: 3, default_value: 35 | | | [warning] the ob_vector_memory_limit_percentage tenant_ids: 1004,1006, value: 10, default_value: 0 | | | [warning] the freeze_trigger_percentage tenant_ids: 1004,1006, value: 50, default_value: 20 | | | [warning] the config_additional_dir value: /redo/oa/etc2;/data/oa/etc3, default_value: etc2;etc3 | | | [warning] the _parallel_ddl_control tenant_ids: 1004,1006, value: TRUNCATE_TABLE:ON, default_value: | +-------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | all-tasks-report | +---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | task | task_report | +---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | bugs.bug_182 | all pass | | bugs.bug_385 | all pass | | bugs.bug_469 | [critical] [remote_192_168_2_9] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_8] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_7] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_5] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_10] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | | [critical] [remote_192_168_2_6] glibc version need less 2.34. This may cause observer crash. More information： https://github.com/oceanbase/obdiag/issues/469 | | bugs.cgroup_kernel_bad_version | all pass | | clog.clog_disk_full | all pass | | clog.clog_hang | all pass | | cluster.no_leader | all pass | | cluster.auto_increment_cache_size | all pass | | cluster.ob_query_timeout | all pass | | cluster.autoinc_cache_refresh_interval | all pass | | cluster.cgroup | [warning] svr_ip: None enable_cgroup is False, recommended is True | | cluster.session_limit | [critical] tenant: epean. The number of connections for the tenant is 5637 | | cluster.clog_sync_time_warn_threshold | all pass | | cluster.core_file_find | all pass | | cluster.cpu_quota_concurrency | all pass | | cluster.data_path_settings | all pass | | cluster.datafile_next | all pass | | cluster.deadlocks | all pass | | cluster.default_compress_func | all pass | | cluster.enable_lock_priority | all pass | | cluster.freeze_trigger_percentage | [warning] Found incorrect freeze_trigger_percentage(!=20): | | | Tenant 1004 has incorrect freeze_trigger_percentage on nodes: 192.168.2.5, 192.168.2.6, 192.168.2.7 | | | Tenant 1006 has incorrect freeze_trigger_percentage on nodes: 192.168.2.10, 192.168.2.8, 192.168.2.9 | | cluster.tenant_locks | all pass | | cluster.global_indexes_too_much | all pass | | cluster.large_query_threshold | [warning] svr_ip: 192.168.2.10. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.8. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.7. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.6. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.5. large_query_threshold is 10s, recommended value is 5s. | | | [warning] svr_ip: 192.168.2.9. large_query_threshold is 10s, recommended value is 5s. | | cluster.ls_nu | all pass | | cluster.major | all pass | | cluster.memory_chunk_cache_size | all pass | | cluster.memory_limit_percentage | [warning] svr_ip: 192.168.2.10 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.6 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.9 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.7 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.8 memory_limit_percentage is 90, default value is 80 | | | [warning] svr_ip: 192.168.2.5 memory_limit_percentage is 90, default value is 80 | | cluster.memstore_limit_percentage | [warning] svr_ip: 192.168.2.6. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.5. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.7. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.8. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.9. memstore_limit_percentage is 20, recommended value is 0 or 50. | | | [warning] svr_ip: 192.168.2.10. memstore_limit_percentage is 20, recommended value is 0 or 50. | | cluster.mod_too_large | [warning] [cluster:obcluster] mod max memory over 10G，Please check on oceanbase.__all_virtual_memory_info to find some large mod | | cluster.memstore_usage | all pass | | cluster.ob_enable_plan_cache_bad_version | all pass | | cluster.ob_enable_prepared_statement | all pass | | cluster.observer_not_active | all pass | | cluster.observer_port | all pass | | cluster.maojor_suspended | all pass | | cluster.optimizer_better_inlist_costing_parmmeter | all pass | | cluster.part_trans_action_max | all pass | | cluster.resource_limit_max_session_num | all pass | | cluster.server_permanent_offline_time | all pass | | cluster.sys_log_level | [warning] [cluster:obcluster] sys_log_level: WARN. on 4.x, the recommended value for sys_log_level is WDIAG | | cluster.sys_obcon_health | all pass | | cluster.syslog_io_bandwidth_limit | all pass | | cluster.table_history_too_many | all pass | | cluster.task_opt_stat | all pass | | cluster.task_opt_stat_gather_fail | all pass | | cluster.tenant_number | all pass | | cluster.trace_log_slow_query_watermark | [warning] ip: 192.168.2.5, 192.168.2.7, 192.168.2.6, 192.168.2.10, 192.168.2.9, 192.168.2.8 trace_log_slow_query_watermark not in [1,2),please check | | cluster.upgrade_finished | all pass | | cluster.upper_trans_version | all pass | | cluster.zone_not_active | all pass | | cluster.logons_check | [normal] [SKIP] This task only applies to versions before 4.2.1.4 | | cluster.logons_check | all pass | | cpu.oversold | all pass | | disk.clog_abnormal_file | all pass | | disk.disk_full | [warning] [cluster:obcluster] log disk usage exceeds 80% of capacity | | disk.disk_hole | all pass | | disk.disk_iops | all pass | | disk.mount_disk_full | [critical] node: remote_192_168_2_8 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_6 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_6 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_7 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_7 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_5 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_10 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_5 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_9 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_10 mount point: /data disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_9 mount point: /redo disk usage: 92%, over 90% | | | [critical] node: remote_192_168_2_8 mount point: /data disk usage: 92%, over 90% | | disk.sstable_abnormal_file | all pass | | disk.xfs_repair | [critical] [remote_192_168_2_10] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_9] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_7] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_8] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_6] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | | [critical] [remote_192_168_2_5] xfs need repair. Please check disk. xfs_repair_log: dmesg: 读取内核缓冲区失败: Operation not permitted | | disk.data_disk_full | all pass | | err_code.find_err_4000 | all pass | | err_code.find_err_4001 | all pass | | err_code.find_err_4012 | all pass | | err_code.find_err_4013 | all pass | | err_code.find_err_4015 | all pass | | err_code.find_err_4016 | all pass | | err_code.find_err_4103 | all pass | | err_code.find_err_4105 | all pass | | err_code.find_err_4108 | all pass | | err_code.find_err_4377 | all pass | | log.log_size | all pass | | log.log_size_with_ocp | all pass | | network.TCP-retransmission | [critical] [remote_192_168_2_10] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_9] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_5] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_7] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_8] tsar is not installed. we can not check tcp retransmission. | | | [critical] [remote_192_168_2_6] tsar is not installed. we can not check tcp retransmission. | | network.network_drop | [critical] [remote_192_168_2_5] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_9] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_8] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_6] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_7] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | | [critical] [remote_192_168_2_10] network: bond0.20 TX drop is not 0, please check by ip -s link show bond0.20 | | network.network_offset | all pass | | network.network_speed | all pass | | network.local_ip_check | all pass | | network.log_easy_slow | [fail] Failed to parse EASY SLOW count on remote_192_168_2_5: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_6: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_8: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_9: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_7: 0 | | | 0 | | | [fail] Failed to parse EASY SLOW count on remote_192_168_2_10: 0 | | | 0 | | network.network_speed_diff | all pass | | network.network_write_cond_wakeup | [fail] Failed to parse wakeup count on remote_192_168_2_9: 0 | | | 0 | | | [fail] Failed to parse wakeup count on remote_192_168_2_5: 0 | | | 0 | | | [fail] Failed to parse wakeup count on remote_192_168_2_7: 0 | | | 0 | | | [fail] Failed to parse wakeup count on remote_192_168_2_6: 0 | | | 0 | | | [critical] Found 788 'write cond wakeup' occurrences in observer.log on remote_192_168_2_10. This indicates potential network issues between client and OBServer. Please check network connectivity and performance. | | | [critical] Found 73 'write cond wakeup' occurrences in observer.log on remote_192_168_2_8. This indicates potential network issues between client and OBServer. Please check network connectivity and performance. | | system.aio | all pass | | system.arm_smmu | all pass | | system.check_command | [warning] node: remote_192_168_2_8 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_10 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_5 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_6 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_9 mtr is not existed. Network testing tools. Suggested installation | | | [warning] node: remote_192_168_2_7 mtr is not existed. Network testing tools. Suggested installation | | system.clock_source | all pass | | system.clock_source_check | [warning] node:remote_192_168_2_5. chronyc is not running. | | | [warning] node:remote_192_168_2_7. chronyc is not running. | | | [warning] node:remote_192_168_2_10. chronyc is not running. | | | [warning] node:remote_192_168_2_6. chronyc is not running. | | | [warning] node:remote_192_168_2_9. chronyc is not running. | | | [warning] node:remote_192_168_2_8. chronyc is not running. | | | [warning] No clock source configuration found. | | system.core_pattern | all pass | | system.dependent_software | all pass | | system.dependent_software_swapon | all pass | | system.dmesg_log | [warning] node:remote_192_168_2_7. dmesg log is empty. | | | [warning] node:remote_192_168_2_8. dmesg log is empty. | | | [warning] node:remote_192_168_2_10. dmesg log is empty. | | | [warning] node:remote_192_168_2_5. dmesg log is empty. | | | [warning] node:remote_192_168_2_6. dmesg log is empty. | | | [warning] node:remote_192_168_2_9. dmesg log is empty. | | system.getenforce | all pass | | system.instruction_set_avx | [critical] CPU on remote_192_168_2_10 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_6 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_9 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_7 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_8 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | | [critical] CPU on remote_192_168_2_5 does not support AVX instruction set. if you want to use observer, please upgrade the observer version to '4.2.5.6 or later' or '4.3.5.4 or later' or '4.4.1.0 or later' | | system.instruction_set_avx2 | all pass | | system.kernel_bad_version | all pass | | system.mount_options | [critical] node: remote_192_168_2_5 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_7 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_8 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_8 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_10 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_10 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_6 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_7 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_9 /obbackup mount option lookupcache=positive is not exist | | | [critical] node: remote_192_168_2_5 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_9 /obbackup mount option nfsvers=4.1 is not exist | | | [critical] node: remote_192_168_2_6 /obbackup mount option lookupcache=positive is not exist | | system.parameter | [warning] [remote_192_168_2_6] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_8] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_10] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_5] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_9] kernel.numa_balancing : 1. recommended: 0. | | | [warning] [remote_192_168_2_7] kernel.numa_balancing : 1. recommended: 0. | | system.parameter_ip_local_port_range | [warning] [remote_192_168_2_9] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_5] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_10] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_8] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_10] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_8] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_5] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_9] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_6] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_7] ip_local_port_range_min : 32768. recommended: 3500 | | | [warning] [remote_192_168_2_6] ip_local_port_range_max : 60999. recommended: 65535 | | | [warning] [remote_192_168_2_7] ip_local_port_range_max : 60999. recommended: 65535 | | system.parameter_tcp_rmem | all pass | | system.parameter_tcp_wmem | all pass | | system.python_version | all pass | | system.tcp_tw_reuse | [warning] node: remote_192_168_2_9 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_6 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_5 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_10 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_8 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | | [warning] node: remote_192_168_2_7 net.ipv4.tcp_tw_reuse is 2. It is recommended to set it to 1 | | system.ulimit_parameter | all pass | | table.information_schema_tables_two_data | all pass | | table.macroblock_utilization_rate_table | all pass | | tenant.macroblock_utilization_rate_tenant | [warning] tenant: META$1004 ratio: 0.86, dataSize: 5.18G, requiredSize: 6.04G. need major | | | [warning] tenant: META$1006 ratio: 0.88, dataSize: 12.12G, requiredSize: 13.82G. need major | | | [warning] tenant: epean ratio: 1.07, dataSize: 4272.34G, requiredSize: 4011.04G. need major | | | [warning] tenant: mengxuan ratio: 0.99, dataSize: 2712.31G, requiredSize: 2743.01G. need major | | tenant.max_stale_time_for_weak_consistency | all pass | | tenant.parameters_default | [warning] the observer_id value: 6, default_value: 0 | | | [warning] the cluster value: oa, default_value: obcluster | | | [warning] the enable_cgroup value: False, default_value: True | | | [warning] the log_storage_compress_all tenant_ids: 1,1004, value: True, default_value: False | | | [warning] the devname value: bond0.20, default_value: bond0 | | | [warning] the max_syslog_file_count value: 1000, default_value: 0 | | | [warning] the observer_id value: 5, default_value: 0 | | | [warning] the ob_enable_batched_multi_statement tenant_ids: 1004,1006, value: True, default_value: False | | | [warning] the data_dir value: /home/admin/oceanbase/store/oa, default_value: store | | | [warning] the syslog_level value: WARN, default_value: WDIAG | | | [warning] the enable_record_trace_log value: False, default_value: True | | | [warning] the log_archive_concurrency tenant_ids: 1004, value: 10, default_value: 0 | | | [warning] the trace_log_slow_query_watermark value: 15s, default_value: 1s | | | [warning] the cpu_quota_concurrency tenant_ids: 1, value: 10, default_value: 4 | | | [warning] the writing_throttling_trigger_percentage tenant_ids: 1004,1006, value: 90, default_value: 60 | | | [warning] the memory_limit_percentage value: 90, default_value: 80 | | | [warning] the alert_log_level value: WARN, default_value: INFO | | | [warning] the observer_id value: 1, default_value: 0 | | | [warning] the observer_id value: 3, default_value: 0 | | | [warning] the enable_syslog_recycle value: True, default_value: False | | | [warning] the memstore_limit_percentage value: 20, default_value: 0 | | | [warning] the enable_rebalance tenant_ids: 1, value: False, default_value: True | | | [warning] the _trace_control_info tenant_ids: 1006, value: { | | | "type_t": { | | | "level":1, | | | "sample_pct":"0.10", | | | "record_policy":"SAMPLE_AND_SLOW_QUERY" | | | } | | | }, default_value: {"type_t":{"level":1,"sample_pct":"0.10","record_policy":"SAMPLE_AND_SLOW_QUERY"}} | | | [warning] the observer_id value: 2, default_value: 0 | | | [warning] the internal_sql_execute_timeout value: 5m, default_value: 30s | | | [warning] the cluster_id value: 1749632252, default_value: 0 | | | [warning] the large_query_threshold value: 10s, default_value: 5s | | | [warning] the observer_id value: 4, default_value: 0 | | | [warning] the sys_bkgd_net_percentage value: 30, default_value: 60 | | | [warning] the sql_plan_management_mode tenant_ids: 1004,1006, value: OnlineEvolve, default_value: Disable | | | [warning] the _enable_dbms_job_package value: False, default_value: True | | | [warning] the rpc_timeout value: 5m, default_value: 2s | | | [warning] the major_freeze_duty_time tenant_ids: 1004,1006, value: 22:00, default_value: 02:00 | | | [warning] the _bloom_filter_ratio value: 3, default_value: 35 | | | [warning] the ob_vector_memory_limit_percentage tenant_ids: 1004,1006, value: 10, default_value: 0 | | | [warning] the freeze_trigger_percentage tenant_ids: 1004,1006, value: 50, default_value: 20 | | | [warning] the config_additional_dir value: /redo/oa/etc2;/data/oa/etc3, default_value: etc2;etc3 | | | [warning] the _parallel_ddl_control tenant_ids: 1004,1006, value: TRUNCATE_TABLE:ON, default_value: | | tenant.tenant_min_resource | all pass | | tenant.writing_throttling_trigger_percentage | all pass | | tenant.tenant_threshold | all pass | | version.bad_version | all pass | | version.old_version | all pass | | index.global_index_unpartitioned | all pass | | ls.paxo_members | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525077584,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525061332,192.168.2.10:2882:1 | | | [critical] tenant_id:1005 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740514856948,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525077584,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525077584,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525052044,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525070775,192.168.2.10:2882:1 | | | [critical] tenant_id:1005 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740514856948,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525061332,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525052044,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525042879,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.7:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525042879,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525052044,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525070775,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525042879,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.6:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525061332,192.168.2.10:2882:1 | | | [critical] tenant_id:1005 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740514856948,192.168.2.10:2882:1 | | | [critical] tenant_id:1006 ls:192.168.2.5:2882:1 is not in paxos_member_list:192.168.2.8:2882:1,192.168.2.9:2882:1752740525070775,192.168.2.10:2882:1 | +---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+