Description
This article provides a method to resolve Clickhouse error when the broken parts size exceeds the configuration in merge tree settings.
Scope
FortiSIEM.
Solution
During replication of Replicas in Clickhouse cluster, users may encounter ClickHouse-server down. The error can be found in /opt/clickhouse/log/clickhouse-server.err.log:
Code: 231. DB::Exception: Suspiciously big size (6 parts, 1.59 GiB in total) of all broken parts to remove while maximum allowed broken parts size is 1.00 GiB. You can change the maximum value with merge tree setting 'max_suspicious_broken_parts_bytes'....
This error indicates that the merge tree settings have limited ClickHouse from proceeding with the broken logs. In that case, it can be resolved temporarily via the settings below:
#vi /etc/clickhouse-server/config.d/max_suspicious_broken_parts.xml
***Add line below***
#<max_suspicious_broken_parts_bytes>2073741824</max_suspicious_broken_parts_bytes>
#systemctl start clickhouse-server
Notes:
The bytes size of '2073741824' can be increased based on the error requirement in the environment. By default, FortiSIEM is configured as 1GB.
After the system is stable and replication is in sync, users is advised to revert the changes by removing the lines of config.
Related article:
Troubleshooting Tip: Resolving Upgrade Failure in ClickHouse Summary Table
