どうも、HAProxy 教日本支部東京分科会のかっぱ (@inokara)です。あまりに万能すぎる HAProxy に翻弄される毎日を送っています。
健康チェックしてますか？
HAProxy でバックエンドサーバーのヘルスチェックを行いたい場合には default セクションに以下を追加します。
log global option log-health-checks
あと、rsyslog 経由でログを出力したいので global セクションにログ出力の設定をしておきます。
log 127.0.0.1 local6
上記のように設定します。
その上で /etc/rsyslog.conf に以下を追記します。
local6.* -/var/log/haproxy/haproxy_healthcheck.log;Haproxy
ログファイル名は任意（適当に…）で。
どんなログが出るか…
バックエンドを停止すると…
バックエンドが停止するパターンとして…
- Web サーバー停止
- Web サーバーは生きているけど、ヘルスチェック用にコンテンツへアクセス出来ない（コンテンツ異常）
上記の二つを想定してヘルスチェックのログを見てみました。
尚、ヘルスチェックの設定は下記の通りです。
check fall 2 rise 2 inter 1000
- 2 回のチェックに失敗したら切り離し
- 2 回のチェックに成功したら切り戻し
- チェック間隔は 1000ms
Web サーバー停止
Health check for server backend_server/app01 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms, status: 1/2 UP. Health check for server backend_server/app01 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms, status: 1/2 UP. Health check for server backend_server/app01 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms, status: 0/2 DOWN. Server backend_server/app01 is DOWN. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Health check for server backend_server/app01 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms, status: 0/2 DOWN. Server backend_server/app01 is DOWN. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Web サーバー復旧
Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 1/2 DOWN. Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 1/2 DOWN. Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 2/2 UP. Server backend_server/app01 is UP. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue. Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 2/2 UP. Server backend_server/app01 is UP. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue.
ヘルスチェックコンテンツ異常
Health check for server backend_server/app01 failed, reason: Layer7 wrong status, code: 403, info: "Forbidden", check duration: 4ms, status: 1/2 UP. Health check for server backend_server/app01 failed, reason: Layer7 wrong status, code: 403, info: "Forbidden", check duration: 5ms, status: 1/2 UP. Health check for server backend_server/app01 failed, reason: Layer7 wrong status, code: 403, info: "Forbidden", check duration: 4ms, status: 0/2 DOWN. Server backend_server/app01 is DOWN. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Health check for server backend_server/app01 failed, reason: Layer7 wrong status, code: 403, info: "Forbidden", check duration: 4ms, status: 0/2 DOWN. Server backend_server/app01 is DOWN. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
コンテンツ復旧
Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 1/2 DOWN. Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 1/2 DOWN. Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 2/2 UP. Server backend_server/app01 is UP. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue. Health check for server backend_server/app01 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms, status: 2/2 UP. Server backend_server/app01 is UP. 2 active and 1 backup servers online. 0 sessions requeued, 0 total in queue.
ほうほう。このログを監視してアラート上げたりとか出来そうですな。
とりあえず
出来ればこのログは出力するようにしておきたいものです。
元記事はこちらです。
「HAProxy でバックエンドサーバーのヘルスチェックログを記録する」