https://chihhung.github.io/Blog/tags/%E5%91%8A%E8%AD%A6/Recent content in 告警 on Eric 個人部落格Hugo -- 0.151.0zh-TWMon, 18 May 2026 00:00:00 +0000https://chihhung.github.io/Blog/posts/%E6%95%99%E5%AD%B8/templates/operations/monitoringalertconfig_template/Mon, 18 May 2026 00:00:00 +0000https://chihhung.github.io/Blog/posts/%E6%95%99%E5%AD%B8/templates/operations/monitoringalertconfig_template/<h1 id="監控與告警設定文件範本monitoring--alert-configuration-template">監控與告警設定文件範本（Monitoring &amp; Alert Configuration Template）</h1> <blockquote> <p><strong>適用標準</strong>：Google SRE Workbook、ISO/IEC 20000-1:2018（Service Monitoring）<br> <strong>適用階段</strong>：維運階段（Operations Phase）<br> <strong>負責角色</strong>：SRE、DevOps Engineer、Tech Lead</p></blockquote> <hr> <h2 id="-章節目錄">📑 章節目錄</h2> <ol> <li><a href="#1-%E6%96%87%E4%BB%B6%E8%B3%87%E8%A8%8A">文件資訊</a></li> <li><a href="#2-%E7%9B%A3%E6%8E%A7%E7%AD%96%E7%95%A5%E6%A6%82%E8%A6%81">監控策略概要</a></li> <li><a href="#3-sli--slo-%E5%AE%9A%E7%BE%A9">SLI / SLO 定義</a></li> <li><a href="#4-%E7%9B%A3%E6%8E%A7%E6%8C%87%E6%A8%99%E8%A8%AD%E8%A8%88">監控指標設計</a></li> <li><a href="#5-%E5%91%8A%E8%AD%A6%E8%A6%8F%E5%89%87%E5%AE%9A%E7%BE%A9">告警規則定義</a></li> <li><a href="#6-dashboard-%E8%A8%AD%E8%A8%88">Dashboard 設計</a></li> <li><a href="#7-%E5%91%8A%E8%AD%A6%E9%80%9A%E7%9F%A5%E8%88%87%E5%8D%87%E7%B4%9A">告警通知與升級</a></li> <li><a href="#8-%E6%97%A5%E8%AA%8C%E7%9B%A3%E6%8E%A7%E7%AD%96%E7%95%A5">日誌監控策略</a></li> <li><a href="#9-%E7%B6%AD%E8%AD%B7%E8%88%87%E6%AA%A2%E8%A8%8E%E6%A9%9F%E5%88%B6">維護與檢討機制</a></li> <li><a href="#10-%E9%99%84%E9%8C%84">附錄</a></li> </ol> <hr> <h2 id="-範本">📝 範本</h2> <hr> <h3 id="1-文件資訊">1. 文件資訊</h3> <table> <thead> <tr> <th>項目</th> <th>內容</th> </tr> </thead> <tbody> <tr> <td><strong>文件名稱</strong></td> <td>[系統名稱] 監控與告警設定文件</td> </tr> <tr> <td><strong>文件編號</strong></td> <td>[專案代碼]-MON-[版本號]-[日期]</td> </tr> <tr> <td><strong>版本</strong></td> <td>v[X.Y]</td> </tr> <tr> <td><strong>建立日期</strong></td> <td>[YYYY-MM-DD]</td> </tr> <tr> <td><strong>負責人</strong></td> <td>[SRE / DevOps]</td> </tr> <tr> <td><strong>審核者</strong></td> <td>[Tech Lead / SA]</td> </tr> </tbody> </table> <hr> <h3 id="2-監控策略概要">2. 監控策略概要</h3> <h4 id="21-監控架構">2.1 監控架構</h4> <table> <thead> <tr> <th>層級</th> <th>工具</th> <th>用途</th> </tr> </thead> <tbody> <tr> <td>Infrastructure</td> <td>[Prometheus / CloudWatch / Azure Monitor]</td> <td>主機/容器/網路</td> </tr> <tr> <td>Application</td> <td>[APM: Datadog / New Relic / OpenTelemetry]</td> <td>應用效能</td> </tr> <tr> <td>Business</td> <td>[Custom metrics / Analytics]</td> <td>業務指標</td> </tr> <tr> <td>Logging</td> <td>[ELK / Loki / Azure Log Analytics]</td> <td>日誌分析</td> </tr> <tr> <td>Tracing</td> <td>[Jaeger / Zipkin / OpenTelemetry]</td> <td>分散式追蹤</td> </tr> </tbody> </table> <h4 id="22-監控原則">2.2 監控原則</h4> <table> <thead> <tr> <th>原則</th> <th>說明</th> </tr> </thead> <tbody> <tr> <td>Four Golden Signals</td> <td>Latency, Traffic, Errors, Saturation</td> </tr> <tr> <td>USE Method</td> <td>Utilization, Saturation, Errors（基礎設施）</td> </tr> <tr> <td>RED Method</td> <td>Rate, Errors, Duration（服務）</td> </tr> <tr> <td>Alerting Philosophy</td> <td>Alert on symptoms, not causes</td> </tr> </tbody> </table> <hr> <h3 id="3-sli--slo-定義">3. SLI / SLO 定義</h3> <h4 id="31-服務層級指標sli">3.1 服務層級指標（SLI）</h4> <table> <thead> <tr> <th>SLI</th> <th>定義</th> <th>量測方式</th> <th>資料來源</th> </tr> </thead> <tbody> <tr> <td>Availability</td> <td>成功回應比例</td> <td>success_requests / total_requests</td> <td>[Load Balancer / App]</td> </tr> <tr> <td>Latency</td> <td>回應時間分佈</td> <td>P50, P95, P99</td> <td>[APM / Prometheus histogram]</td> </tr> <tr> <td>Throughput</td> <td>處理量</td> <td>Requests per second</td> <td>[Metrics]</td> </tr> <tr> <td>Error Rate</td> <td>錯誤回應比例</td> <td>5xx_count / total_count</td> <td>[Ingress / App]</td> </tr> </tbody> </table> <h4 id="32-服務層級目標slo">3.2 服務層級目標（SLO）</h4> <table> <thead> <tr> <th>服務</th> <th>SLO 指標</th> <th>目標值</th> <th>Error Budget (月)</th> <th>量測視窗</th> </tr> </thead> <tbody> <tr> <td>[API Service]</td> <td>Availability</td> <td>≥ 99.9%</td> <td>43.8 min</td> <td>30 days rolling</td> </tr> <tr> <td>[API Service]</td> <td>P95 Latency</td> <td>&lt; [N]ms</td> <td>—</td> <td>30 days rolling</td> </tr> <tr> <td>[Web Frontend]</td> <td>Availability</td> <td>≥ 99.5%</td> <td>3.6 hr</td> <td>30 days rolling</td> </tr> <tr> <td>[Batch Job]</td> <td>Success Rate</td> <td>≥ 99.0%</td> <td>—</td> <td>Per execution</td> </tr> </tbody> </table> <hr> <h3 id="4-監控指標設計">4. 監控指標設計</h3> <h4 id="41-基礎設施指標">4.1 基礎設施指標</h4> <table> <thead> <tr> <th>指標名稱</th> <th>類型</th> <th>描述</th> <th>標籤</th> <th>閾值</th> </tr> </thead> <tbody> <tr> <td>node_cpu_usage_percent</td> <td>Gauge</td> <td>CPU 使用率</td> <td>host, instance</td> <td>warn: 80%, crit: 95%</td> </tr> <tr> <td>node_memory_usage_percent</td> <td>Gauge</td> <td>記憶體使用率</td> <td>host, instance</td> <td>warn: 85%, crit: 95%</td> </tr> <tr> <td>node_disk_usage_percent</td> <td>Gauge</td> <td>磁碟使用率</td> <td>host, mountpoint</td> <td>warn: 80%, crit: 90%</td> </tr> <tr> <td>node_network_errors_total</td> <td>Counter</td> <td>網路錯誤數</td> <td>host, interface</td> <td>crit: &gt; [N]/min</td> </tr> </tbody> </table> <h4 id="42-應用程式指標">4.2 應用程式指標</h4> <table> <thead> <tr> <th>指標名稱</th> <th>類型</th> <th>描述</th> <th>標籤</th> <th>閾值</th> </tr> </thead> <tbody> <tr> <td>http_requests_total</td> <td>Counter</td> <td>HTTP 請求總數</td> <td>method, path, status</td> <td>—</td> </tr> <tr> <td>http_request_duration_seconds</td> <td>Histogram</td> <td>請求處理時間</td> <td>method, path</td> <td>P95 &lt; [N]s</td> </tr> <tr> <td>http_requests_errors_total</td> <td>Counter</td> <td>HTTP 錯誤數</td> <td>method, path, status</td> <td>rate &gt; [N]/min</td> </tr> <tr> <td>active_connections</td> <td>Gauge</td> <td>活躍連線數</td> <td>service</td> <td>warn: [N], crit: [N]</td> </tr> <tr> <td>db_connection_pool_active</td> <td>Gauge</td> <td>DB 連線池使用數</td> <td>pool_name</td> <td>warn: 80%, crit: 95%</td> </tr> <tr> <td>queue_depth</td> <td>Gauge</td> <td>訊息佇列深度</td> <td>queue_name</td> <td>warn: [N], crit: [N]</td> </tr> </tbody> </table> <h4 id="43-業務指標">4.3 業務指標</h4> <table> <thead> <tr> <th>指標名稱</th> <th>類型</th> <th>描述</th> <th>標籤</th> <th>告警條件</th> </tr> </thead> <tbody> <tr> <td>[business_metric_1]</td> <td>Counter</td> <td>[描述]</td> <td>[labels]</td> <td>[condition]</td> </tr> <tr> <td>[business_metric_2]</td> <td>Gauge</td> <td>[描述]</td> <td>[labels]</td> <td>[condition]</td> </tr> </tbody> </table> <hr> <h3 id="5-告警規則定義">5. 告警規則定義</h3> <h4 id="51-告警嚴重度定義">5.1 告警嚴重度定義</h4> <table> <thead> <tr> <th>嚴重度</th> <th>定義</th> <th>回應時間</th> <th>通知方式</th> </tr> </thead> <tbody> <tr> <td>P1 - Critical</td> <td>服務完全中斷 / 資料遺失</td> <td>5 分鐘</td> <td>PagerDuty + 電話 + SMS</td> </tr> <tr> <td>P2 - High</td> <td>核心功能受損 / 效能嚴重惡化</td> <td>15 分鐘</td> <td>PagerDuty + Teams</td> </tr> <tr> <td>P3 - Medium</td> <td>非核心功能異常 / 效能下降</td> <td>1 小時</td> <td>Teams Channel</td> </tr> <tr> <td>P4 - Low</td> <td>預警 / 需關注但無立即影響</td> <td>下個工作日</td> <td>Email / Ticket</td> </tr> </tbody> </table> <h4 id="52-告警規則清單">5.2 告警規則清單</h4> <table> <thead> <tr> <th>Alert ID</th> <th>名稱</th> <th>嚴重度</th> <th>條件</th> <th>For</th> <th>說明</th> </tr> </thead> <tbody> <tr> <td>ALT-001</td> <td>ServiceDown</td> <td>P1</td> <td>up == 0</td> <td>1m</td> <td>服務實例離線</td> </tr> <tr> <td>ALT-002</td> <td>HighErrorRate</td> <td>P1</td> <td>error_rate &gt; 5%</td> <td>5m</td> <td>錯誤率過高</td> </tr> <tr> <td>ALT-003</td> <td>HighLatency</td> <td>P2</td> <td>p95_latency &gt; [N]ms</td> <td>5m</td> <td>回應時間過慢</td> </tr> <tr> <td>ALT-004</td> <td>HighCPU</td> <td>P3</td> <td>cpu_usage &gt; 80%</td> <td>10m</td> <td>CPU 使用率高</td> </tr> <tr> <td>ALT-005</td> <td>DiskAlmostFull</td> <td>P2</td> <td>disk_usage &gt; 85%</td> <td>5m</td> <td>磁碟空間不足</td> </tr> <tr> <td>ALT-006</td> <td>DBConnectionPoolHigh</td> <td>P2</td> <td>pool_usage &gt; 80%</td> <td>5m</td> <td>DB 連線池接近上限</td> </tr> <tr> <td>ALT-007</td> <td>CertExpiringSoon</td> <td>P3</td> <td>cert_expiry &lt; 30d</td> <td>—</td> <td>憑證即將到期</td> </tr> <tr> <td>ALT-008</td> <td>ErrorBudgetBurnRate</td> <td>P2</td> <td>burn_rate &gt; 1.0</td> <td>1h</td> <td>Error Budget 消耗過快</td> </tr> </tbody> </table> <h4 id="53-告警規則範例prometheus-alertmanager">5.3 告警規則範例（Prometheus AlertManager）</h4> <div class="highlight"> <pre tabindex="0"><code class="language-yaml" data-lang="yaml">groups: - name: [service_name].rules rules: - alert: [AlertName] expr: [PromQL expression] for: [duration] labels: severity: [critical/warning/info] team: [team_name] service: [service_name] annotations: summary: &#34;[簡短摘要]&#34; description: &#34;[詳細描述，可含 {{ $labels }} 和 {{ $value }}]&#34; runbook_url: &#34;[Runbook 連結]&#34; dashboard_url: &#34;[Dashboard 連結]&#34;</code></pre> </div> <hr> <h3 id="6-dashboard-設計">6. Dashboard 設計</h3> <h4 id="61-dashboard-清單">6.1 Dashboard 清單</h4> <table> <thead> <tr> <th>Dashboard</th> <th>用途</th> <th>目標受眾</th> <th>更新頻率</th> </tr> </thead> <tbody> <tr> <td>Service Overview</td> <td>服務健康狀態總覽</td> <td>SRE / 管理層</td> <td>10s</td> </tr> <tr> <td>API Performance</td> <td>API 效能細節</td> <td>SRE / Dev</td> <td>10s</td> </tr> <tr> <td>Infrastructure</td> <td>基礎設施資源</td> <td>Infra / SRE</td> <td>30s</td> </tr> <tr> <td>Business Metrics</td> <td>業務指標</td> <td>PM / PO</td> <td>1m</td> </tr> <tr> <td>SLO Tracking</td> <td>SLO 達成率與 Error Budget</td> <td>SRE / 管理層</td> <td>1m</td> </tr> </tbody> </table> <h4 id="62-service-overview-dashboard-設計">6.2 Service Overview Dashboard 設計</h4> <table> <thead> <tr> <th>Panel</th> <th>視覺化類型</th> <th>指標</th> <th>說明</th> </tr> </thead> <tbody> <tr> <td>Service Status</td> <td>Stat (up/down)</td> <td>up{service=&quot;&hellip;&quot;}</td> <td>紅綠燈</td> </tr> <tr> <td>Request Rate</td> <td>Time Series</td> <td>rate(http_requests_total[5m])</td> <td>QPS</td> </tr> <tr> <td>Error Rate</td> <td>Time Series</td> <td>error_rate</td> <td>錯誤率趨勢</td> </tr> <tr> <td>P95 Latency</td> <td>Time Series</td> <td>histogram_quantile(0.95, &hellip;)</td> <td>延遲趨勢</td> </tr> <tr> <td>Active Users</td> <td>Stat</td> <td>active_sessions</td> <td>當前活躍用戶</td> </tr> <tr> <td>SLO Status</td> <td>Gauge</td> <td>slo_compliance</td> <td>SLO 達標率</td> </tr> </tbody> </table> <hr> <h3 id="7-告警通知與升級">7. 告警通知與升級</h3> <h4 id="71-通知路由">7.1 通知路由</h4> <table> <thead> <tr> <th>嚴重度</th> <th>工作時間 (09-18)</th> <th>非工作時間</th> <th>假日</th> </tr> </thead> <tbody> <tr> <td>P1</td> <td>On-call + Team Lead (即時)</td> <td>On-call + Backup (即時)</td> <td>On-call + Manager (即時)</td> </tr> <tr> <td>P2</td> <td>On-call (15 min)</td> <td>On-call (30 min)</td> <td>On-call (30 min)</td> </tr> <tr> <td>P3</td> <td>Teams Channel</td> <td>下個工作日</td> <td>下個工作日</td> </tr> <tr> <td>P4</td> <td>Email / Ticket</td> <td>—</td> <td>—</td> </tr> </tbody> </table> <h4 id="72-升級機制">7.2 升級機制</h4> <table> <thead> <tr> <th>時間</th> <th>未回應動作</th> </tr> </thead> <tbody> <tr> <td>T + 5 min</td> <td>重新通知 On-call</td> </tr> <tr> <td>T + 15 min</td> <td>通知 Backup On-call</td> </tr> <tr> <td>T + 30 min</td> <td>通知 Team Lead / Manager</td> </tr> <tr> <td>T + 60 min</td> <td>通知 Director / VP</td> </tr> </tbody> </table> <h4 id="73-on-call-輪值">7.3 On-Call 輪值</h4> <table> <thead> <tr> <th>週次</th> <th>Primary</th> <th>Backup</th> <th>電話</th> </tr> </thead> <tbody> <tr> <td>Week 1</td> <td>[姓名]</td> <td>[姓名]</td> <td>[電話]</td> </tr> <tr> <td>Week 2</td> <td>[姓名]</td> <td>[姓名]</td> <td>[電話]</td> </tr> </tbody> </table> <hr> <h3 id="8-日誌監控策略">8. 日誌監控策略</h3> <h4 id="81-日誌等級與保留">8.1 日誌等級與保留</h4> <table> <thead> <tr> <th>Log Level</th> <th>用途</th> <th>保留期間</th> <th>告警</th> </tr> </thead> <tbody> <tr> <td>ERROR</td> <td>需處理的錯誤</td> <td>90 days</td> <td>rate &gt; [N]/min → P2</td> </tr> <tr> <td>WARN</td> <td>潛在問題</td> <td>30 days</td> <td>rate &gt; [N]/min → P3</td> </tr> <tr> <td>INFO</td> <td>正常營運記錄</td> <td>14 days</td> <td>—</td> </tr> <tr> <td>DEBUG</td> <td>開發除錯用</td> <td>3 days (STG only)</td> <td>—</td> </tr> </tbody> </table> <h4 id="82-關鍵日誌監控">8.2 關鍵日誌監控</h4> <table> <thead> <tr> <th>#</th> <th>監控模式</th> <th>觸發條件</th> <th>告警</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>Exception stack trace</td> <td>rate &gt; [N]/min</td> <td>P2</td> </tr> <tr> <td>2</td> <td>&ldquo;OutOfMemoryError&rdquo;</td> <td>出現 1 次</td> <td>P1</td> </tr> <tr> <td>3</td> <td>&ldquo;Connection refused&rdquo;</td> <td>rate &gt; [N]/min</td> <td>P2</td> </tr> <tr> <td>4</td> <td>Authentication failure</td> <td>rate &gt; [N]/min</td> <td>P2 (可能攻擊)</td> </tr> <tr> <td>5</td> <td>[自訂業務異常模式]</td> <td>[條件]</td> <td>[等級]</td> </tr> </tbody> </table> <hr> <h3 id="9-維護與檢討機制">9. 維護與檢討機制</h3> <table> <thead> <tr> <th>項目</th> <th>頻率</th> <th>負責人</th> <th>說明</th> </tr> </thead> <tbody> <tr> <td>Alert Noise 檢討</td> <td>每月</td> <td>SRE</td> <td>刪除/調整誤報告警</td> </tr> <tr> <td>SLO 檢討</td> <td>每季</td> <td>SRE + PO</td> <td>調整目標值</td> </tr> <tr> <td>Dashboard 更新</td> <td>需求變動時</td> <td>SRE</td> <td>新增/移除指標</td> </tr> <tr> <td>Runbook 更新</td> <td>每次事件後</td> <td>SRE</td> <td>補充處理步驟</td> </tr> <tr> <td>On-Call 回顧</td> <td>每月</td> <td>SRE Team</td> <td>改善值班體驗</td> </tr> </tbody> </table> <hr> <h3 id="10-附錄">10. 附錄</h3> <h4 id="101-alertmanager-設定檔位置">10.1 AlertManager 設定檔位置</h4> <table> <thead> <tr> <th>檔案</th> <th>位置</th> <th>說明</th> </tr> </thead> <tbody> <tr> <td>alertmanager.yml</td> <td>[path]</td> <td>通知路由設定</td> </tr> <tr> <td>prometheus-rules/</td> <td>[path]</td> <td>告警規則目錄</td> </tr> <tr> <td>grafana-dashboards/</td> <td>[path]</td> <td>Dashboard JSON</td> </tr> </tbody> </table> <h4 id="102-相關文件">10.2 相關文件</h4> <table> <thead> <tr> <th>文件</th> <th>連結</th> </tr> </thead> <tbody> <tr> <td>Runbook</td> <td>[link]</td> </tr> <tr> <td>SOP</td> <td>[link]</td> </tr> <tr> <td>Incident Response Plan</td> <td>[link]</td> </tr> </tbody> </table> <hr> <h2 id="-使用說明">📖 使用說明</h2> <h3 id="建立監控的優先順序">建立監控的優先順序</h3> <ol> <li><strong>Phase 1</strong>：健康檢查 + 基本告警（Service Up/Down, Error Rate）</li> <li><strong>Phase 2</strong>：Golden Signals 完整覆蓋（Latency, Traffic, Errors, Saturation）</li> <li><strong>Phase 3</strong>：SLI/SLO 追蹤 + Error Budget</li> <li><strong>Phase 4</strong>：業務指標 + 進階分析</li> </ol> <h3 id="告警設計原則">告警設計原則</h3> <table> <thead> <tr> <th>原則</th> <th>說明</th> </tr> </thead> <tbody> <tr> <td>Alert on symptoms</td> <td>告警使用者可感知的症狀，非原因</td> </tr> <tr> <td>Actionable</td> <td>每個告警都必須有對應的處理動作</td> </tr> <tr> <td>Low noise</td> <td>避免無意義的告警（Alert fatigue）</td> </tr> <tr> <td>Have a runbook</td> <td>每個告警必須連結到 Runbook</td> </tr> </tbody> </table> <hr> <h2 id="-範例以-hrms-人力資源管理系統為例">💡 範例（以 HRMS 人力資源管理系統為例）</h2> <hr> <h3 id="範例slislo">範例：SLI/SLO</h3> <table> <thead> <tr> <th>服務</th> <th>SLI</th> <th>SLO</th> <th>Error Budget/月</th> </tr> </thead> <tbody> <tr> <td>HRMS API</td> <td>Availability</td> <td>≥ 99.9%</td> <td>43.8 min</td> </tr> <tr> <td>HRMS API</td> <td>P95 Latency</td> <td>&lt; 200ms</td> <td>—</td> </tr> <tr> <td>薪資批次作業</td> <td>Success Rate</td> <td>≥ 99.99%</td> <td>0.44 min (每月一次不容失敗)</td> </tr> <tr> <td>出缺勤打卡</td> <td>Availability</td> <td>≥ 99.95% (上班時段)</td> <td>21.9 min</td> </tr> </tbody> </table> <h3 id="範例告警規則">範例：告警規則</h3> <div class="highlight"> <pre tabindex="0"><code class="language-yaml" data-lang="yaml">groups: - name: hrms.rules rules: - alert: HRMSHighErrorRate expr: | sum(rate(http_requests_total{service=&#34;hrms-api&#34;, status=~&#34;5..&#34;}[5m])) / sum(rate(http_requests_total{service=&#34;hrms-api&#34;}[5m])) &gt; 0.01 for: 5m labels: severity: critical team: hrms service: hrms-api annotations: summary: &#34;HRMS API 錯誤率超過 1%&#34; description: &#34;目前錯誤率為 {{ $value | humanizePercentage }}，超過 SLO 閾值&#34; runbook_url: &#34;https://wiki/runbook/hrms-high-error-rate&#34; - alert: HRMSPayrollJobFailed expr: hrms_payroll_job_status == 0 for: 1m labels: severity: critical team: hrms service: hrms-payroll annotations: summary: &#34;HRMS 薪資計算作業失敗&#34; description: &#34;月薪計算批次作業執行失敗，需立即處理&#34; runbook_url: &#34;https://wiki/runbook/hrms-payroll-failure&#34;</code></pre> </div> <hr> <blockquote> <p>📌 <strong>審閱重點</strong></p>