Skip to content

Instantly share code, notes, and snippets.

@nerdalert
Created April 28, 2026 05:07
Show Gist options
  • Select an option

  • Save nerdalert/e35609d4b38693fb27ae01e4b9644abd to your computer and use it in GitHub Desktop.

Select an option

Save nerdalert/e35609d4b38693fb27ae01e4b9644abd to your computer and use it in GitHub Desktop.

Revisions

  1. nerdalert created this gist Apr 28, 2026.
    175 changes: 175 additions & 0 deletions prometheus-pr-manual-validation.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,175 @@
    ### Manual Validation for Prometheus metrics foundation with /metrics admin endpoint PR

    ```
    Validation output are inline after the curl cmds
    Start Praxis and a dummy endpoint:
    cargo run -p praxis -- -c examples/configs/operations/admin-interface.yaml
    python3 -m http.server 3000 both running.
    Step 2: /metrics returns Prometheus text format
    $ curl -si http://localhost:9901/metrics
    HTTP/1.1 200 OK
    Content-Type: text/plain; version=0.0.4; charset=utf-8
    Date: Tue, 28 Apr 2026 05:06:20 GMT
    Connection: keep-alive
    # TYPE praxis_http_requests_total counter
    praxis_http_requests_total{method="GET",status_class="4xx",route="unknown",cluster="backend"} 2
    praxis_http_requests_total{method="HEAD",status_class="2xx",route="unknown",cluster="backend"} 2
    praxis_http_requests_total{method="GET",status_class="2xx",route="unknown",cluster="backend"} 4
    praxis_http_requests_total{method="POST",status_class="5xx",route="unknown",cluster="backend"} 2
    # TYPE praxis_http_request_duration_seconds summary
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="GET",status_class="2xx",route="unknown",cluster="backend"} 0.008274873
    praxis_http_request_duration_seconds_count{method="GET",status_class="2xx",route="unknown",cluster="backend"} 4
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="GET",status_class="4xx",route="unknown",cluster="backend"} 0.010357936
    praxis_http_request_duration_seconds_count{method="GET",status_class="4xx",route="unknown",cluster="backend"} 2
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="POST",status_class="5xx",route="unknown",cluster="backend"} 0.002428343
    praxis_http_request_duration_seconds_count{method="POST",status_class="5xx",route="unknown",cluster="backend"} 2
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="HEAD",status_class="2xx",route="unknown",cluster="backend"} 0.003298737
    praxis_http_request_duration_seconds_count{method="HEAD",status_class="2xx",route="unknown",cluster="backend"} 2
    Step 3: Existing admin endpoints still work
    $ curl -si http://localhost:9901/healthy
    HTTP/1.1 200 OK
    Content-Type: application/json
    Date: Tue, 28 Apr 2026 04:57:15 GMT
    Connection: keep-alive
    $ curl -si http://localhost:9901/ready
    HTTP/1.1 200 OK
    Content-Type: application/json
    Date: Tue, 28 Apr 2026 04:57:15 GMT
    Connection: keep-alive
    {"status":"ok"}
    Step 4: Request counter increments with traffic
    $ curl -s http://localhost:8080/ # 2xx
    $ curl -s http://localhost:8080/nonexistent # 4xx
    $ curl -s http://localhost:9901/metrics | grep praxis_http_requests_total
    # TYPE praxis_http_requests_total counter
    praxis_http_requests_total{method="GET",status_class="4xx",route="unknown",cluster="backend"} 2
    praxis_http_requests_total{method="HEAD",status_class="2xx",route="unknown",cluster="backend"} 2
    praxis_http_requests_total{method="GET",status_class="2xx",route="unknown",cluster="backend"} 4
    praxis_http_requests_total{method="POST",status_class="5xx",route="unknown",cluster="backend"} 2
    Step 5: Duration histogram appears with quantiles
    $ curl -s http://localhost:9901/metrics | grep praxis_http_request_duration_seconds
    # TYPE praxis_http_request_duration_seconds summary
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="2xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="GET",status_class="2xx",route="unknown",cluster="backend"} 0.008274873
    praxis_http_request_duration_seconds_count{method="GET",status_class="2xx",route="unknown",cluster="backend"} 4
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="GET",status_class="4xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="GET",status_class="4xx",route="unknown",cluster="backend"} 0.010357936
    praxis_http_request_duration_seconds_count{method="GET",status_class="4xx",route="unknown",cluster="backend"} 2
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="POST",status_class="5xx",route="unknown",cluster="backend"} 0.002428343
    praxis_http_request_duration_seconds_count{method="POST",status_class="5xx",route="unknown",cluster="backend"} 2
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="HEAD",status_class="2xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="HEAD",status_class="2xx",route="unknown",cluster="backend"} 0.00161027
    praxis_http_request_duration_seconds_count{method="HEAD",status_class="2xx",route="unknown",cluster="backend"} 1
    Step 6: Separate label sets per HTTP method
    $ curl -s -I http://localhost:8080/
    HTTP/1.1 200 OK
    Server: SimpleHTTP/0.6 Python/3.12.3
    Date: Tue, 28 Apr 2026 05:03:28 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 986
    via: 1.1 praxis
    Connection: keep-alive
    $ curl -s http://localhost:9901/metrics | grep praxis_http_requests_total
    # TYPE praxis_http_requests_total counter
    praxis_http_requests_total{method="HEAD",status_class="2xx",route="unknown",cluster="backend"} 1
    praxis_http_requests_total{method="GET",status_class="2xx",route="unknown",cluster="backend"} 4
    praxis_http_requests_total{method="POST",status_class="5xx",route="unknown",cluster="backend"} 2
    praxis_http_requests_total{method="GET",status_class="4xx",route="unknown",cluster="backend"} 2
    Step 7: Unknown admin route returns 404
    $ curl -si http://localhost:9901/nonexistent
    HTTP/1.1 404 Not Found
    Content-Type: application/json
    Date: Tue, 28 Apr 2026 04:57:31 GMT
    Connection: keep-alive
    {"error":"not found"}
    Step 8: No body-byte metrics present
    $ curl -s http://localhost:9901/metrics | grep body_bytes
    (no output — body_bytes metrics correctly absent)
    Step 9: Cluster label preserved on 5xx
    $ curl -s http://localhost:9901/metrics | grep 'status_class="5xx"'
    praxis_http_requests_total{method="POST",status_class="5xx",route="unknown",cluster="backend"} 2
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.5"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.9"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.95"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.99"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="0.999"} 0
    praxis_http_request_duration_seconds{method="POST",status_class="5xx",route="unknown",cluster="backend",quantile="1"} 0
    praxis_http_request_duration_seconds_sum{method="POST",status_class="5xx",route="unknown",cluster="backend"} 0.002428343
    praxis_http_request_duration_seconds_count{method="POST",status_class="5xx",route="unknown",cluster="backend"} 2
    ```