Milter load and monitoring
When the milter is full
The milter takes POOL_SIZE + QUEUE_SIZE connections at once. It closes every connection beyond that, and Postfix then applies milter_default_action: with the recommended tempfail the message is refused and retried later, with accept it is delivered unfiltered, with no signatures, disclaimers or policy rules applied (see Milter configuration).
While a server dropped connections in the last hour, the Servers page shows a notice such as 34 messages were not filtered on mta1 in the last hour: the milter was full.
Check the CPU of that host first. With typical rules the CPU is usually the limit, not the number of workers: on a server with few CPUs more workers compete for the same CPUs and make every message slower. If the CPU has room and the drops come from short bursts or pauses, lengthen the queue.
These settings are in /opt/msh-zr/etc/config.cf on the MTA host. Restart msh-zr-milter.service after changing them.
| Setting | Default | Meaning |
|---|---|---|
QUEUE_SIZE | 200 | Sessions that can wait for a worker. A longer queue absorbs short pauses without adding threads. Keep it short enough to empty well within the 30 seconds Postfix waits for the milter. |
POOL_SIZE | 10 | The largest number of workers. Threads over CORE_POOL_SIZE are added only when the queue is full. |
CORE_POOL_SIZE | 0 | The workers the milter runs with until the queue is full, at most POOL_SIZE. 0 follows the number of CPUs: 5 on 2 CPUs, about 2.5 per CPU. |
Monitoring
A monitoring system such as Nagios or Zabbix can watch the installation with one HTTP check. The web admin answers without a login, with the state only:
curl http://<web-admin-host>:9010/api/health
{"status":"DEGRADED","nodes":[{"name":"mta1","reachable":true,"status":"DEGRADED","settingsInStep":true}]}
status | Meaning | HTTP |
|---|---|---|
UP | Every server is up. | 200 |
DEGRADED | A server is degraded, down or cannot be used, or no server is configured. | 200 |
DOWN | No server is left. | 503 |
With several servers, one server down gives DEGRADED with HTTP 200, so a check that should alert on it has to read status from the answer, not only the HTTP code.
Each node has its own answers on port 9012, over HTTPS and with the token from /opt/msh-zr/etc/node.token. /api/health adds the checks behind the state, and answers 503 when the milter is down. /api/metrics gives the milter's numbers in Prometheus text format.
curl -k -H "Authorization: Bearer <token>" https://<mta-host>:9012/api/health
curl -k -H "Authorization: Bearer <token>" https://<mta-host>:9012/api/metrics
What /api/metrics returns
| Metric | Type | Meaning |
|---|---|---|
mshzr_milter_connections_total{result="accepted"} | counter | Connections from Postfix handed to the worker pool since the milter started. |
mshzr_milter_connections_total{result="dropped"} | counter | Connections closed because the pool and the queue were full. Each one is a message refused or passed unfiltered, depending on milter_default_action. |
mshzr_milter_workers_busy | gauge | Workers processing a session right now. |
mshzr_milter_workers | gauge | Threads currently in the pool. |
mshzr_milter_queue_length | gauge | Sessions waiting for a worker. |
mshzr_milter_queue_capacity | gauge | The size of the queue (QUEUE_SIZE). |
The numbers come from milter.stats, so they change every STATS_UPDATE_TIME seconds, 5 by default. The counters start again from zero when the milter restarts. The answer is empty until the milter has written the file.
If the milter dies, this endpoint keeps returning its last numbers. Use /api/health to detect that: it answers 503 when the milter is down.
Health and metrics on the management ports
The web admin and the node also serve health, info and Prometheus metrics on a management port of their own. The ports listen on 127.0.0.1 only and need no login, which is why they are not reachable from the network.
| Service | Setting in etc/config.cf | Default |
|---|---|---|
| Web admin | WEB_MANAGEMENT_PORT | 9013 |
| Node | NODE_MANAGEMENT_PORT | 9014 |
An empty value switches the endpoints off. Before the first start, check with ss -ltn that nothing else on the host uses these ports, or move them in etc/config.cf and restart the service.
| Endpoint | Answers |
|---|---|
/actuator/health | The state of the service, with the databases and the free disk it checks |
/actuator/health/liveness, /actuator/health/readiness | Liveness and readiness, for systemd watchdogs and orchestrators |
/actuator/info | The version of the service |
/actuator/prometheus | Metrics in Prometheus text format |
Check them by hand on the host:
curl http://127.0.0.1:9013/actuator/health
curl -s http://127.0.0.1:9014/actuator/prometheus | grep ^mshzr
To let a Prometheus server on another host scrape them, bind the port to an address it can reach and limit that address with the firewall, in etc/msh-zr-web.properties or etc/msh-zr-node.properties:
management.server.address=10.0.0.5
Metrics
Besides the metrics of the JVM, HTTP, Tomcat and the database pool, /actuator/prometheus has these, all with the label application:
| Metric | From | Meaning |
|---|---|---|
mshzr_build_info{version} | both | 1, with the version as label |
mshzr_milter_up | node | 1 while the milter runs, 0 when it stopped or is down |
mshzr_milter_connections_total{result} | node | Connections from Postfix, accepted or dropped because the pool and the queue were full |
mshzr_milter_dropped_5m, mshzr_milter_dropped_1h | node | Connections dropped in the last 5 minutes and the last hour |
mshzr_milter_messages_per_minute | node | Messages the milter processed in the last minute |
mshzr_milter_workers_busy, mshzr_milter_workers, mshzr_milter_workers_max, mshzr_milter_queue_length, mshzr_milter_queue_capacity | node | The milter's pool and queue; empty while the milter is down |
mshzr_node_check_up{check} | node | 1 when the check milter, license, settings or disk of the node is up |
mshzr_node_settings_version, mshzr_node_disk_free_bytes | node | The version of the settings the node holds; the free disk |
mshzr_servers | web | Servers on the Servers page |
mshzr_server_reachable{server}, mshzr_server_healthy{server}, mshzr_server_settings_in_step{server} | web | Per server: the node answers, it reports up, it has the current settings |
mshzr_server_last_contact_age_seconds{server} | web | Seconds since the node last answered |
mshzr_server_milter_messages_per_minute{server}, mshzr_server_milter_dropped_5m{server}, mshzr_server_milter_dropped_1h{server}, mshzr_server_disk_free_bytes{server} | web | What the node reported last, while it answers |
mshzr_license_valid, mshzr_license_expiry_days | web | The license key is valid; the days until it expires |
mshzr_directory_mailboxes, mshzr_directory_last_sync_age_seconds | web | Mailboxes synced from the directory; seconds since the last sync |
The mshzr_server_* metrics of the web admin cover every server, so one scrape of the web admin shows the whole installation. Scrape the nodes too for the full numbers of each milter. With Prometheus running on the host itself:
scrape_configs:
- job_name: msh-zr-web
metrics_path: /actuator/prometheus
static_configs:
- targets: ["127.0.0.1:9013"]
- job_name: msh-zr-node
metrics_path: /actuator/prometheus
static_configs:
- targets: ["127.0.0.1:9014"]
Alert rules to start with:
groups:
- name: msh-zr
rules:
- alert: MshZrServerDown
expr: mshzr_server_reachable == 0
for: 5m
- alert: MshZrServerUnhealthy
expr: mshzr_server_healthy == 0
for: 10m
- alert: MshZrMilterDropsConnections
expr: mshzr_server_milter_dropped_5m > 0
- alert: MshZrSettingsBehind
expr: mshzr_server_settings_in_step == 0
for: 15m
- alert: MshZrLicenseExpiresSoon
expr: mshzr_license_expiry_days < 14
- alert: MshZrDirectorySyncStale
expr: mshzr_directory_last_sync_age_seconds > 2 * 86400
The milter itself serves no endpoint; its numbers come through the node.