Skip to main content
Version: 1.4

Milter load and monitoring

When the milter is full​

The milter takes POOL_SIZE + QUEUE_SIZE connections at once. It closes every connection beyond that, and Postfix then applies milter_default_action: with the recommended tempfail the message is refused and retried later, with accept it is delivered unfiltered, with no signatures, disclaimers or policy rules applied (see Milter configuration).

While a server dropped connections in the last hour, the Servers page shows a notice such as 34 messages were not filtered on mta1 in the last hour: the milter was full.

Check the CPU of that host first. With typical rules the CPU is usually the limit, not the number of workers: on a server with few CPUs more workers compete for the same CPUs and make every message slower. If the CPU has room and the drops come from short bursts or pauses, lengthen the queue.

These settings are in /opt/msh-zr/etc/config.cf on the MTA host. Restart msh-zr-milter.service after changing them.

SettingDefaultMeaning
QUEUE_SIZE200Sessions that can wait for a worker. A longer queue absorbs short pauses without adding threads. Keep it short enough to empty well within the 30 seconds Postfix waits for the milter.
POOL_SIZE10The largest number of workers. Threads over CORE_POOL_SIZE are added only when the queue is full.
CORE_POOL_SIZE0The workers the milter runs with until the queue is full, at most POOL_SIZE. 0 follows the number of CPUs: 5 on 2 CPUs, about 2.5 per CPU.

Monitoring​

A monitoring system such as Nagios or Zabbix can watch the installation with one HTTP check. The web admin answers without a login, with the state only:

curl http://<web-admin-host>:9010/api/health
{"status":"DEGRADED","nodes":[{"name":"mta1","reachable":true,"status":"DEGRADED","settingsInStep":true}]}
statusMeaningHTTP
UPEvery server is up.200
DEGRADEDA server is degraded, down or cannot be used, or no server is configured.200
DOWNNo server is left.503

With several servers, one server down gives DEGRADED with HTTP 200, so a check that should alert on it has to read status from the answer, not only the HTTP code.

Each node has its own answers on port 9012, over HTTPS and with the token from /opt/msh-zr/etc/node.token. /api/health adds the checks behind the state, and answers 503 when the milter is down. /api/metrics gives the milter's numbers in Prometheus text format.

curl -k -H "Authorization: Bearer <token>" https://<mta-host>:9012/api/health
curl -k -H "Authorization: Bearer <token>" https://<mta-host>:9012/api/metrics

What /api/metrics returns​

MetricTypeMeaning
mshzr_milter_connections_total{result="accepted"}counterConnections from Postfix handed to the worker pool since the milter started.
mshzr_milter_connections_total{result="dropped"}counterConnections closed because the pool and the queue were full. Each one is a message refused or passed unfiltered, depending on milter_default_action.
mshzr_milter_workers_busygaugeWorkers processing a session right now.
mshzr_milter_workersgaugeThreads currently in the pool.
mshzr_milter_queue_lengthgaugeSessions waiting for a worker.
mshzr_milter_queue_capacitygaugeThe size of the queue (QUEUE_SIZE).

The numbers come from milter.stats, so they change every STATS_UPDATE_TIME seconds, 5 by default. The counters start again from zero when the milter restarts. The answer is empty until the milter has written the file.

If the milter dies, this endpoint keeps returning its last numbers. Use /api/health to detect that: it answers 503 when the milter is down.

Health and metrics on the management ports​

The web admin and the node also serve health, info and Prometheus metrics on a management port of their own. The ports listen on 127.0.0.1 only and need no login, which is why they are not reachable from the network.

ServiceSetting in etc/config.cfDefault
Web adminWEB_MANAGEMENT_PORT9013
NodeNODE_MANAGEMENT_PORT9014

An empty value switches the endpoints off. Before the first start, check with ss -ltn that nothing else on the host uses these ports, or move them in etc/config.cf and restart the service.

EndpointAnswers
/actuator/healthThe state of the service, with the databases and the free disk it checks
/actuator/health/liveness, /actuator/health/readinessLiveness and readiness, for systemd watchdogs and orchestrators
/actuator/infoThe version of the service
/actuator/prometheusMetrics in Prometheus text format

Check them by hand on the host:

curl http://127.0.0.1:9013/actuator/health
curl -s http://127.0.0.1:9014/actuator/prometheus | grep ^mshzr

To let a Prometheus server on another host scrape them, bind the port to an address it can reach and limit that address with the firewall, in etc/msh-zr-web.properties or etc/msh-zr-node.properties:

management.server.address=10.0.0.5

Metrics​

Besides the metrics of the JVM, HTTP, Tomcat and the database pool, /actuator/prometheus has these, all with the label application:

MetricFromMeaning
mshzr_build_info{version}both1, with the version as label
mshzr_milter_upnode1 while the milter runs, 0 when it stopped or is down
mshzr_milter_connections_total{result}nodeConnections from Postfix, accepted or dropped because the pool and the queue were full
mshzr_milter_dropped_5m, mshzr_milter_dropped_1hnodeConnections dropped in the last 5 minutes and the last hour
mshzr_milter_messages_per_minutenodeMessages the milter processed in the last minute
mshzr_milter_workers_busy, mshzr_milter_workers, mshzr_milter_workers_max, mshzr_milter_queue_length, mshzr_milter_queue_capacitynodeThe milter's pool and queue; empty while the milter is down
mshzr_node_check_up{check}node1 when the check milter, license, settings or disk of the node is up
mshzr_node_settings_version, mshzr_node_disk_free_bytesnodeThe version of the settings the node holds; the free disk
mshzr_serverswebServers on the Servers page
mshzr_server_reachable{server}, mshzr_server_healthy{server}, mshzr_server_settings_in_step{server}webPer server: the node answers, it reports up, it has the current settings
mshzr_server_last_contact_age_seconds{server}webSeconds since the node last answered
mshzr_server_milter_messages_per_minute{server}, mshzr_server_milter_dropped_5m{server}, mshzr_server_milter_dropped_1h{server}, mshzr_server_disk_free_bytes{server}webWhat the node reported last, while it answers
mshzr_license_valid, mshzr_license_expiry_dayswebThe license key is valid; the days until it expires
mshzr_directory_mailboxes, mshzr_directory_last_sync_age_secondswebMailboxes synced from the directory; seconds since the last sync

The mshzr_server_* metrics of the web admin cover every server, so one scrape of the web admin shows the whole installation. Scrape the nodes too for the full numbers of each milter. With Prometheus running on the host itself:

scrape_configs:
- job_name: msh-zr-web
metrics_path: /actuator/prometheus
static_configs:
- targets: ["127.0.0.1:9013"]
- job_name: msh-zr-node
metrics_path: /actuator/prometheus
static_configs:
- targets: ["127.0.0.1:9014"]

Alert rules to start with:

groups:
- name: msh-zr
rules:
- alert: MshZrServerDown
expr: mshzr_server_reachable == 0
for: 5m
- alert: MshZrServerUnhealthy
expr: mshzr_server_healthy == 0
for: 10m
- alert: MshZrMilterDropsConnections
expr: mshzr_server_milter_dropped_5m > 0
- alert: MshZrSettingsBehind
expr: mshzr_server_settings_in_step == 0
for: 15m
- alert: MshZrLicenseExpiresSoon
expr: mshzr_license_expiry_days < 14
- alert: MshZrDirectorySyncStale
expr: mshzr_directory_last_sync_age_seconds > 2 * 86400

The milter itself serves no endpoint; its numbers come through the node.

Additional resources​

  • Servers - the milter states and numbers each server shows
  • Logs - the same numbers in milter.stats