Logs
The services write their log files to var/log inside the installation directory, i.e. /opt/msh-zr/var/log. The directory is created automatically on the first service start, and all files are owned by the service user (msh by default). A typical listing on a single server looks like this:
drwxr-x--- 3 msh msh 4096 Jul 15 05:35 message-events
-rw-r----- 1 msh msh 94 Jul 15 04:36 milter.out
-rw-r----- 1 msh msh 160 Jul 15 05:35 milter.stats
-rw-r----- 1 msh msh 4492 Jul 15 05:35 msh-zr-milter.log
-rw-r----- 1 msh msh 2310 Jul 15 05:35 msh-zr-node.log
-rw-r----- 1 msh msh 5120 Jul 15 05:35 msh-zr-web.log
-rw-r----- 1 msh msh 188 Jul 15 04:36 node.out
-rw-r----- 1 msh msh 204 Jul 15 04:58 web.out
Launcher console output: milter.out, node.out and web.out
milter.out, node.out and web.out hold the lines the launcher of each service prints before the application starts, such as the Java checks. Each file is emptied when its service starts. The same lines also reach the systemd journal, so journalctl -u msh-zr-milter.service shows them too.
Look here for problems that occur before the application logging is up: launcher errors (missing or too old Java, invalid MSH_JAVA_BIN) and JVM startup failures.
The web app can also write its command line and all of its resolved configuration properties, environment and system properties included, at every start. This startup log is off by default: it is written at DEBUG level while the web app logs at INFO. When support asks you to switch it on, the values of properties whose name contains password, secret, key or token show as ******, so the log can be shared without giving away the keystore password.
Application log: msh-zr-milter.log
The main application log of the milter, written at info level with the format:
HH:mm:ss.SSS [thread] LEVEL logger - message
It records startup, connections from the Zimbra MTA, settings reloads from the shared database, and message processing errors.
Every processed message gets one line with its MSH-ID, the Postfix queue ID, the sender, the To recipients, the subject and the answer given to Postfix, followed by how long it took:
14:03:22.517 [pool-2-thread-3] INFO m.app.milter.MilterHandlerBase - mid(3f2a9c) qid(4XyZ1n2abc) f(alice@example.com) t(bob@example.org) s(Quarterly report) st(ACCEPT) ms(total 182, receive 120, rules 55, flush 7)
receive- from MAIL FROM to the end of the message, mostly the time the client takes to send it.rules- from the end of the message until the answer is known: the rules and the changes they make to the message.flush- writing the message history.
A slow receive points at the network or the client, a slow rules at the rules or the CPU, a slow flush at the disk.
An error of the milter protocol is logged with the connection it happened on, for example Milter protocol error on the connection from 203.0.113.5. A client that disconnects in the middle logs Connection from 203.0.113.5 ended early at info level.
When the milter stops, it lets the running sessions finish for up to 20 seconds and logs how many finished, and how many were cut off when the time ran out:
Milter stopping: waiting up to 20 s for the running sessions
Milter stopped: 3 session(s) finished
Postfix applies milter_default_action to a session that was cut off. The file rolls over daily; rotated files are named msh-zr-milter.YYYY-MM-DD.log, the last 7 days are kept, and the total size of the rotated history is capped at 10 MB.
Application logs: msh-zr-web.log and msh-zr-node.log
The web admin writes its log to msh-zr-web.log and the node to msh-zr-node.log, and both also to the systemd journal (journalctl -u msh-zr-web.service, journalctl -u msh-zr-node.service). The files roll over daily and when they reach 10 MB; rotated files are gzipped and kept for 14 days, at most 200 MB in total.
To change the log levels or these limits, put the settings in /opt/msh-zr/etc/msh-zr-web.properties or /opt/msh-zr/etc/msh-zr-node.properties and restart the service. Neither file exists until you create it. For example:
# More detail from MSH Zimbra Rules itself
logging.level.msh.zr=DEBUG
# Keep more history
logging.logback.rollingpolicy.max-history=30
logging.logback.rollingpolicy.total-size-cap=1GB
An error the admin panel did not expect is logged here with the id the panel showed in its notification, the user and the page, see Unexpected errors.
Worker pool statistics: milter.stats
A small snapshot of the milter's state and worker thread pool, rewritten on a timer every stats update interval (STATS_UPDATE_TIME, default 5 seconds), also when no mail arrives. It contains the current values as key=value lines:
state=UP
updated_ms=1794249600000
stats_update_sec=5
dropped_5m=0
dropped_1h=0
messages_1m=42
connections_accepted=1042
connections_dropped=0
workers_busy=2
workers=5
workers_core=5
workers_largest=5
workers_max=10
tasks_total=1042
tasks_completed=1040
queue_length=0
queue_capacity=200
state-UP,DEGRADEDorDOWN, see Milter states.updated_ms/stats_update_sec- when the file was written (milliseconds since 1970) and how often it is. A file that stopped changing means the milter is gone.dropped_5m/dropped_1h- connections dropped in the last 5 minutes and the last hour.messages_1m- messages received in the last minute.connections_accepted/connections_dropped- connections since the service started. Every dropped connection is a message that was refused or passed unfiltered, depending onmilter_default_action.workers_busy- threads currently processing a message.workers/workers_core/workers_max/workers_largest- current, core (CORE_POOL_SIZE), maximum (POOL_SIZE) and historical peak thread counts.tasks_total/tasks_completed- connections submitted and finished since the service started.queue_length/queue_capacity- connections waiting for a free thread and the configured queue capacity (QUEUE_SIZE).
A queue_length that keeps growing toward queue_capacity means the milter cannot keep up with the mail volume. Check the CPU of the host before you change anything, see When the milter is full.
Each dropped connection also logs this warning in msh-zr-milter.log:
Message skipped from milter processing, no space in queue.
Message history database: msh-zr-history.db
The milter writes the processing traces of the messages, which the Message history page shows, and the refusals of the SMTP rules, which the SMTP events page shows, to /opt/msh-zr/etc/msh-zr-history.db on every MTA server. The node of the server reads them from there for the web admin. HISTORY_DB_FILENAME in /opt/msh-zr/etc/config.cf sets another path.
The file is apart from msh-zr-milter.db, which holds the limit counters. Writing and removing history therefore never makes the decisions about accepting mail wait. The file holds nothing the milter needs to process mail: when it is damaged or the disk is full, only the history is lost. To start the history from scratch, stop the milter, delete the file and start the milter again; it creates an empty one.
The history never holds up the mail:
- The history of a message is queued and written by a thread of its own, many messages at a time. When it cannot be written fast enough, messages are left out of Message History, and
msh-zr-milter.logsays how many, at most once a minute:Messages are processed faster than their history can be written; 120 were left out of Message History so far - History older than the retention period of Diagnostic logging is removed once a minute, in short batches, and the file gives the space back to the disk. A large backlog, for example after the retention was shortened, is removed over several minutes.
Message body dumps: message-events/
Event traces of processed messages are stored in the message history database. When persisting message bodies is enabled in the diagnostic logging settings, the milter additionally stores the complete MIME message on disk, twice per message:
var/log/message-events/<date>/<id>-before.eml
var/log/message-events/<date>/<id>-after.eml
-before.eml- the message exactly as it arrived, before any rules were applied.-after.eml- the message after processing, as it was handed back to the MTA.
Comparing the two files shows precisely what the rules changed. The web UI reads these files from the same var/log directory and displays them in the Message History message details. Old events are pruned automatically together with their .eml files once they are older than the retention period configured in the diagnostic logging settings.
The node reads the .eml files from the milter's var/log and passes them to the web admin. Both run on the same MTA server, so they always see the same directory.
Additional resources
- Services - using these files to diagnose a service that does not start
- Message history - browsing processed messages in the web UI
- Diagnostic logging - enabling body persistence and setting the retention period