Fortified

Building production-grade security around a 19-container Mailcow Docker stack on a peered dedicated server. From zero to a 94/100 automated security audit score.

Why Self-Host in 2026

Every email you send through Gmail, Outlook, or any hosted provider crosses infrastructure you don't control, subject to terms you didn't negotiate, scanned by systems you can't audit. For a domain that's been operating since 1998, that dependency was no longer acceptable.

The goal: run a complete, self-hosted email stack on dedicated iron — SMTP, IMAP, spam filtering, antivirus, webmail, DKIM signing, automated TLS — and harden it to a standard that would survive a real attack. Not a lab exercise. Production mail for a production domain.

The Stack

The foundation is Mailcow — an open-source Docker Compose stack purpose-built for mail hosting. A full deployment runs 18 containers: Postfix (SMTP), Dovecot (IMAP/Sieve), Rspamd (spam/DKIM), ClamAV (antivirus), SOGo (webmail), nginx (reverse proxy), MariaDB, Redis, Memcached, Unbound (DNS resolver), a dedicated firewall container (netfilter-mailcow), Let's Encrypt automation (ACME), a health watchdog, a cron scheduler (Ofelia), document analysis (Olefy), TLS policy enforcement, and an internal Docker API proxy. Add Dockge for container lifecycle management and the server runs 19 containers total.

The host is a Debian 13 (Trixie) server sitting on a high-bandwidth peered backbone out of a Boston carrier hotel — multi-homed transit through Cogent (AS174) and Lumen (AS3356), with IX presence at BOSIX and DE-CIX Remote. DNS managed through Cloudflare with the domain's MX record pointing directly to the server's public IP.

Out of the box, Mailcow works. Out of the box, it's also exposed. The containers handle mail. They don't handle the host.

Layer 1 — Kernel Hardening

The Linux kernel is the first line of defense. A custom sysctl configuration (/etc/sysctl.d/99-noc-hardening.conf) enables:

• SYN cookies (tcp_syncookies=1) — when the TCP SYN backlog fills, the kernel generates cookies instead of allocating memory for half-open connections. This neutralizes SYN flood attacks, the most common volumetric vector against mail servers.
• Reverse-path filtering (rp_filter=1) — every incoming packet is checked against the routing table. If the source IP wouldn't route back through the same interface, the packet is dropped. This eliminates spoofed-source attacks at the kernel level before they reach iptables.
• ICMP rate limiting — ping flood mitigation. A 12 Gbps ICMP flood becomes 1 packet per second at the kernel.
• Ptrace restriction (ptrace_scope=2) — prevents processes from attaching to other processes for debugging. Blocks an entire class of privilege escalation exploits.
• Kernel pointer hiding (kptr_restrict=2) — kernel memory addresses are hidden from userspace. Even if an attacker gets a shell, they can't read kernel address layouts for ROP chains or return-to-libc attacks.
• dmesg restriction (dmesg_restrict=1) — kernel ring buffer requires CAP_SYSLOG. Stops unprivileged users from reading hardware and driver information useful for exploit development.

These parameters cost zero performance. They're passive armor.

Layer 2 — The Firewall Problem

On a standard Linux server, firewall management is straightforward: write iptables rules, persist them, done. On a server running Docker and Mailcow, it's a minefield.

Docker manages its own iptables chains — DOCKER, DOCKER-USER, DOCKER-ISOLATION — to route traffic between containers and the outside world. Mailcow adds its own: the MAILCOW chain, inserted at position 1 in the INPUT chain by the netfilter-mailcow container. This chain handles Mailcow's internal fail2ban — banning IPs that brute-force the webmail login, SMTP auth, or IMAP.

Run iptables -F (flush) on this server and you destroy Docker networking, Mailcow's firewall, and container connectivity in one command. Run iptables -X (delete chains) and Docker can't recreate its routing until the entire daemon restarts. Run iptables-restore and you replace the live table — overwriting everything Docker and Mailcow have built.

The solution: a custom firewall management script (noc-firewall) built from scratch specifically for this environment. It manages the INPUT chain exclusively — appending and removing rules without touching FORWARD, DOCKER, NAT, or RAW chains. It detects and preserves the MAILCOW chain jump at position 1 during all operations. It persists rules through netfilter-persistent and includes a safe Docker restart command that reapplies firewall rules after Docker rebuilds its chains.

The script includes a 22-point automated audit that verifies on every run:

• Chain integrity (INPUT, FORWARD, MAILCOW present)
• Default policies correct (INPUT: DROP, FORWARD: ACCEPT)
• Public ports open (25, 80, 143, 443, 465, 587, 993, 4190)
• Restricted ports locked to trusted IPs (22, 5001, 61209)
• No rogue listeners
• Live rules match persisted rules
• Docker chains untouched
• IPv6 rules consistent with IPv4

Policy: public mail ports are open to the world (mail servers must accept connections from any sender). SSH, management UIs, and monitoring endpoints are restricted to a whitelist of trusted IP ranges. Everything else is DROP.

Layer 3 — Fail2ban Coordination

Three jails run on the host: SSH (24-hour ban, 3 retries), Postfix (1 hour, 5 retries), Dovecot (1 hour, 5 retries). The backend is systemd's journal — Debian 13 eliminated /var/log/auth.log, so fail2ban reads directly from journald.

The coordination problem: Mailcow runs its own internal fail2ban via the netfilter-mailcow container, which inserts bans into the MAILCOW iptables chain. That chain sits at position 1 in INPUT — it evaluates before the host firewall rules. If your home IP triggers a Mailcow ban (three failed webmail logins), you lose SSH access too, because the MAILCOW chain drops all traffic from the banned IP before your SSH whitelist rule is ever evaluated.

The mitigation: trusted IPs must be whitelisted in three independent systems — the host firewall script, Mailcow's netfilter whitelist (via API), and the host fail2ban ignoreip directive. If any one of the three is missing the current IP, a lockout is possible. When a dynamic home IP changes, all three must be updated. This is not elegant. It's correct.

Layer 4 — Protocol and Service Lockdown

POP3 is disabled at the Dovecot level (protocols = imap sieve lmtp). One less protocol accepting connections, one less attack surface to defend.

Docker log rotation is configured globally (10 MB per file, 3 rotations) in /etc/docker/daemon.json. Without this, container logs grow unbounded — a 19-container stack generating authentication logs, spam filter output, and antivirus scan results will consume disk silently over weeks.

The deprecated system MTA (exim4) is disabled — it conflicts with Mailcow's Postfix on port 25. nftables is disabled and its config neutralized — the server uses iptables-nft backend exclusively, avoiding the dual-framework confusion that plagues Debian 13 deployments.

Layer 5 — DNS as Armor

DNS configuration isn't cosmetic for a mail server — it's functional security.

• SPF with hard fail (-all): only the server's IP is authorized to send mail for the domain. Any other source is explicitly rejected. No soft fail hedging.
• DKIM signing on all outbound: every message gets a cryptographic signature verified against a public key in DNS. Forgery is detectable.
• DMARC at p=quarantine with dual reporting: policy tells receiving servers to quarantine (not just flag) messages that fail SPF or DKIM alignment. Reports flow to both Cloudflare's aggregate processor and a local mailbox for independent analysis.
• PTR record verified: reverse DNS for the server IP resolves back to the hostname. Forward-confirmed reverse DNS (FCrDNS) is a baseline deliverability requirement — without it, most receiving servers reject or downrank your mail.
• Mail DNS records are not proxied through Cloudflare. The mail A record, MX record, and related entries use DNS-only mode (grey cloud). Cloudflare's HTTP proxy doesn't pass SMTP traffic, and proxying would break HELO/PTR/certificate hostname matching. This is a requirement, not a preference.

Layer 6 — Automated Auditing

Trust degrades without verification. A server that's hardened on deployment day and never checked again is a server drifting toward compromise.

A custom audit script runs nightly at midnight via systemd timer. It executes 18 sections:

Results are scored: PASS, WARN, FAIL, or ROGUE. The composite score (out of 100) provides a single number for at-a-glance health assessment. The report is compiled into markdown and emailed through the local Postfix — the audit uses the infrastructure it's auditing to deliver its own results. If the email doesn't arrive, that's a finding too.

Reports are also written to a synced directory for off-box archival.

Layer 7 — Off-Box Monitoring

The monitoring infrastructure does not run on the server it monitors. This is a deliberate architectural decision.

Metrics pipeline: Glances runs natively on the VPS (not in Docker — Docker socket access requires removing systemd's ProtectSystem=strict, a lesson learned the hard way) with the Docker plugin enabled. A remote Prometheus instance scrapes the Glances API every 15 seconds. Grafana dashboards on a separate host visualize system metrics, per-container resource usage, and network throughput.

Log pipeline: Server logs replicate every 60 seconds via Syncthing to a monitoring host. There, Promtail ships them into a Loki instance for centralized log aggregation and search. Grafana connects to Loki for log-based alerting and forensic queries.

Shell command auditing: Every command executed on the server — by any user, interactive or scripted — is captured by auditd via execve syscall rules. These logs sync off-box every 10 minutes. If the server is compromised, the forensic trail survives on separate infrastructure the attacker doesn't control.

The principle: the server produces telemetry. It does not store, analyze, or alert on its own telemetry. An attacker who roots the VPS can delete local logs — they cannot reach the monitoring host.

The Result

19/19 containers running and healthy. All systemd services and timers active. Zero fail2ban bans on a clean day. Mail delivery scoring clean on every major reputation checker (Google Postmaster Tools, mail-tester.com, MXToolbox). TLS valid across all three SANs (noc, mail, webmail) with automated Let's Encrypt renewal. DKIM alignment passing. DMARC reports showing zero spoofing attempts succeeding.

The server has been running production mail — inbound and outbound, with spam filtering, antivirus, DKIM signing, Sieve filtering, and full webmail — without a single service interruption since deployment.

What This Means

Self-hosted email is not dead. It demands more operational discipline than delegating to Google or Microsoft, but the return is complete sovereignty over your communications infrastructure. No vendor lock-in. No opaque filtering. No terms of service that change without notice. No scanning. Full audit transparency — every rule, every log, every configuration is yours to inspect.

The tools are mature. Mailcow handles the mail stack. Docker handles process isolation. Linux kernel parameters handle volumetric defense. Custom scripting fills the gaps between managed components. Automated auditing enforces the discipline of daily verification. Off-box monitoring solves the self-trust problem.

Scaling Up — What Comes Next

The current NOC server proves the methodology at the single-host level. The next phase scales it.

The immediate upgrade path is dedicated hardware — moving from shared compute to bare metal with 64+ GB ECC RAM, NVMe storage arrays, and a 10 Gbps bonded uplink sitting directly on the peering fabric. At that tier, the server doesn't just survive volumetric attacks — it has the raw throughput to observe, classify, and respond to them in real time without upstream scrubbing. The kernel hardening, custom firewall, and audit framework transfer directly. The container stack stays identical. Only the ceiling changes.

Beyond single-host, the architecture is already container-native — and that means Kubernetes is the natural evolution. The 19-container Mailcow stack, the monitoring pipeline, the audit system, the AI-augmented threat analysis — all of it runs in Docker today. Migrating to K8s unlocks horizontal scaling, automated failover, rolling deployments, and pod-level network policy that makes iptables chain management look primitive. Imagine the noc-firewall's 22-point audit translated into Kubernetes NetworkPolicy admission controllers — every pod validated at deploy time, every ingress rule audited continuously, every anomaly flagged before traffic flows.

The monitoring stack scales with it. Prometheus already scrapes the NOC server. In a K8s cluster, it scrapes every node, every pod, every service mesh sidecar. Loki aggregates logs from every container without Syncthing replication hacks — the pipeline becomes native. Grafana dashboards that today show 19 containers on one host would show hundreds of pods across a multi-node cluster, with the same off-box trust model: the telemetry plane never runs on the infrastructure it monitors.

This is where INTRAC is headed. The hardening methodology proven on a single peered server — kernel, firewall, audit, monitor — becomes the security baseline for an entire orchestrated platform. The principles don't change at scale. The kernel parameters propagate to every node via DaemonSet. The firewall logic becomes Calico or Cilium eBPF policy. The nightly audit becomes a continuous compliance scan. The 94/100 score becomes an SLA.

The infrastructure we operate today handles production mail for a domain that's been live since 1998. The infrastructure we're building will handle AI workloads, edge compute, and service mesh traffic at backbone speeds — with the same hardening discipline, the same audit rigor, and the same fundamental principle: if you can't verify it nightly, you don't trust it.

The Methodology Transfers

FreeBSD, Ubuntu, RHEL, Alpine, Talos — the principles are the same regardless of the OS or orchestration layer. Harden the kernel. Isolate the services. Audit the firewall. Monitor from elsewhere. Verify continuously. Score the results.

If you're running exposed infrastructure — whether it's a single mail server or a multi-node cluster — and you want guidance on hardening Linux or FreeBSD systems, reach out. Twenty-seven years of running internet-facing infrastructure is in the methodology. The conversation is free.