Generating Prometheus Metrics from Logs with mtail

Overview

mtail is an open-source log parsing tool developed by Google. It extracts lines matching specific patterns from log files, generates metrics based on them, and exposes them in Prometheus format at the /metrics endpoint. This enables collecting and monitoring business metrics and system state from existing logs using Prometheus.

How to Use mtail

To use mtail, you need to start the mtail program (daemon) and specify the log files to monitor and the mtail program (configuration file) that defines the metrics.

mtail Program (Configuration File) Example: sample.mtail

mtail programs are written in a unique syntax similar to Go.

# Metric definition and initialization
# Define a counter-type metric named 'errors_total'
counter errors_total

# Pattern matching rules within log files
# Increment the errors_total counter if a line contains "error"
/error/ {
  errors_total++
}

# Example: Count HTTP request status codes
# Capture the status code using a regular expression and use it as a label
/^HTTP\/1\.[01] (\d{3})/ {
  http_requests_total[$1]++
}
counter http_requests_total by status_code

• counter errors_total: Defines a metric for use with Prometheus. counter is a counter-type metric whose value monotonically increases.
• /error/ { errors_total++ }: When each log line matches the regular expression /error/, the errors_total counter is incremented by 1.
• counter http_requests_total by status_code: Defines a counter metric called http_requests_total with a status_code label.
• /^HTTP\/1\.[01] (\d{3})/ { http_requests_total[$1]++ }: Extracts status codes (e.g., 200, 404, 500) from HTTP access logs and increments the http_requests_total counter using the status code as a label.

Running mtail with Docker

By running mtail as a Docker container, you can simplify environment setup and deployment.

# Use CentOS 7 as base image
FROM centos:7

# Install wget and clean cache
RUN yum install -y wget && yum clean all

# Set working directory to /tmp
WORKDIR /tmp

# Download mtail binary, extract, and grant execute permission
# Update the release version to the latest as needed
RUN wget -O mtail.tar.gz https://github.com/google/mtail/releases/download/v3.0.0-rc52/mtail_3.0.0-rc52_linux_amd64.tar.gz && \
    tar xzvf mtail.tar.gz && \
    chmod +x mtail

# Command to execute when the container starts
# -progs: Specify the path to the mtail program (configuration file)
# -logs: Specify the path to the log file that mtail monitors
# Example: CMD ["/tmp/mtail", "-progs", "/etc/mtail/sample.mtail", "-logs", "/var/log/nginx/access.log"]
CMD ["/tmp/mtail", "-progs", "/path/to/sample.mtail", "-logs", "/path/to/logfile"]

# Default port where mtail exposes metrics (port for Prometheus to scrape)
EXPOSE 3903

CMD Instruction Arguments

• -progs /path/to/sample.mtail: Specifies the path to the configuration file (mtail program) that mtail uses. Specify the path to the mtail program placed inside the container.
• -logs /path/to/logfile: Specifies the path to the log file that mtail monitors. mtail watches this log file, detects lines matching the patterns defined in the configuration, and generates corresponding metrics. This log file needs to be provided inside the container using Docker volume mounts, for example.

By building and running this Docker image, mtail will monitor logs and expose metrics that Prometheus can collect.

Benchmark: How Fast Is Regex-Based Log Parsing?

At its core, mtail parses each log line with a regular expression and turns matched values into metrics. To get a concrete sense of the throughput this kind of processing achieves, I implemented the same idea in Python and measured it directly.

Benchmark setup

I generated a synthetic Apache-combined-log-style access log with 500,000 lines (63.6 MB) and measured three implementations that extract the HTTP status code from each line and tally counts:

1. naive_regex: calls re.search(pattern_string, line) on every line (no explicitly retained compiled object)
2. compiled_regex: uses .search(line) on a pattern object created once via re.compile()
3. str_split: no regex at all — relies on the fixed log format and extracts the field with str.split('"')

import re
import time

STATUS_RE_STR = r'"\s(\d{3})\s\d+\s"'
STATUS_RE_COMPILED = re.compile(STATUS_RE_STR)

def approach_naive_regex(lines):
    counts = {}
    for line in lines:
        m = re.search(STATUS_RE_STR, line)
        if m:
            status = m.group(1)
            counts[status] = counts.get(status, 0) + 1
    return counts

def approach_compiled_regex(lines):
    counts = {}
    search = STATUS_RE_COMPILED.search
    for line in lines:
        m = search(line)
        if m:
            status = m.group(1)
            counts[status] = counts.get(status, 0) + 1
    return counts

def approach_str_split(lines):
    counts = {}
    for line in lines:
        # "IP - - [ts] "METHOD PATH PROTO" STATUS SIZE "REF" "UA" RESPMS
        parts = line.split('"')
        status = parts[2].split()[0]
        counts[status] = counts.get(status, 0) + 1
    return counts

Each implementation ran 3 times against the same file; I took the fastest run (best-of-3) and derived lines/sec and MB/sec from it (timed with time.perf_counter()).

Measured results

All three implementations produced identical aggregation results (per-status-code counts and summed response time), confirmed by the test run.

Discussion

The pre-compiled regex was 1.35x faster than the naive implementation that calls re.search every time. That said, Python’s re module internally caches up to 512 recently used pattern strings, so even the “uncompiled” version reuses a cached compiled object from the second call onward. The gap we still measured shows that the overhead of that cache lookup (a dictionary lookup) isn’t negligible.

Interestingly, the regex-free str.split-based implementation came out slightly slower than the pre-compiled regex (1.26x vs. naive, or about 0.93x relative to compiled_regex). For this log format, splitting on " and then extracting a token required multiple steps, and the accumulated Python function-call overhead outweighed the cost of a single regex match. So “avoiding regex always makes things faster” doesn’t hold universally — the honest conclusion from this measurement is that the result depends on the interaction between format complexity and the extraction method used.

mtail’s own implementation (in Go) optimizes this kind of string processing at a lower level, but the broad shape of the cost — matching a regex against each log line and turning captured values into metrics — lands in the same several-million-lines-per-second ballpark that this simple benchmark shows.