Generating Prometheus Metrics from Logs with mtail

Overview

mtail is an open-source log parsing tool developed by Google. It extracts lines matching specific patterns from log files, generates metrics based on them, and exposes them in Prometheus format at the /metrics endpoint. This enables collecting and monitoring business metrics and system state from existing logs using Prometheus.

How to Use mtail

To use mtail, you need to start the mtail program (daemon) and specify the log files to monitor and the mtail program (configuration file) that defines the metrics.

mtail Program (Configuration File) Example: sample.mtail

mtail programs are written in a unique syntax similar to Go.

# Metric definition and initialization
# Define a counter-type metric named 'errors_total'
counter errors_total

# Pattern matching rules within log files
# Increment the errors_total counter if a line contains "error"
/error/ {
  errors_total++
}

# Example: Count HTTP request status codes
# Capture the status code using a regular expression and use it as a label
/^HTTP\/1\.[01] (\d{3})/ {
  http_requests_total[$1]++
}
counter http_requests_total by status_code

• counter errors_total: Defines a metric for use with Prometheus. counter is a counter-type metric whose value monotonically increases.
• /error/ { errors_total++ }: When each log line matches the regular expression /error/, the errors_total counter is incremented by 1.
• counter http_requests_total by status_code: Defines a counter metric called http_requests_total with a status_code label.
• /^HTTP\/1\.[01] (\d{3})/ { http_requests_total[$1]++ }: Extracts status codes (e.g., 200, 404, 500) from HTTP access logs and increments the http_requests_total counter using the status code as a label.

Running mtail with Docker

By running mtail as a Docker container, you can simplify environment setup and deployment.

# Use CentOS 7 as base image
FROM centos:7

# Install wget and clean cache
RUN yum install -y wget && yum clean all

# Set working directory to /tmp
WORKDIR /tmp

# Download mtail binary, extract, and grant execute permission
# Update the release version to the latest as needed
RUN wget -O mtail.tar.gz https://github.com/google/mtail/releases/download/v3.0.0-rc52/mtail_3.0.0-rc52_linux_amd64.tar.gz && \
    tar xzvf mtail.tar.gz && \
    chmod +x mtail

# Command to execute when the container starts
# -progs: Specify the path to the mtail program (configuration file)
# -logs: Specify the path to the log file that mtail monitors
# Example: CMD ["/tmp/mtail", "-progs", "/etc/mtail/sample.mtail", "-logs", "/var/log/nginx/access.log"]
CMD ["/tmp/mtail", "-progs", "/path/to/sample.mtail", "-logs", "/path/to/logfile"]

# Default port where mtail exposes metrics (port for Prometheus to scrape)
EXPOSE 3903

CMD Instruction Arguments

• -progs /path/to/sample.mtail: Specifies the path to the configuration file (mtail program) that mtail uses. Specify the path to the mtail program placed inside the container.
• -logs /path/to/logfile: Specifies the path to the log file that mtail monitors. mtail watches this log file, detects lines matching the patterns defined in the configuration, and generates corresponding metrics. This log file needs to be provided inside the container using Docker volume mounts, for example.

By building and running this Docker image, mtail will monitor logs and expose metrics that Prometheus can collect.