Every system administrator would be wise to keep a close eye on the health and performance of their server infrastructure. Prometheus, an open-source monitoring and alerting toolkit, empowers system administrators to gain deep insights into their Linux servers, ensuring optimal performance and availability. In this article, I'll walk you through the step-by-step process of installing Prometheus, configuring it to monitor various metrics, adding authentication for secure access, and incorporating exporters to enhance its monitoring capabilities.
Installation
Go to https://prometheus.io/download/.
In the Operating System dropdown menu, choose linux.
In the Architecture dropdown menu, choose amd64 (or whichever architecture your server uses).
In the prometheus section, find the version that is labeled LTS.
Copy the listed URL. It will be labeled something like prometheus-2.37.6.linux-amd64.tar.gz.
The exact filename will likely differ depending on when you're reading this.
In your terminal, use wget
to download the file at the copied URL.
wget https://github.com/prometheus/prometheus/releases/download/v2.37.6/prometheus-2.37.6.linux-amd64.tar.gz
The URL in the line above is just an example, use the actual URL you copied earlier.
The file will be downloaded to your machine. Unzip it.
tar xvfz prometheus-2.37.6.linux-amd64.tar.gz
To stay organized, you can remove the compressed file as it's no longer needed.
rm prometheus-2.37.6.linux-amd64.tar.gz
Move all of the Prometheus files to /opt/prometheus
.
sudo mv prometheus-2.37.6.linux-amd64 /opt/prometheus
Create a new user that will run the Prometheus daemon.
sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus
Set the new user as the owner of /opt/prometheus
.
sudo chown -R prometheus:prometheus /opt/prometheus
Create a new systemd service file called /etc/systemd/system/prometheus.service
.
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
WorkingDirectory=/opt/prometheus
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml
ReadWriteDirectories=/opt/prometheus
[Install]
WantedBy=multi-user.target
Reload systemd daemons.
sudo systemctl daemon-reload
Start the Prometheus service.
sudo systemctl start prometheus
Enable the Prometheus service so it starts when the system starts.
sudo systemctl enable prometheus
Verify it works by going to http://[your server's IP address/hostname]:9090.
Add Authentication
Right now, anyone who visits the URL will see all monitoring information. This could leak sensitive data. Prometheus has built-in basic authentication available for configuration.
First, a password hash must be created. There are many methods to do this. I suggest using Python for this as many Linux distributions come preinstalled with Python 3.
Install python3-bcrypt
. It is a 3rd-party Python package that helps creating bcrypt hashes.
sudo apt update
sudo apt install -y python3-bcrypt
Create a file called gen-pass.py
.
import getpass
import bcrypt
password = getpass.getpass("password: ")
hashed_password = bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt())
print(hashed_password.decode())
Run the file with Python.
python3 gen-pass.py
You will be prompted for a password. Enter a strong and unique password that you will use to log in to Prometheus's web user interface, then press Enter.
The password hash is output. Copy it.
Create a file called /opt/prometheus/web.yml
.
basic_auth_users:
[username]: [copied password hash]
For the username, choose any username you like. For the password hash, paste the password hash you copied earlier.
The brackets [] are only there for demonstration and should not be included in your file.
Edit /etc/systemd/system/prometheus.service
to change the ExecStart
line.
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --web.config.file=/opt/prometheus/web.yml
Reload systemd daemons.
sudo systemctl daemon-reload
Restart the Prometheus service.
sudo systemctl restart prometheus
Now, when you access Prometheus in your web browser, a username/password prompt will appear. Enter the username and password you chose earlier to gain access.
Adding an Exporter
Prometheus comes with some basic metrics to monitor by default, but you'll want to use "exporters" which provide access to even more metrics. There are metrics for...
Your hardware and OS: the "Node exporter"
node_exporter
MySQL/MariaDB: the "MySQL Server Exporter"
mysqld_exporter
Many, many more
Create a directory to store the exporters.
sudo mkdir /opt/prometheus/exporters
Go to https://prometheus.io/download/.
In the Operating System dropdown menu, choose linux.
In the Architecture dropdown menu, choose amd64.
Find the exporter you want to install. In this example, we'll go with the Node exporter.
In the node_exporter section, copy the listed URL. It will be labeled something like node_exporter-1.5.0.linux-amd64.tar.gz.
In your terminal, download the file at the copied URL.
wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
Unzip the file.
tar xvfz node_exporter-1.5.0.linux-amd64.tar.gz
Remove the compressed file.
rm node_exporter-1.5.0.linux-amd64.tar.gz
Move the exporter executable to the exporters directory.
sudo mv node_exporter-1.5.0.linux-amd64/node_exporter /opt/prometheus/exporters
Remove the remaining files.
rm -rf node_exporter-1.5.0.linux-amd64
The binary you just copied is still owned by your user account. Change the ownership of it (and all other files in /opt/prometheus
for good measure) to the prometheus
user you created earlier.
sudo chown -R prometheus:prometheus /opt/prometheus
Create a new systemd service file at /etc/systemd/system/prometheus_node_exporter.service
.
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/exporters/node_exporter
[Install]
WantedBy=multi-user.target
Reload systemd daemons
sudo systemctl daemon-reload
Start the exporter
sudo systemctl start prometheus_node_exporter
Enable the exporter so it starts when the system starts.
sudo systemctl enable prometheus_node_exporter
Edit /opt/prometheus/prometheus.yml
.
Indented under the scrape_configs
section, add the following.
- job_name: node
static_configs:
- targets: ['localhost:9100']
Restart the Prometheus service.
sudo systemctl restart prometheus
Verify it works by going to http://[your server IP address]:9090/graph?g0.expr=rate(node_disk_io_time_seconds_total[1m])
. That page will show you the rate of I/O operations of your system disks.
Repeat the process to add as many exporters as you need to fulfill your server monitoring needs.
Implementing Prometheus as your Linux server monitoring solution equips you with powerful tools to keep your infrastructure in check. By following the steps outlined in this guide, you have effectively set up Prometheus, configured it to monitor essential metrics, strengthened security through authentication, and expanded monitoring capabilities with exporters. With Prometheus in place, you now have the ability to proactively identify and address issues, optimize performance, and ensure the stability and reliability of your Linux server environment.
Cover photo by Safwan Thottoli on Unsplash.