How I Created a System Information Retrieval and Monitoring Tool

How I Created a System Information Retrieval and Monitoring Tool

Introduction

In today's fast-paced software development and operations world, efficiency and timely access to reliable information are critical. Effective server monitoring and information retrieval are essential for preserving system health and guaranteeing smooth operations.

Modern server environments are complex ecosystems comprising many interrelated components, including web servers, databases, containerized applications, and others. Staying on top of this complex landscape can be difficult for DevOps engineers and system administrators to manage.

Traditional approaches frequently require juggling many command-line programs, reading through extensive configuration files, and manually correlating data from diverse sources.

This method is time-consuming and prone to human error, which may result in issues being neglected or data being misconstrued. In an industry where minutes of downtime can result in large financial losses, there is an obvious need for a more streamlined, effective approach.

To streamline these processes, I created devopsfetch, a command line tool that collects and shows system information such as active ports, user logins, Nginx configurations, Docker images, and container statuses.

In this article, I'll walk you through the steps of creating this tool and deploying it as a systemd service for continuous monitoring, and ensuring correct logging and log rotation.


Overview of Features

  • Ports: Display all active ports and services and provide detailed information about specific ports.

  • Docker: List all Docker images and containers and provide detailed information about specific containers.

  • Nginx: Display all Nginx domains and their ports and provide detailed configuration information for specific domains.

  • Users: List all users and their last login times and provide detailed information about specific users.

  • Time Range: Display activities within a specified time range.


Usage

The tool was designed to be used this way:

  1. Display help
devopsfetch -h  or devopsfetch --help
  1. Display all active ports and services:
devopsfetch -p or devopsfetch --port
  • Display detailed information about a specific port:
devopsfetch -p <port_number>
  1. Display Docker images and containers
devopsfetch -d or devopsfetch --docker
  • Provide detailed information about a specific container:
devopsfetch -d <container_name>
  1. Display Nginx domains and ports:
devopsfetch -n or devopsfetch --nginx
  • Display configuration information for a specific Nginx domain:
devopsfetch -n <domain>
  1. List all users and last login times:
devopsfetch -u  or devopsfetch --users
  • Provide detailed information about a specific user:
devopsfetch -u <username>
  1. To track activities on the server
devopsfetch -t <start_time> <end_time>

Two-step Process of Creating this Tool

The first was to create a script that had all the functions to create the features highlighted above

#!/bin/bash

# Function to display help
devopsfetch_help() {
    echo "Usage: devopsfetch [OPTION]..."
    echo "Retrieve and display system information for DevOps purposes."
    echo
    echo "Options:"
    echo "  -p, --port [PORT_Number]    Display active ports or info about a specific port"
    echo "  -d, --docker [CONTAINER_NAME]  Display Docker images/containers or info about a specific container"
    echo "  -n, --nginx [DOMAIN] Display Nginx domains or info about a specific domain"
    echo "  -u, --users [USER]   Display user logins or info about a specific user"
    echo "  -t, --time [start-date][end-date]     Display activities within a specified time range "
    echo "  -h, --help           Display this help message"
    echo
    echo "For more information, see the full documentation."
    echo "Example: devopsfetch -p 80"
    echo "         devopsfetch -d nginx"
    echo "         devopsfetch -n example.com"
    echo "         devopsfetch -u john"
    echo "         devopsfetch -t '2024-07-22'"
}

# Function to display active ports
ports_available() {
    echo "############################### List of Active Ports #############################"
    if [ -z "$1" ]; then
        echo "+----------------------+-------+------------+"
        printf "| %-20s | %-5s | %-10s |\n" "USER" "PORT" "SERVICE"
        echo "+----------------------+-------+------------+"
        sudo lsof -i -P -n | grep LISTEN | awk '{
            port = $9
            sub(/.*:/, "", port)
            user = $3
            service = $1
            if (length(service) > 10) service = substr(service, 1, 10)
            printf "| %-20s | %-5s | %-10s |\n", user, port, service
        }' | sort -k2 -n | uniq
        echo "+----------------------+-------+------------+"
    else
        echo "Information for port $1:"
        echo "+----------------------+-------+------------+"
        printf "| %-20s | %-5s | %-10s |\n" "USER" "PORT" "SERVICE"
        echo "+----------------------+-------+------------+"
        result=$(sudo lsof -i :$1 -P -n | grep LISTEN | awk '{
            port = $9
            sub(/.*:/, "", port)
            user = $3
            service = $1
            if (length(service) > 10) service = substr(service, 1, 10)
            printf "| %-20s | %-5s | %-10s |\n", user, port, service
        }')
        if [ -z "$result" ]; then
            printf "| %-20s | %-5s | %-10s |\n" "N/A" "$1" "N/A"
        else
            echo "$result"
        fi
        echo "+----------------------+-------+------------+"
    fi
}
# Function to display Docker information
docker_info() {
    echo "##################################### List of Docker Images and Containers ######################################"
    if [ -z "$1" ]; then
        echo "Docker images:"
        docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.ID}}"
        echo
        echo "Docker containers:"
        docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.ID}}\t{{.Status}}"
    else
        echo "Information for Docker container $1:"
        docker inspect $1
    fi
}
# Function to display Nginx information
nginx_server_info() {
    echo "##################################### Nginx Domains and Config Files ######################################"
    if [ -z "$1" ]; then
        # Print the table header
        printf "+-------------------------------+-------------------------------+-------------------------------+----------------------------------------------------+\n"
        printf "| %-30s | %-29s | %-29s | %-49s |\n" "DOMAIN" "PROXY" "CONFIGURATION" "CONFIG FILE"
        printf "+-------------------------------+-------------------------------+-------------------------------+----------------------------------------------------+\n"

        # Loop through the Nginx configuration files
        for config in /etc/nginx/sites-enabled/*; do
            if [ -f "$config" ]; then
                # Extract domain, proxy, and listen information
                domain=$(grep -m1 "server_name" "$config" | awk '{print $2}' | sed 's/;$//')
                proxy=$(grep -m1 "proxy_pass" "$config" | awk '{print $2}' | sed 's/;$//')
                listen=$(grep -m1 "listen" "$config" | awk '{print $2}' | sed 's/;$//')

                config_info="listen: $listen"

                # Print the row for each configuration file
                printf "| %-30s | %-29s | %-29s | %-49s |\n" "${domain:-N/A}" "${proxy:-<No Proxy>}" "$config_info" "$config"
            fi
        done

        # Print the table footer
        printf "+-------------------------------+-------------------------------+-------------------------------+----------------------------------------------------+\n"
    else
        echo "Nginx configuration for domain $1:"
        grep -A 20 -R "server_name $1" /etc/nginx/sites-enabled/* | sed -n '/server_name/,$p'
    fi
}
# Function to display user information
get_users() {
     echo "##################################### List of Regular Users ######################################"
    if [ -z "$1" ]; then
        printf "+-----------------+------------+-------------------------------+\n"
        printf "| %-15s | %-10s | %-30s |\n" "Users" "Status" "Last-Login"
        printf "+-----------------+------------+-------------------------------+\n"

        awk -F: '$3 >= 1000 { print $1 }' /etc/passwd | while read user; do
            status="Regular"
            last_login=$(last -n 1 "$user" | head -n 1 | awk '{print $4, $5, $6, $7, $8, $9}')
            if [[ -n "$last_login" ]]; then
                printf "| %-15s | %-10s | %-30s |\n" "$user" "$status" "$last_login"
            else
                printf "| %-15s | %-10s | %-30s |\n" "$user" "$status" "Never logged in"
            fi
        done

        printf "+-----------------+------------+-------------------------------+\n"
    else
        # Display details for a specific user
        if id "$1" &>/dev/null; then
            printf "%-10s %-10s %-20s %-20s\n" "UID" "GID" "Groups" "Last-Login"
            echo "-------------------------------------------------------------------------"

            uid=$(id -u "$1")
            gid=$(id -g "$1")
            groups=$(id -Gn "$1" | tr ' ' '\n')
            last_login=$(last -n 1 "$1" | head -n 1 | awk '{print $4, $5, $6, $7, $8, $9}')
            if [[ -z "$last_login" ]]; then
                last_login="Never logged in"
            fi

            printf "%-10s %-10s %-20s %-20s\n" "$uid" "$gid" "$(echo "$groups" | head -n1)" "$last_login"
            echo "$groups" | tail -n +2 | sed 's/^/                    /'
        else
            echo "User $1 does not exist."
        fi
    fi
}
# Function to filter system logs based on time range
filter_logs() {
    local start_date="$1"
    local end_date="$2"

    # If only one date is provided, set end_date to current date
    if [ -z "$end_date" ]; then
        end_date=$(date +"%Y-%m-%d")
    fi

    # Convert dates to the format journalctl expects
    local start_time="${start_date} 00:00:00"
    local end_time="${end_date} 23:59:59"

    echo "Displaying system logs from $start_time to $end_time"

    # Use journalctl to display logs within the specified time range
    journalctl --since "$start_time" --until "$end_time"

    # Check if journalctl command was successful
    if [ $? -ne 0 ]; then
        echo "Error occurred while fetching logs."
        echo "Available log range:"
        journalctl --list-boots
    fi
}
# Main script logic
case "$1" in
    -p|--port)
        ports_available "$2"
        ;;
    -d|--docker)
        docker_info "$2"
        ;;
    -n|--nginx)
        nginx_server_info "$2"
        ;;
    -u|--users)
        get_users "$2"
        ;;
    -t|--time)
        if [ -z "$2" ]; then
            echo "Please specify at least a start date (YYYY-MM-DD)"
        else
            filter_logs "$2" "$3"
        fi
        ;;
    -h|--help)
        devopsfetch_help
        ;;
    *)
        echo "Invalid option. Use -h or --help for usage information."
        ;;
esac

The next step was to create another script to install the one above

#!/bin/bash
# Ensure the script is run as root
if [ "$EUID" -ne 0 ]; then
  echo "Please run this script as root or using sudo."
  # Rerun the script with sudo
  sudo -E "$0" "$@"
  exit 1
fi

# Install dependencies
echo "####################### Installing Dependencies... ####################################"
sudo apt-get update
sudo apt-get install -y docker.io nginx

# Copy the main script to /usr/local/bin
echo "########################## Copy the devopsfetch script to the bin directoty #######################"
sudo cp devopsfetch.sh /usr/local/bin/devopsfetch

# Make the script executable
echo "########################## Making the devopsfetch command line script executable #######################"
chmod +x /usr/local/bin/devopsfetch

# Create the logfile
echo "####################### Creating log file...####################################"
create_log_directory() {
    local dir="/var/log"

    # Check if directory exists
    if [ ! -d "$dir" ]; then
        echo "Directory $dir does not exist. Creating..."
        mkdir -p "$dir"
        echo "Directory $dir created."
    else
        echo "Directory $dir already exists."
    fi
}

touch /var/log/devopsfetch.log
chmod 666 /var/log/devopsfetch.log

echo "####################### Creating monitoring script...####################################"
cat << 'EOF' | sudo tee /usr/local/bin/devopsfetch_monitor.sh
#!/bin/bash

LOG_FILE="/var/log/devopsfetch.log"

while true; do
    timestamp=$(date "+%Y-%m-%d %H:%M:%S")
    echo "---" | sudo tee -a "${LOG_FILE}"
    echo "${timestamp}: Running DevOpsFetch" | sudo tee -a "${LOG_FILE}"
    echo "${timestamp}: Ports" | sudo tee -a "${LOG_FILE}"
    /usr/local/bin/devopsfetch -p | sudo tee -a "${LOG_FILE}"
    echo "${timestamp}: Docker" | sudo tee -a "${LOG_FILE}"
    /usr/local/bin/devopsfetch -d | sudo tee -a "${LOG_FILE}"
    echo "${timestamp}: Nginx" | sudo tee -a "${LOG_FILE}"
    /usr/local/bin/devopsfetch -n | sudo tee -a "${LOG_FILE}"
    echo "${timestamp}: Users" | sudo tee -a "${LOG_FILE}"
    /usr/local/bin/devopsfetch -u | sudo tee -a "${LOG_FILE}"
    sleep 3600  # Run every 1 hr
done
EOF

# Make the monitoring script executable
echo "######################### Making the monitoring for devopsfetch executable  ####################"
chmod +x /usr/local/bin/devopsfetch_monitor.sh

# Set up log rotation
echo "######################### Creating a log rotation file  ####################"
cat << EOF | sudo tee /etc/logrotate.d/devopsfetch
/var/log/devopsfetch.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
}
EOF

# Create systemd service file
echo "######################### Creating a systemd service file  ####################"
cat << EOF | sudo tee /etc/systemd/system/devopsfetch.service
[Unit]
Description=DevOpsFetch Service
After=network.target
[Service]
ExecStart=/usr/local/bin/devopsfetch_monitor.sh
Restart=always
[Install]
WantedBy=multi-user.target
EOF

# Enable and start the service
sudo systemctl enable devopsfetch.service
sudo systemctl start devopsfetch.service
sudo systemctl daemon-reload
sudo systemctl enable devopsfetch.service
sudo systemctl start devopsfetch.servic

echo "Installation complete. DevOpsFetch is now installed and the monitoring service is running."
echo "You can use 'devopsfetch' command to retrieve system information."
echo "Logs are being written to /var/log/devopsfetch.log"

To always view the logs, a monitoring script was created to run indefinitely and output the information executed in the first script every hour.

Log rotation was configured for /var/log/devopsfetch.log. The log file will be rotated daily, keeping 7 days of logs, and will be compressed to save space.

Outputs from the Usage of devopsfetch

For Active Ports

zenitugo@Zenitugo:~$ devopsfetch -p
############################### List of Active #################
+----------------------+-------+------------+
| USER                 | PORT  | SERVICE    |
+----------------------+-------+------------+
| epmd                 | 4369  | epmd       |
| postgres             | 5432  | postgres   |
| rabbitmq             | 25672 | beam.smp   |
| rabbitmq             | 5672  | beam.smp   |
| root                 | 4369  | systemd    |
| systemd-resolve      | 53    | systemd-r  |
+----------------------+-------+------------+

Detailed information for port number 5432

zenitugo@Zenitugo:~$ devopsfetch -p 5432
####### List of Active Ports ###########
Information for port 5432:
+----------------------+-------+------------+
| USER                 | PORT  | SERVICE    |
+----------------------+-------+------------+
| postgres             | 5432  | postgres   |
+----------------------+-------+------------+

To learn how to use the tool

zenitugo@Zenitugo:~$ devopsfetch -h  or devopsfetch --help
Usage: devopsfetch [OPTION]...
Retrieve and display system information for DevOps purposes.

Options:
  -p, --port [PORT_Number]    Display active ports or info about a specific port
  -d, --docker [CONTAINER_NAME]  Display Docker images/containers or info about a specific container
  -n, --nginx [DOMAIN] Display Nginx domains or info about a specific domain
  -u, --users [USER]   Display user logins or info about a specific user
  -t, --time [start-date][end-date]     Display activities within a specified time range
  -h, --help           Display this help message

For more information, see the full documentation.
Example: devopsfetch -p 80
         devopsfetch -d nginx
         devopsfetch -n example.com
         devopsfetch -u john
         devopsfetch -t '2024-07-22'

For Docker images and containers

zenitugo@Zenitugo:~$ devopsfetch --docker
########### List of Docker Images and Containers ###########
Docker images:
REPOSITORY       TAG          IMAGE ID
appimg           latest       99824736de6b
python           latest       d1d39f5c5b14
<none>           <none>       874e811b2479
Docker containers:
NAMES                  IMAGE              CONTAINER ID   STATUS
flask                  appimg             dd11b028d264   Exited (255) 5 weeks ago
condescending_nash     03956cca7569       9904035f27cc   Exited (1) 5 weeks ago
voting-site-server-1   voting-site-server fe7795bf40ad   Exited (255) 2 months ago
heal                   zenitugo/healet    62a349f8c027   Created

To display the Nginx domain and ports

zenitugo@Zenitugo:~$ devopsfetch -n
##################################### Nginx Domains and Config Files ######################################
+-------------------------------+-------------------------------+-------------------------------+----------------------------------------------------+
| DOMAIN                         | PROXY                         | CONFIGURATION                 | CONFIG FILE                                       |
+-------------------------------+-------------------------------+-------------------------------+----------------------------------------------------+
| catony.com                     | http://localhost:8000         | listen: 80                    | /etc/nginx/sites-enabled/catony.conf              |
| chino.com                      | http://localhost:7000         | listen: 80                    | /etc/nginx/sites-enabled/chino.conf               |
| _                              | <No Proxy>                    | listen: 80                    | /etc/nginx/sites-enabled/default                  |
| hng.com                        | http://localhost:5000         | listen: 80                    | /etc/nginx/sites-enabled/hng.conf                 |
| test.com                       | /                             | listen: 80                    | /etc/nginx/sites-enabled/test.conf                |
+-------------------------------+-------------------------------+-------------------------------+----------------------------------------------------+

To display all active regular users and their last login

zenitugo@Zenitugo:~$ devopsfetch -u
################ List of Regular Users ######################
+-----------------+------------+-------------------------------+
| Users           | Status     | Last-Login                     |
+-----------------+------------+-------------------------------+
| nobody          | Regular    |                                |
| zenitugo        | Regular    | Aug 24 07:37 gone - no         |
| Ugochi          | Regular    |                                |
| Temi            | Regular    |                                |
| Tim             | Regular    |                                |
| Uriel           | Regular    |                                |
| Harry           | Regular    |                                |
| Benita          | Regular    |                                |
| Ann             | Regular    |                                |
| Felix           | Regular    |                                |
| Kelvin          | Regular    |                                |
| Stella          | Regular    |                                |
| dhebby          | Regular    |                                |
| ugochi          | Regular    |                                |
| hng             | Regular    |                                |
+-----------------+------------+-------------------------------+

Note: The reason you don't see the last login for the other users is because they were newly created and I have never tried to log in as the users. For the user "zenitugo", that's me. It shows I am still currently login in because at the moment I am.

Track your Activities

To keep track of all this information, you don't need to type the command on the terminal because the installation script creates a systemd service file that is set up devopsfetch_monitor.sh to run as a service. The service will restart automatically if it fails and will start on boot.

All you need to do is run the command below every hour.

cat /var/log/devopsfetch.log

Conclusion

Developing devopsfetch has been a fantastic experience via system information retrieval and monitoring. By merging many functionalities into a single program, I was able to improve how to manage and monitor vital system resources.

This project not only helped me better understand system monitoring but also helped me understand how to use case statements in Bash scripting.

I invite you to explore devopsfetch and tailor it to your requirements. Creating this application has taught me the importance of developing tailored solutions for system administration.