Nginx Log Management in Ubuntu

In this tutorial, we've covered Nginx log management in Ubuntu server. Advanced logging and rotation techniques. 

Nginx is a popular web server and reverse proxy server known for its performance and efficiency. Proper logging and log rotation are essential for monitoring and maintaining the health of your Nginx server. This tutorial will guide you through configuring logging and setting up log rotation in Nginx on Ubuntu.

Prerequisites

• A server running Ubuntu dedicated server or KVM VPS (version 20.04 or later).
• Nginx installed and running on your server.
• Root or sudo privileges.

Nginx Log Management in Ubuntu

Step 1: Locate Nginx Configuration Files

Nginx configuration files are typically located in the /etc/nginx/ directory. The main configuration file is /etc/nginx/nginx.conf, and site-specific configurations are usually found in the /etc/nginx/sites-available/ directory.

Step 2: Configure Nginx Logging

By default, Nginx logs two types of data:

• Access Logs: These logs record all requests processed by the server.
• Error Logs: These logs capture any errors encountered during processing requests.

2.1 Access Logs

Access logs are usually configured in the server block of your Nginx configuration. Open the main Nginx configuration file or a specific site configuration file:

sudo nano /etc/nginx/nginx.conf

Or for a specific site:

sudo nano /etc/nginx/sites-available/your-site.conf

In the server block, add or modify the access_log directive:

server {
    listen 80;
    server_name your_domain.com;

    access_log /var/log/nginx/your_site_access.log;
    error_log /var/log/nginx/your_site_error.log;

    # Other configurations...
}

• /var/log/nginx/your_site_access.log: Path to the access log file.
• /var/log/nginx/your_site_error.log: Path to the error log file.

2.2 Error Logs

The error logs can be configured similarly using the error_log directive:

error_log /var/log/nginx/your_site_error.log;

You can specify the log level for error logging (e.g., error, warn, crit, etc.):

error_log /var/log/nginx/your_site_error.log warn;

Step 3: Test Nginx Configuration

After configuring the logs, it's important to test the Nginx configuration to ensure there are no syntax errors:

sudo nginx -t

If the configuration test is successful, restart Nginx to apply the changes:

sudo systemctl restart nginx

Step 4: Configure Log Rotation with Logrotate

Logrotate is a utility in Linux used to manage log files, allowing automatic rotation, compression, and deletion of old log files.

4.1 Create a Logrotate Configuration File for Nginx

Create a new Logrotate configuration file specifically for Nginx:

sudo nano /etc/logrotate.d/nginx

Add the following configuration:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

Explanation of the parameters:

• daily: Rotate logs daily.
• missingok: Do not issue an error if the log file is missing.
• rotate 14: Keep 14 days’ worth of logs.
• compress: Compress the rotated logs to save space.
• delaycompress: Delay compression until the next rotation cycle.
• notifempty: Do not rotate empty log files.
• create 0640 www-data adm: Create new log files with specified permissions and ownership.
• sharedscripts: Run the postrotate script only once, no matter how many logs are rotated.
• postrotate: This script will be executed after the log file is rotated. It sends a signal to Nginx to reopen the log files.

4.2 Test Logrotate Configuration

You can test the log rotation for Nginx using the following command:

sudo logrotate -d /etc/logrotate.d/nginx

This command will simulate the log rotation process without actually rotating the logs. If everything looks good, you can run the actual log rotation:

sudo logrotate -f /etc/logrotate.d/nginx

1. Advanced Log Format Customization

Nginx allows you to customize the log format to include specific details like request time, upstream response time, and more.

Custom Log Format

Edit the Nginx configuration file to define a custom log format:

sudo nano /etc/nginx/nginx.conf

Add the following log_format directive under the http block:

http {
    log_format custom '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" '
                      '$request_time $upstream_response_time $pipe';

    access_log /var/log/nginx/custom_access.log custom;
    
    # Other configurations...
}

This custom log format includes:

• $remote_addr: The client’s IP address.
• $remote_user: The username provided by the client.
• $time_local: The local time when the request was received.
• $request: The request line from the client.
• $status: The HTTP status code sent to the client.
• $body_bytes_sent: The number of bytes sent to the client.
• $http_referer: The Referer header from the client.
• $http_user_agent: The User-Agent header from the client.
• $request_time: The total time spent processing the request.
• $upstream_response_time: The time taken to receive the response from the upstream server.
• $pipe: "p" if the request was pipelined, "." otherwise.

After editing, test the configuration:

sudo nginx -t

Then, restart Nginx:

sudo systemctl restart nginx

2. Limiting Log File Size

If you're concerned about the log files growing too large between rotations, you can implement a size-based log rotation using Logrotate.

Size-Based Log Rotation

Open the Nginx Logrotate configuration file:

sudo nano /etc/logrotate.d/nginx

Modify it to rotate the logs when they reach a certain size:

/var/log/nginx/*.log {
    size 100M
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

This configuration rotates the logs when they reach 100MB in size.

3. Filtering Logs Based on Status Code

You can configure Nginx to log only specific status codes, such as errors, to separate log files.

Logging Only 4xx and 5xx Errors

Add a conditional access_log directive in your site configuration:

sudo nano /etc/nginx/sites-available/your-site.conf

Within the server block, add:

map $status $loggable {
    ~^[45] 1;
    default 0;
}

access_log /var/log/nginx/error_access.log combined if=$loggable;

This configuration uses a map directive to create a variable $loggable, which is set to 1 for 4xx and 5xx status codes and 0 otherwise. The access_log directive logs requests only if $loggable is 1.

4. Real-Time Log Monitoring

You can monitor Nginx logs in real-time to observe incoming requests or errors as they happen.

Using tail with Filtering. To monitor access logs in real-time:

sudo tail -f /var/log/nginx/access.log

For real-time monitoring of errors:

sudo tail -f /var/log/nginx/error.log

To filter and watch only certain types of requests, such as 404 errors:

sudo tail -f /var/log/nginx/access.log | grep "404"

5. Conditional Logging Based on Request

You can log only specific types of requests, such as those with a certain header or from a certain IP address.

Example: Log Requests from a Specific IP Address

Edit your site configuration file:

sudo nano /etc/nginx/sites-available/your-site.conf

Add a conditional logging rule:

map $remote_addr $loggable_ip {
    default 0;
    192.168.1.100 1;
}

access_log /var/log/nginx/special_access.log combined if=$loggable_ip;

This configuration logs only requests from the IP address 192.168.1.100.

6. Using ngrep for HTTP Traffic Analysis

ngrep is a network packet analyzer that you can use to capture and analyze HTTP traffic going through your Nginx server.

Install ngrep

sudo apt-get install ngrep -y

Capture HTTP Traffic

To capture HTTP traffic on port 80:

sudo ngrep -d any -W byline port 80

This command will display the HTTP traffic passing through port 80 in real-time, which can be helpful for debugging or monitoring.

7. Implementing GeoIP Logging

Nginx can log the geographical location of the client by integrating GeoIP modules.

Install GeoIP Modules

sudo apt-get install libnginx-mod-http-geoip -y

Configure GeoIP in Nginx

Edit the Nginx configuration file:

sudo nano /etc/nginx/nginx.conf

Add the following under the http block:

http {
    geoip_country /usr/share/GeoIP/GeoIP.dat;
    geoip_city /usr/share/GeoIP/GeoLiteCity.dat;

    log_format geo '$remote_addr - $remote_user [$time_local] '
                   '"$request" $status $body_bytes_sent '
                   '"$http_referer" "$http_user_agent" '
                   '$geoip_country_name $geoip_city';
    
    access_log /var/log/nginx/geo_access.log geo;
    
    # Other configurations...
}

Note: We had issue with GeoLiteCity.dat file. It doesn't create during the installation. Please find way to fix it.

This setup will log the country and city information based on the client's IP address.

Conclusion

You have now successfully configured logging and log rotation for Nginx on your Ubuntu server. Regular log rotation ensures that your server’s disk space is managed effectively, preventing logs from consuming too much space over time. By following this tutorial, you can maintain a robust logging mechanism, allowing you to monitor and troubleshoot your Nginx server efficiently.

These advanced commands and configurations can help you gain deeper insights into your Nginx server's operation, improve logging efficiency, and troubleshoot issues more effectively. By implementing these techniques, you can take full control of your server's logging infrastructure.