Server management is not only about the initial configuration of services. It also involves monitoring these services and ensuring that they run as smoothly as possible. One of the most important sources of knowledge for administrators is log files, which contain information about system events.
For web servers (such as Nginx), the log contains valuable information about each attempt to access resources through the web server. Every website visitor and the images seen or downloaded files are carefully registered in the log. When errors occur, they are also saved in the log. It is much easier to use a well-structured log file.
In this guide, we will learn how to use Nginx's logging module. We will set up separate log files for different server blocks, and then customize the log output. We will also add additional information about the request to the access log (in the example of this tutorial, the additional information is the time required to service the request), which is beyond what Nginx contains by default.
To follow this tutorial, you need:
In this step, we will create multiple test files in the default Nginx website directory. We will use them to test our log configuration.
When Nginx (or any other web server) receives an HTTP request for a file, it opens the file and transmits its content over the network to provide it to the user. The smaller the file, the faster the transfer speed. When the file is completely transferred, the request is considered complete, and then the record is transferred.
Later in this tutorial, we will modify the logging configuration to include useful information about how much time each request takes. The easiest way to test the modified configuration and pay attention to the differences between different requests is to create multiple test files of different sizes, which will be transferred at different times.
Let's use truncate to create a 1 megabyte file named 1mb.test in the default Nginx directory.
sudo truncate -s 1M /var/www/html/1mb.test
Similarly, we create two more files of different sizes, first 10 megabytes, then 100 megabytes, and name them accordingly.
sudo truncate -s 10M /var/www/html/10mb.test
sudo truncate -s 100M /var/www/html/100mb.test
Last but not least, let's create an empty file:
sudo touch /var/www/html/empty.test
In the next step, we will use these files to fill the log file with the default configuration, and then we will demonstrate the customized configuration later in this tutorial.
The logging module is the core Nginx module, which means it can be used without a separate installation. However, the default configuration is the minimum. In this step, we will see how the default configuration works.
In a fresh installation, Nginx logs all requests to two separate files: access log and error log. The error log located in /var/log/nginx/error.log stores information about abnormal server errors or processing request errors.
The access log located in /var/log/nginx/access.log is more commonly used. This is where all the information requested by Nginx is saved. In this log, you can see which files the user is accessing, which web browsers they are using, which IP addresses they have, and the HTTP status code that Nginx responds to each request.
Let's see what a sample line of the access log file looks like. First, request the empty file we created from Nginx in step 1, so that the log file will not be empty.
curl -i http://localhost/empty.test
In response, you should see several HTTP response headers:
HTTP/1.1200 OK
Server: nginx/1.10.0(Ubuntu)
Date: Thu,30 Jun 201618:10:15 GMT
Content-Type: application/octet-stream
Content-Length:0
Last-Modified: Thu,30 Jun 201618:10:07 GMT
Connection: keep-alive
ETag:"5775607f-0"
Accept-Ranges: bytes
Through this response, you can understand the following:
HTTP/1.1 200 OK tells us that when Nginx responds with a 200 OK status code, it tells us that there is no error. Content-Length: 0 means that the returned document is zero length.Thu, 30 Jun 2016 18:10:15 GMT.Let's see if this matches what Nginx stores in its access log. The log files can only be read by administrative users, so sudo must be used to access them.
sudo tail /var/log/nginx/access.log
The log will contain a line like this, corresponding to the test request we posted earlier.
127.0.0.1- - [30 /Jun/2016:14:10:15-0400]"GET /empty.test HTTP/1.1"2000"-""curl/7.47.0"
Nginx uses combined log format, which is a standard format for access logs that web servers often use for interoperability. In this format, each piece of information is separated by a space; hyphens represent missing information.
From left to right, the categories are:
curl locally, the address points to the local host - 127.0.0.1.GET), the file requested by the path (/empty.text), and the protocol used (HTTP/1.1).200 OK, which means success.0 because the file is empty.curl.Even a single log entry in the access log contains a lot of valuable information about the request. However, when an important piece of information is missing, although we have requested the exact location of http://localhost/empty.test, only the path of the /empty.test file is in the log entry; the hostname ( The information of localhost here will also be lost.
Next, we will overwrite the default log configuration (Nginx stores an access log file for all requests) and let Nginx store a separate log file for the default server block that comes with clean Nginx installation. You can familiarize yourself with the Nginx server block by reading related articles in [Tencent Cloud + Community] (https://cloud.tencent.com/developer?from=10680).
It is a good practice to store separate log files for each server block, which can effectively separate logs from different websites from each other. This not only makes the log file smaller, but also importantly makes the log easier to analyze to find errors and suspicious activity.
To change the default Nginx server block configuration, open the server block Nginx configuration file in nano or another text editor of your choice.
sudo nano /etc/nginx/sites-available/default
Find the server configuration block as shown below:
...
# Default server configuration
#
server {
listen 80 default_server;
listen [::]:80 default_server;
...
And add the last two lines to the configuration:
...
# Default server configuration
#
server {
listen 80 default_server;
listen [::]:80 default_server;
access_log /var/log/nginx/default-access.log;
error_log /var/log/nginx/default-error.log;...
The access_log directive sets the file path for storing the access log, and performs the same operation on the error_log error log. We use the same directory as the default Nginx logs (/var/log/nginx), but use a different file name. If you have multiple server blocks, it is best to name the log files in a consistent and meaningful way, such as using the domain name in the file name.
Save and close the file to exit.
**Note: **Remember that in order to maintain a separate log file for each server block, the above configuration changes must be applied every time a new server block is created in the Nginx configuration.
To enable the new configuration, restart Nginx.
sudo systemctl restart nginx
To test the new configuration, perform the same request to our empty test file as before.
curl -i http://localhost/empty.test
Check if the same log line as the one we saw before is written to the separate file we just configured.
sudo tail /var/log/nginx/default-access.log
In the next step, we will customize the log format in this new file and include other information.
Here, we will set up a custom logging format to make Nginx log other information (how long does it take to process the request), and configure the default server block to use this new format.
We need to define a new log format before using it. In Nginx, each log format has a unique name, which is global to the entire server. Individual server blocks can be configured to later use these formats only by quoting their names.
To define a new logging format, create a new configuration file named timed-log-format.conf in the Nginx additional configuration directory.
sudo nano /etc/nginx/conf.d/timed-log-format.conf
Add the following content:
log_format timed '$remote_addr - $remote_user [$time_local] ''"$request" $status $body_bytes_sent ''"$http_referer" "$http_user_agent" $request_time';
Save and close the file to exit.
The log_format setting instruction defines a new log format. The next element is a unique identifier for this format; here we use timing, but you can choose any name.
Next is the log format itself, which is divided into three lines for readability. Nginx discloses information about all requests in named system variables beginning with a dollar sign. When writing the request details to the access log, these will be replaced by the actual information about the request (for example, $request_addr will be replaced with the visitor's IP address).
The above format is the same as the general log format discussed earlier, with only one difference: the $request_time system variable is added at the end. Nginx uses this variable to store the time (in milliseconds) the request took, and by using this variable in our log format, we tell Nginx to write this information to the log file.
Now we have defined a custom log format called timed in the Nginx configuration, but the default server block has not yet used this format. Next, open the server block Nginx configuration file.
sudo nano /etc/nginx/sites-available/default
Find the configuration block server that we modified earlier, and add the name of the timed log format to the access_log setting:
...
# Default server configuration
#
server {
listen 80 default_server;
listen [::]:80 default_server;
access_log /var/log/nginx/default-access.log timed;
error_log /var/log/nginx/default-error.log;...
Save and close the file to exit.
To enable the new configuration, restart Nginx.
sudo systemctl restart nginx
Now that everything is set up, let's check if it works.
We can test the new configuration curl by calling some requests to Nginx, just like we did in step 2. This time we will use the sample file created in step 1:
curl -i http://localhost/empty.test
curl -i http://localhost/1mb.test
curl -i http://localhost/10mb.test
curl -i http://localhost/100mb.test
You will notice that each subsequent command will take longer to execute, because the files become larger so it takes more time to transfer them.
Let's display the access log after executing these requests.
sudo tail /var/log/nginx/default-access.log
The log will now contain more lines, but the last four lines will correspond to the test request you just executed.
127.0.0.1- - [04 /Jul/2016:14:57:02-0400]"GET /empty.test HTTP/1.1"2000"-""curl/7.47.0"0.000127.0.0.1--[04/Jul/2016:14:57:51-0400]"GET /1mb.test HTTP/1.1"2001048576"-""curl/7.47.0"0.000127.0.0.1--[04/Jul/2016:14:57:57-0400]"GET /10mb.test HTTP/1.1"20010485760"-""curl/7.47.0"1.901127.0.0.1--[04/Jul/2016:14:58:52-0400]"GET /100mb.test HTTP/1.1"200104857600"-""curl/7.47.0"49.232
You will see that the path is different each time, the correct file name is displayed, and the size of each request increases. The important part is the number highlighted at the end, which is the request processing time (in milliseconds) that we just configured in a custom log format. As you might expect, the larger the file, the longer the transfer will take.
If this is the case, you have successfully configured a custom log format in Nginx!
Although it is not particularly useful to see that larger files require longer transmission time, when Nginx is used to serve dynamic websites, request processing time can be very useful. It can be used to track down the bottleneck of the website and easily find requests that take longer than they should.
$request_time is just one of many system variables exposed by Nginx and can be used in custom logging configuration. Others include, for example, the value of the response header sent to the client in response. Adding other variables to the log format is as simple as putting them into the log format string, just like we add $request_time. It is a powerful tool that you can use when configuring logging for your website.
The Nginx log module documentation describes the list of variables that can be used with the Nginx log format.
For more Ubuntu tutorials, please go to [Tencent Cloud + Community] (https://cloud.tencent.com/developer?from=10680) to learn more.
Reference: "How To Add the log Module to Nginx on Ubuntu 16.04
Recommended Posts