How to Centralize Server Logs with Rsyslog?

January 26, 2016 / General Discussion
  1.  Overview

    Centralizing logs from multiple servers on one server can be of great interest to the security level within a system of information. Indeed, it is easier for log analysis tools to compare and read. Scan files being on a single server rather than doing it remotely or through remote agents.

    Also in case of a server crash, you will be able to recover the mistakes. Actions on your server before it does crash, facilitating the restoration activity of this and future security.

    On most modern Linux systems the log management tool is by default Rsyslog. It is one that we will use in this tutorial.

    Here, we will work on two machines running Debian 7, one acting as a server. The other as a “client” that will send its logs to the server.

    In both cases, the configuration file to edit is always /etc/rsyslog.conf. One can nevertheless put in /etc/rsylog.d/ and make a “include” in the main configuration file to import.

  2. Server Configuration

    For the server, simply set the server to accept logs from outside by opening the appropriate port, for the time we will open port (UDP 514) for simplicity. For this, will uncomment the following two lines:

    $ModLoad imudp
    $UDPServerRun 514

    We can then restart the rsyslog Service:

    service rsyslog restart

    To verify that our server is listening, you can perform the following command:

    netstat –npul

    We then see our open UDP port 514…

  3. Client Configuration

    At the client level, we will go to the same file and then redirect all logs on the server. Add the following line to the end of the file:

    *.* @IP_SERVER:514

    Note: The “@” must appear in the line
    It will then restart rsyslog our Service:

    service rsyslog restart

    Then we can generate logs; for example, restart any service to see if they are on the server rsyslog. By default, as we did, customer logs will be in the same file as the server logs, for example in the /var/log/syslog server. You just see the name of the host that generated the log line is not that of the server, but the client’s.

  4. TCP / UDP

    Previously, we configured our client and our server to exchange using the transport protocol (layer 4) UDP. Briefly, UDP allows a single exchange to transmit a packet is a protocol that does not try to synchronize, or track package status or exchanges between two assets.

    Instead, TCP will try to find each packet if it has been received by the correspondent, which generates additional packets over UDP. TCP is able to check whether an information exchange or not been received and retransmitted if necessary that does not UDP.

    All this is to underline that in our case, TCP exchanges have a disadvantage to be taken into account in relation to the UDP. Indeed, the TCP will generate more trade (and therefore resource consumption) over UDP but will ensure that the information received is in full.

    So there is a choice that depends on your available resources and the number of customers and information managed by your server. To trade in TCP mode, you must refer to these lines in the server configuration file, note that both methods can coexist if we put them on different ports, for example:

    Otherwise, it will review the top two lines. On the client side, add this line in the configuration for that trade is TCP:

    *.* @@IP_Server:1514

    Note: The @@ must appear in the command line, the fact that there have two states that exchange will TCP.

    One can also leave two lines to indicate that the logs will be sent once TCP and once UDP. This is of little interest but for understanding that we can also send our logs to several different IP servers if you change one of the two lines. Then we restart the server and the client if needed:

    service rsyslog restart
  5. Redirect or copy according to the priority or services

    When talking about log management, especially in Linux, the services are categories in which the logs will store to better archive and sort. These services include, for example:

    auth: used for events relating to the security or authentication through access applications (SSH type)
    authpriv: Used for messages related to access control
    daemon: Used by different systems and application process
    kern: used for messages about the kernel
    Email: Used for Event services email
    user: default services when none is specified
    local7: Indicates the boot messages
    *: Refers to all services, for simplicity this is what we specified when we set first redirection rule slightly above logs
    none: Indicates no services

    In addition to these services, we find services for each severity level (called Priority) that goes from worst to mere information:

    Emerg: Emergency System unusable
    Alert: Immediate action required
    Crit: Critical System Error
    Err: Operation error
    Warning: Warning
    Notice: normal Reportable Events
    Info: For information
    Debug: debug Message

    One then finds many ways to specify the logs that interest us and therefore those we will redirect. For example if you try to redirect to xxx.xxx.xx.xxx our server logs only display critical messages and higher for messages on UDP port 514, add the following line:

    mail.err @xxx.xxx.xx.xxx:514

    It can also redirect all mail logs:

    mail.* @xxx.xxx.xx.xxx:514

    One can also enter in a line several types of facilities and priority; there is no example in the default configuration file lines like these:

    *.=debug;\
           auth,authpriv.none;\
           news.none;mail.none     -/var/log/debug
    *.=info;*.=notice;*.=warn;\
           auth,authpriv.none;\
           cron,daemon.none;\
           mail,news.none          -/var/log/messages
    

    Here we see that all the debug priorities are redirected to the file /var/log/debug and that all priorities info notice and warn will be in /var/log/messages.

    For these filters are redirected to the log server, just specify the server IP and the port done above in place of the file name.

  6. Redirect logs to a folder/file by the host

    It is also to facilitate prioritization and archive our logs when you have a large number of customers Rsyslog use a tree with a folder/file by host rather than putting all logs in the same file as the server logs. We will do this using a template that we will after the block ‘RULES’ in the server configuration file:

    $template syslog,"/var/log/clients/%fromhost%/syslog.log"

    We will then apply this template to all incoming logs:

    *.* ?syslog

    It then suffices to restart rsyslog our services and generate logs from clients. Then we will end up with a file “/ var / log / clients /” containing a folder by IP / customer name and a file containing respectively “syslog.log” with logs of each respective client, which simplifies the search for information in the logs for a specific customer.