TeleMate Log Collector

The TeleMate Log Collector was written for customers using TeleMate.Net's log analyzing and reporting products, like NetAuditor and TeleMate. It consists of a user interface for configuring the supported collection methods and a service that collects the log records. Because it runs as a service, you do not need to stay logged in for the collector to do its job. The service is capable of collecting from several syslog, telnet, RSP, and Checkpoint LEA devices simultaneously. Each connection gets its own thread, which is given a high priority to minimize the risk of data loss when the collecting workstation is under a heavy load (i.e. when the logs are being processed). If multiple devices send log records to the same port on the collecting workstation, the collector will automatically split them up into subdirectories by the device's IP address.

Log files are switched daily, and each daily log file is given a name like "syslog-YYYYMMDD.log", "telnet-YYYYMMDD.log", "ftp-YYYYMMDD.log", "rsp-YYYYMMDD.log", or "lea-YYYYMMDD.log". If the collection folder you select is on an NTFS partition, the collector will automatically change it to a compressed folder to conserve disk space. The collector also provides an optional cleanup feature that can delete old log files automatically when they reach a certain age, when the partition gets too low on disk space, or when the collection folder gets too large. We have also made an effort to ensure that it is safe to process log files directly from the collection folder at any time of day. The collector does not keep locks on any of the files, and if a processing application locks the current day's log file, the collector will start appending records to "syslog.tmp", "telnet.tmp", "rsp.tmp", or "lea.tmp". Once the lock is cleared on the current day's log file, the contents of the temporary file are appended to it, and normal collection resumes.

Note: If you have a personal firewall, anti-virus, or other similar program running that may block unknown network connections, you should make sure that it does not block the connections made to/from VNACollectorService.exe, which is configured as a service called VNACollectorService. The service generates log files in a folder called "C:\TeleMate Log Collector\LogFiles". If the service can't open a port it's supposed to listen to or fails to connect to a device, it will log any errors or warnings in there.

Syslog Collection

Syslog is an open UDP protocol used by many network devices, like the Pix and Netscreen firewalls, for logging events. It is a simple protocol where each event is sent as a plain text message to a specific IP/port. Each network device must be configured to send the syslog events to the collecting workstation, and the collecting workstation must be configured to listen on the correct port. The default syslog port is 514, and it is ok to have several devices sending data to the same port. UDP protocols do not guarantee safe packet delivery. If the collecting workstation is not actively listening when a packet is sent, or if the workstation or network is too slow to keep up, packets will disappear without a trace. To avoid packets from being lost or interecpted, you should not use this protocol across the Internet or a VPN/WAN. If necessary, you should intall the syslog collector at each remote location and use a more reliable and/or secure protocol to transmit the logs to a central location.

Although there are several free syslog collectors available for you to use to collect this data, some syslog collectors have some problems or limitations that might make them less desirable to use. For instance, Cisco's syslog collector only listens on one port, mixes up records sent by multiple devices, uses a poor file naming convention (i.e. monday.log), loses packets when the machine is under a heavy load, and stops collecting when the target drive starts to get low on disk space. Some collectors don't put a date-time stamp on the record, and some use a date-time format NetAuditor doesn't recognize. The TeleMate Log Collector attempts to address all of these problems and make collection painless for the customer. It is very easy to configure the primary syslog port through the user interface. The default settings are ideal in most cases, and all you have to do is press "Start".

Note: Although we're not sure if anyone will ever want to use multiple ports for syslog, you can configure additional ports by manually editing "C:\TeleMate Log Collector\syslog.conf". One reason you may want to do this is if you need to store data from different devices on different disk partitions. If you need help setting this up, contact TeleMate.Net technical support at support@telemate.net or +1.678.589.7120.

Telnet Collection

Telnet is an open TCP protocol used by many network devices. It is a simple protocol where data is sent as plain text over a connected TCP socket. The telnet collector is essentially a TCP version of our syslog collector. It does not send any data to the devices it collects from. It simply takes the data sent from each device and writes it to disk.

Unlike syslog, which doesn't need an established connection to send/receive data, telnet requires TCP connections to be maintained. For devices that can be configured to initiate TCP connections to a collector, you may choose a port for the collector to listen on. For devices that listen for a collector to initiate a connection, you may create a list of addresses to connect to. When the telnet collector is acting as a server (listening on a port), it is up to the device to re-establish the connection if it goes down (i.e. when you reboot the machine the collector is running on). When the telnet collector is acting as a client, it will automatically try to re-establish any connections that go down.

Note: The telnet collector also supports a relay feature. When collecting in relay mode, data coming from all devices will be merged into one stream. Every 10 seconds, it will attempt to relay all collected data to another telnet collector (the other collector must be listening on the same port). If your device does not buffer data when the connection goes down, and you need to collect across an unreliable network like the Internet, you can use this feature to turn a local PC into a buffer box.

FTP Collection

FTP is an open TCP protocol used by many network devices. It is a simple protocol for managing files across the network. It does more than Syslog or Telnet in that it allows you to find files remotely, move them around, delete them, etc. It requires a login, but as everything is sent across the network in plain text, it is not secure in any way.

To collect data from an FTP server, you must set up each device separately so that the TeleMate Log Collector will know how to connect to the server, which directory to get log files from, and which files in that directory to get. You may set up as many FTP devices as you want. At a global level, you may choose how often the collector service checks the FTP servers for new log files. You may also choose whether it should leave the log files on the FTP server or delete them as it's downloading them. (Unless you need to leave those files there for some other application, we recommend that you let the collector remove them.)

Note: When you choose to leave the files on the server, the collector does not use "ftp-YYYYMMDD.log" locally. It uses the same filenames it finds on the server so it will know which files it has already downloaded. Because of this, the collector has no way to know when it is safe to delete the files it has downloaded. You must clean them up manually. You also must clean them up manually on the server, or the collector will simply download them again after you delete them locally.

RSP Collection

RSP, or Reliable Session Protocol, is a proprietary TCP protocol developed by Avaya for many of their voice/network devices. RSP is a TCP protocol that adds its own layer of hand-shaking messages to ensure that every log record is accounted for reliably. The RSP device must be configured to connect to a specific port on the collecting workstation. The default port is 9000. If the connection is broken, the device starts buffering records while it attempts to reconnect to the collecting workstation. As long as the connection isn't down long enough for the device's buffer to fill up, collection will resume where it left off without losing any data when the connection is re-established. RSP should never lose packets like syslog, but it still transmits data in plain text. If you are concerned about the data being intercepted, you should not collect records across the Internet.

Avaya provides a tool called RDTT, or the Reliable Data Transport Tool, and some of our customers have tried to use this as a collection utility. However, the RDTT documentation clearly states that RDTT was designed to be a tool for developers to test their RSP implementations. It contains a number of bugs that make it unstable, and it should not be used for anything more than testing RSP connections. Some CSD's, or Call Storage Devices, support the RSP protocol, along with a number of other software products. If you decide to use the TeleMate RSP collector, it is very easy to configure from the user interface. The default settings are ideal in most cases, and all you have to do is press "Start".

Note: Although we're not sure if anyone will ever want to use multiple ports for RSP, you can configure additional ports by manually editing "C:\TeleMate Log Collector\rsp.conf". One reason you may want to do this is if you need to store data from different devices on different disk partitions. If you need help setting this up, contact TeleMate.Net technical support at support@telemate.net or +1.678.589.7120.

LEA Collection

LEA, or Log Export API, is a proprietary protocol developed by Checkpoint for their NG and NGX firewalls, and it can only be used through Checkpoint's OPSEC API. It is much more complicated than the protocols mentioned above, and as a result it is more difficult to configure. Instead of listening for connections from other devices, the collector must initiate the connection to each firewall. This means that each device must be configured separately, both at the firewall and at the collecting workstation. Clear connections are supported along with many different types of secure connections. LEA collectors can connect in offline mode, which collects old log records and then stops, or online mode, which stays connected and connects new records as they are generated in real-time.

The TeleMate LEA collector collects using the online mode, but it keeps track of the last record it collected so it can resume where it left off if the connection is ever closed. As long as you don't delete any logs from the Checkpoint firewall, you can stop the collector for months or even years. The next time it comes online, it will attempt to collect all the data it missed.

Note: There seems to be a bug in some versions of the OPSEC libraries. As a result, if the connection with the firewall is lost, the LEA collector may not notice it, and you may have to restart it to get it to start collecting again. Because the collector resumes where it left off, no data should be lost, but it is annoying. Because the problem is in Checkpoint's closed libraries, I must wait for them to provide a fix for this.

Setting up a clear connection

To set up a clear connection, you must first log into the firewall and find the "fw1/conf/fwopsec.conf" file. You must open this file in a text editor, make a change to it, save it, and restart the firewall. (How you do this depends on the platform it is installed on.) The fwopsec.conf file must contain this line for the firewall to listen for clear LEA connections:

lea_server port 18184

You may change the port number, but otherwise the line must be identical to the one above. Next you must change the firewall policy to allow the collecting workstation to connect to the firewall using the FW1-lea service. (If you changed the port number, you will need to create a new service for it and use that instead of FW1-lea.) When testing the connection, this rule should be at the top of the list to make sure no other rules override it. Once you have applied the policy changes, you can launch the TeleMate Log Collector interface, select Checkpoint LEA, and add a new LEA device. Once you have saved the device, wait a few minutes and traffic should start to appear for it. If no data appears, double-check the IP address and port, and then check the logs on the firewall itself to make sure new log records are being generated, and to make sure the firewall didn't reject the LEA connection for any reason.

Setting up a secure connection

We have tested the TeleMate Log Collector with an authenticated/encrypted LEA connection to make sure that it works. However, it is a very complicated setup and we do not have a licensed Checkpoint firewall here to train our support reps on, so we can not officially support it. We will try to point you in the right direction toward getting it set up, but after that you will have to rely on Checkpoint support.

Now that we've made that clear, if you still feel strongly about setting up an authenticated/encrypted LEA connection, find and open "C:\TeleMate Log Collector\opsec.conf" in a text editor. This file contains the configuration we set up in our development lab, along with step-by-step instructions for how we set it up. Keep in mind that each firewall must be configured separately. We strongly recommend that you set up a clear connection for each firewall as a test before you attempt a secure connection.