| LogStatistics |
Monitoring an application frequently involves monitoring it's log file(s). Log files may contain hundreds or thousands of events per minute. Parsing the entire log file can be a very cpu intensive task making near-real-time reporting or monitoring difficult.
logstatsd was designed to solve this problem. logstatsd runs as a daemon on the server where the log file resides, parsing new entries as they enter the log, and storing statistics. The daemon can be signaled to export current xml reports or to update an RRD.
In addition to collecting near-real-time data for monitoring, logstatsd also makes an excellent tool for generating offline reports from single log file, multiple log files, or even multiple log files scattered across multiple servers.
Additionally, statistics may be collected for multi-dimensional data. For example, if you specify collection of data about transaction name grouped with status, you can generate a report about the numbers of each status for each transaction. If you collect summary statistics about status grouped with time, then you might see statistics about the successful and unsuccessful transactions per minute. You can even group three or more fields, e.g. grouping status, transaction, and minute will show the statuses for each transaction broken down by minute. See the "Example XML output" section in the included documentation.
Thresholds may be defined to categorize response times. For example, by specifying two thresholds of 1 second and 5 seconds, you would gather data on the number of transactions that were less than 1 second, the number between 1 and 5 seconds, and the number over 5 seconds. See the "THRESHOLDS" section in the POD for more details.
In order to monitor useful data, you must start with a log file that contains the information you want to track. For web applications, apache access can provide data such as end user information, status codes, transaction names, date/time, and durations. In some cases it may make sense to create a custom log for your application that contain entries for other events you want to monitor, e.g. database transaction times, back end system response times, etc.