Log management and intelligence
Log management (LM) comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc.). LM covers log collection, centralized aggregation, long-term retention, log analysis (in real-time and in bulk after storage) as well as log search and reporting.
Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection logs)[examples needed].
Users and potential users of LM can build their own log management and intelligence tools, assemble the functionality from various open-source components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it.
Lately, more and more the suggestion is made to change the definition of logging. This change would keep matters both more pure and more easily maintainable. Logging would then be defined as all instantly discardable data on the technical process of an application or website, as it represents and processes data and user input. Auditing, then, would be used to denote data that is not immediately discardable. In other words: data that is assembled in the auditing process, is stored persistently, is protected by authorization schemes and is, always, connected to some end-user functional requirement. Logging is technical information used for the maintenance process of applications or websites - either to define whether a reported bug is actually a bug, or to help analyze, reproduce and solve a bug, and even to help test new features in a development stage.
- Level 1: in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security-perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.
- Level 2: with increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security-perimeter.
- Level 3: at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the enterprise — especially of those information-assets whose availability organizations regard as vital.
- Level 4: organizations integrate the logs of various business-applications into an enterprise log manager for better value proposition.
- Level 5: organizations merge the physical-access monitoring and the logical-access monitoring into a single view.
- Audit trail
- Common Base Event
- Common Log Format
- DARPA PRODIGAL and Anomaly Detection at Multiple Scales (ADAMS) projects.
- Data logging
- Log analysis
- Log management knowledge base
- Server log
- Web counter
- Web log analysis software
- Chris MacKinnon: "LMI In The Enterprise". Processor November 18, 2005, Vol.27 Issue 46, page 33. Online at http://www.processor.com/editorial/article.asp?article=articles%2Fp2746%2F09p46%2F09p46.asp, retrieved 2007-09-10
- MITRE: Common Event Expression (CEE) Proposed Log Standard. Online at http://cee.mitre.org, retrieved 2010-03-03
- NIST 800-92: Guide to Security Log Management. Online at http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf, retrieved 2010-03-03