Log management and intelligence

Log management (LM) comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc.). LM covers log collection, centralized aggregation, long-term retention, log analysis (in real-time and in bulk after storage) as well as log search and reporting.

Log management is driven by reasons of security, system and network operations (such as system or network administration) and regulatory compliance.

Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection logs).

Users and potential users of LM can build their own log management and intelligence tools, assemble the functionality from various open-source components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it .

Log Management Key Features and Technology

The deployment of a Log Management architecture generally involves the following steps:

• Step 1: Define the requirement and goals. Needs can be security log analysis, application problem analysis, or reporting for the purposes of regulatory compliance.

• Step 2: Define the logging framework, log types, and system specification where logs are generated.

• Step 3: Determine what you’re going to use log management for according to your goals. Are you going to collect the logs? Maybe you need to analyze or even report and monitor the logs on remote machine. If you plan on collecting log data, how long will it need to be archived? Is it going to be encrypted? Regulatory compliance may provide specification for such needs.

• Step 4: What information and intelligence are you planning to extract out of your log? End user patterns reports, application problems and more can be taken.

• Step 5: Evaluate technology and vendors solution to select the best fit to your needs. You may also select to build a log management solution internally, leveraging open source solutions. Add a reporting and analysis layer later on for intelligence.

Deployment life-cycle

One view^{[citation needed]} of assessing the maturity of an organization in terms of the deployment of log-management tools might use successive categories such as:

• Level 1: in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security-perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.

• Level 2: with increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security-perimeter.

• Level 3: at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the enterprise — especially of those information-assets whose availability organizations regard as vital.

• Level 4: organizations integrate the logs of various business-applications into an enterprise log manager for better value proposition.

• Level 5: organizations merge the physical-access monitoring and the logical-access monitoring into a single view.

See also

• Log management knowledge base
• Audit trail
• Server log
• Log analysis
• Web log analysis software
• Web counter
• Data logging
• Common Log Format
• Syslog
• Common Base Event
• DARPA PRODIGAL and Anomaly Detection at Multiple Scales (ADAMS) projects.

References

• Chris MacKinnon: "LMI In The Enterprise". Processor November 18, 2005, Vol.27 Issue 46, page 33. Online at http://www.processor.com/editorial/article.asp?article=articles%2Fp2746%2F09p46%2F09p46.asp, retrieved 2007-09-10

• MITRE: Common Event Expression (CEE) Proposed Log Standard. Online at http://cee.mitre.org , retrieved 2010-03-03

• NIST 800-92: Guide to Security Log Management. Online at http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf , retrieved 2010-03-03

External links

• InfoWorld review and comparison of commercial Log Management products