How has the increase in regulations affected the need for log management? How does log management contribute to overall network security?
JON OLTSIK: The increase in regulations has driven a more formal approach to log management. In the past, logs were used in a tactical manner by different constituencies such as security professionals, IT operations staff, and IT auditors. Regulations create the need for a more formal process for log collection, analysis, and retention, making the entire log management process more strategic across the organization. ESG now sees large organizations building enterprise-wide log management architecture. This effort was driven by regulatory compliance but many firms are now using log management for additional IT and business benefit.
As for network security, log data provides a picture of normal and anomalous network behavior. When a network is under attack, log data can be used to detect an event and assess the extent of the damages. This is critical for rapid emergency response and remediation. Without log data, security administrators are essentially “flying blind.”
REED HENRY: The change has been significant. Regulators are driven by actual breaches and documented unfair practices. Both have driven a line into the sand for log management regulations. Regulators will apply pressure to log an appropriate level of activity across far more devices and types than ever before and at a faster rate for a longer period of time, especially at the application/user layer. It is imperative to know who did what, where and when even at a high level. Log management will both help identify and contain attacks on regulated data and also allow investigators to perform a full post-mortem and report on new vulnerabilities and threats. For example, incident investigators increasingly point to areas in log management that need to be enhanced in order to assist future investigations and reduce risks. These changes are often communicated to organizations through updated legislation and outside governance, if not adopted directly through self-regulation.
JAMES FOSTER:
The increase in regulations that include elements regarding business processes and technology has lead to increase in log management. Robust log management should be viewed as an affective intelligence tool to gather transactional security and audit events from business critical machines. While most organizations will gather, analyze, report and store logs as that is typically the compliance and business requirement; the best organizations will use this information to enhance complementary security, network, and administration systems.
Most logs require a lot of IT administrative attention to ensure compliance; how can organizations automate log management processes without compromising network security?
REED HENRY: Automation must focus on the log collection and analysis processes to ensure network security with minimal administrative burden, which is a necessity to avoid human error. These are the exact areas where nearly a third of organizations struggle with log management according to the 2008 SANS Log Management survey. The log collection process must not only automatically aggregate raw logs across distributed networks but also automatically normalize the array of log events into a common language across all devices and vendors including custom sources that can be easily analyzed and reported on. The automation of the log analysis process begins with the review of the normalized event stream, alerting when compliance violations occur, and allowing for interactive drill down from alerts into the source events behind violations. The audit portion of the analysis process must deliver interactive reports and dashboards that allow interactive drill down into underlying events.
JAMES FOSTER: An effective log management practice should contain elements that enable a more robust network security practice while adhering to compliance standards. Where appropriate, compliance standards should be viewed as the new “minimum requirements for security organizations.” Proactive and optimized organizations should be looking to exceed compliance standards through integration and automation. Big wins include creating an automated NOC alert system, ticket and help desk system integration, as well as intrusion prevention and perimeter security integration.
JON OLTSIK: Automating the log management process really requires two elements: 1) Organizational policies and processes, and 2) Log management tools. First, compliance officers and IT administrators must collaborate on which events need to be logged on a system-by-system basis in order to monitor the status of various controls. Once this list is detailed, IT administrators must configure system logs accordingly to capture this data.
Once all systems are properly configured, it is up to IT to build a log management architecture that can collect, process, and store all of the log data. To enhance security, log data can be encrypted to maintain confidentiality while administrator access can be limited to a few key personnel. All access to log data can also be logged to protect against an insider attack. In this way, proper log management policies and procedures couples with a well designed log management architecture can actually improve overall security.
How can auditors use network logs in their investigations?
JAMES FOSTER: Network logs, if handled properly, can be significant assets during an investigation. If the proper information is included, it could encompass source, destination, timestamp information, transaction, and payload information. However, as we have seen, most organizations do not have their critical systems configured to properly log or audit user activity; nor is that data being centrally managed, monitored, or stored. The continued focus and proliferation of next generation log management appliances and solutions will help with this issue.
REED HENRY: Logs are also very helpful in detecting security and compliance violations before they evolve into a breach or incident. However, when a breach occurs, organizations must be able to determine the underlying cause(s). Logs are used in this vein to replay who did what, where and when. This is typically a slow and manual investigation effort when done without effective network and system log management tools. The ease and speed of an investigation is greatly affected by the ability to quickly identify whether 100 or 100,000 identities were breached, for example. Similarly, when alarms sound long after an event, such as a breach discovered several months late, investigators need to be able to isolate and retrieve all relevant data in a timely manner. Effective solutions in log management can provide the details, historical context and even the analysis required to make investigations much more accurate and efficient.
JON OLTSIK: Network logs provide forensic data for auditors that help them assess the existence and effectiveness of IT controls. For example, the Gramm-Leach-Bliley Act places strict controls on who has access to the financial data of publicly traded companies. It is fairly easy to write an access control policy but how can anyone really know if proper access controls have been implemented or whether they are really enforcing the corporate policy? Log data provides data that helps answer these questions. Sound log management can capture the network logs, provide detailed audit reports, and alert IT auditors to problems in a timely manner.
While some enterprises have larger budgets for high-end security solutions, small and medium-sized businesses are increasingly subject to compliance regulations as well. Is there an approach to log management and security that suits both sizes of business?
JAMES FOSTER: The key to decreasing the overall cost to any log management solution is automation. This is especially important for SMBs who are extremely resource constraint. Emphasis should be put on rule and trigger fine tuning with respect to alerts. As with any real-time alerting system, too many alerts and they will be ignored, too few and you wonder if it’s doing the right job.
JON OLTSIK: Yes. Log management doesn’t have to be complex or expensive. As log management becomes more strategic, many IT vendors are providing “turnkey” appliances for log management, security, and compliance auditing. For smaller firms, these systems are affordable and provide “good enough” functionality. ESG has also seen an increase in log management and security services offerings. Smaller firms with limited security knowledge or resources may be best served by outsourcing log management and its associated risks and costs.
Larger firms with more complex needs should think in terms of a log management architecture as a foundation for security and compliance management. A log management architecture can be used to collect, process, secure, and store all of the log data across the enterprise. Analysis engines for security, compliance, and IT operations can then tap into the log management architecture through APIs to gather the data needed for specific analysis tasks.
REED HENRY: Although large companies report the largest breaches, most breaches occur in small to medium sized environments. Regardless of the size of an organization breaches and violations manifest from the same gaps and have the same risk of fines, bad press, class action lawsuits and customer attrition. The Payment Card Industry clearly says if a business is breached, no matter what its size it will be re-classified into the most stringent reporting requirements.
Log management solutions initially targeted large enterprises because they were the first targets of compliance enforcement. Lessons learned these deployments have been now turned into appliance-based options for both the SMB and enterprise market. The collection and analytical compliance requirements are essentially the same across business size, but vary in scale. Whether businesses are small or big they need to select a solution that automates log collection and analysis while removing the complexity of installation or administration regardless of scale.
What are some best practices companies can consider to maintain compliance without interfering with larger business objectives?
JAMES FOSTER: First and foremost, create a collaborative environment within the compliance, security, and system administration teams to ensure that all requirements are understood and communicated. If managed properly, an effective log management program would enable compliance, more robust security, and provide metrics that would support a quantifiable return on investment model without “interfering” with business objectives.
JON OLTSIK: Compliance should not be viewed as a burden but rather as a catalyst to formalize processes and controls, and improve overall security. This starts with sound policies mutually agreed to by business and IT managers. Policies should lead to the creation of IT controls that meet regulatory compliance objectives and internal governance policies. These controls must then be constantly monitored to ensure they are efficient and effective.
Organizations that embrace compliance as a part of overall corporate governance can actually use these initiatives to improve business objectives. With strong controls in place, IT should be able to react more quickly to new business initiatives while maintaining strong security.
REED HENRY: Businesses often take two approaches to compliance. The first is to adapt to governance as an evolutionary step by aligning business processes and IT systems including automation to both meet business objectives and protect information and systems – the intent of compliance. Companies that choose this approach achieve compliance with the least overall cost while facilitating their business objectives. The second approach is to approach compliance as an annual obligation, and allocate one-time resources for a quick fix. Companies taking this second approach will struggle to keep up with the continually evolving regulatory compliance environment.
Regardless of past approaches taken, businesses should consider automating all manual compliance oversight processes with log management automation solutions. This approach will ensure violations are detected, audits are automated, and IT staffers are freed up to work on projects that facilitate larger business objectives. This approach will provide a flexible foundation for evolving regulations and compliance initiatives.
Reed Henry is the senior vice president of marketing at ArcSight, Jon Oltsik is a senior analyst at Enterprise Strategy Group, and James Foster is CEO and founder of Ciphent.