Dr Anton Chuvakin, Chief Logging Evangelist at LogLogic, talks iQ through the difference between the wood and the trees...

1. What?

Logs. System logs, audit trails, network logs, IDS (Intrusion Detection System) and IPS (Intrusion Prevention System) alerts. These – and basically all the other transactional data our information systems are spewing out at us at an ever-increasing rate – can all be classified as logs.

Log files can in turn be classified by the various sources producing them. The kind of system log files produced by operating systems like UNIX, Linux, and Windows are, for instance, different from the network device logs produced by networking gear (routers, switches, and so on) from the likes of Cisco, Nortel, and Lucent.

Security appliance logs produced by firewalls, IDSs, IPSs, or messaging security appliances are different again. In fact, security devices display a wide diversity in what they log and the format in which they do it.

2. So?

Historically little real attention has been paid to basic system logs. They are, however, an important and even crucial source of information about the activity taking place across our systems at any given time, and are becoming particularly vital from a governance and compliance perspective for example.

3. And?

This is being complicated by the fact that more and more log sources are being added to today’s already nebulous mix of logs coming from server, application, firewall, and security appliances. This includes the increasing amounts of log data produced by mobile devices and internet-connected appliances. The average large company will already typically produce many gigabytes worth of log data each week.

4. So now what?

System logs are the source of three vital types of insight:

1. Security – logs are useful for detecting attacks; they are also indispensable for incident investigations and forensics.

2. Regulatory compliance – many regulations in the UK, the US, and other countries mandate the collection and review of logs; more and more auditors now demand proof of log collection and review.

3.  IT operational efficiency – systems administrators can reach for logs when troubleshooting the system problems; logs are also used to measure network and application performance.

Essentially then, at a high level, logs equal accountability – from answerability and enforcement to responsibility, blameworthiness, and liability. There are many other corporate accountability mechanisms of course, but logs pervade all of IT, and if your IT isn’t accountable, your business isn’t either.

The argument is therefore, that if you’re not serious about logs, you’re not serious about accountability. Not a message any organisation wants to be sending; particularly in the current climate.

5. So what do we do about it?

While your IT infrastructure will already be producing heaps of logs by default, some parts will probably need to be configured to produce even larger volumes if they’re to be useful for security, compliance, and operations. While some technologies, such as databases, generally have almost no logging configured by default, so will need their logging settings turned on or enabled.

Overall, from the point of view of issues like regulatory compliance, data theft investigation, and operational system troubleshooting, it’s better to err on the side of more rather than less logs. Detailed file access logs will, for example, help to track which users are accessing sensitive data (security), review regulated document access (compliance) and monitor server performance (operational research).

6. Who needs logs?

A diverse range people will typically utilise their organisation’s log-based information. IT administrators, IT managers, security analysts, incident responders, IT directors and even CIOs and compliance officers are commonly be among those either looking at logs directly, or at reports based on them; as a recent poll demonstrates.



While senior officers and managers don’t generally want to look at raw logs, and instead favour log-based reports, analysts and administrators tend to want both original raw logs (usually accessible via a search) as well as reports (customised and provided by the system).

7. But how to make that happen?

A good way to take control of your business’s logs is to deploy a log management system, though larger, more advanced organisations often favour a log data warehouse approach. Such systems centralise the collection, storage, and access of log data across the enterprise, freeing it from the more device-by-device based approach to log management, which tends to be inefficient and heavily reliant on manual tasks.

Providing a central repository for all log data, the warehouse enables users to quickly and easily query and report on this data, and to efficiently manage the massive amounts of log data generated through network devices, security gear, operating systems, network servers, databases, and more.

It also goes beyond simple storage, allowing users to discover and act on relationships between data from these heterogeneous data sources.

8. But what’s wrong with what they have now?

Today many organisations employ ad hoc approaches to logs: whereby the security team “owns” network IDS logs, the network team owns the firewall and router logs (as well as all SNMP traps) and the system administrator owns the logs produced by servers and desktops. 
That’s not only counterproductive, inefficient, and wasteful; it’s also dangerous.

9. Why?

Such approaches – divided by both technical and political chasms – can break down painfully. In the case of an incident response, for instance.

Here, instead of being able to run a single query across all logs and all log sources (or a defined subset of logs or log sources), the user often ends up having to beg for the log data they need; having to connect, wait, and download logs; having to look in many places at once; and generally wasting huge amounts of time and energy.

All this when they could simply connect to a log management system and run a few reports, drilldowns, and searches.

It’s also an issue from a compliance perspective. Most regulations and mandates don't single out logs by the type of the log source, but apply to all logs across the organisation. Using one system to verify compliance status therefore makes much more sense than having to dig around in several different systems.