Honours Year Project Report

Assessing the Performance Impact of the

Linux Kernel Audit Trail

By

Qiu Jiang

Department of Computer Science

School of Computing

National University of Singapore

2004/05
 Honours Year Project Report

Assessing the Performance Impact of the

Linux Kernel Audit Trail

By

Qiu Jiang

Department of Computer Science

School of Computing

National University of Singapore

2004/05

Project No: H089020

Advisor: Prof. Sandeep Kumar
Deliverables:
Report: 1 Volume
 Abstract

Security breach into information systems is almost always caused by unpatched or unknown
vulnerabilities. To counter this threat, system administrators often rely on effective auditing to
discover such security violations. The benefits of auditing include accountability, reconstruction
of higher-level events, intrusion detection, and problem analysis. Furthermore, the mere pres-
ence of auditing is a powerful deterrence to potential attackers. Despite these well-understood
benefits, system administrators are reluctant to turn on auditing on their systems because of
the fear of performance overhead. In this thesis, we design, implement and benchmark a low
overhead auditing subsystem for the Linux 2.4 kernel.

Subject Descriptors:
C5 Computer System Implementation
D.4.6 Security and Protection

Keywords:
audit, kernel, operating system, linux

Implementation Software and Hardware:

Linux Kernel 2.4
 Acknowledgement

I would like to express my utmost gratitude to my project supervisor, Assistant Professor

Sandeep Kumar, who provided me with inspiration and guidance that set me on the right course

in this study.

Also, my hat goes off to Jeremy Banford and Nenad Ocelic for their attempt to contribute

a high quality kernel-level auditing tool to the open source community.

List of Figures

3.1 Audit records are generated at system calls . . . . . . . . . . . . . . . . . . . . . 12

3.2 Security sensitive system calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Auditor device with double buffering . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.1 Integer, float and double opreations . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.2 Memory and file system results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.3 Process related results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.4 Communication results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4
List of Tables

3.1 System call group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Threat level classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.1 Linux kernel compilation macrobenchmarks, time in seconds . . . . . . . . . . . . 27

5
Table of Contents

Title 1

Abstract 2

Acknowledgement 3

List of Figures 4

List of Tables 5

1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Report Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Related Work 4
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Audit Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Built-in Logging Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.2 Custom Audit Trails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Auditing Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 System Design and Implementation 9

3.1 Naive Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Improved Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.1 Kernel Patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.2 Auditor Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2.3 Audit Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4 Evaluation 19
4.1 Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.2 Microbenmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2.1 Arithmetic Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2.2 Memory and File System Operations . . . . . . . . . . . . . . . . . . . . . 22
4.2.3 Process Related Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.2.4 Communication Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3 Macrobenchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.4 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.4.1 Ideal Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.4.2 Worse Case Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

6
 4.4.3 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 Conclusions and Future Work 31

5.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1.1 Tamper Proof Audit Trail . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1.2 LSM Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

References 33

7
Chapter 1

Introduction

Information security is becoming an ever critial aspect of computing today. Widely publicized

security breaches, for example (Leyden, 2004) and (CERT, 2003), highlight the need for a

forensic tool that will greatly assist system administrators in understanding the modus operandi

of an attack and assessing the extent of damage to their systems. This information can also help

fix the vulnerabilities that lead to successful exploitation. In this thesis, we describe the design

and implementation of a kernel-level subsystem that can record process activity at a very fine

granualarity. We also argue that, contrary to popular belief, there is little performance impact

resulting from a well-implemented kernel-level auditing system.

1.1 Background

An audit trail, as defined in (NIST, 1995), is a sequential record of computer events about an

operating system, an application, or user activity. The process of constructing an audit trail

and its subsequent examination is called auditing. The benefits of auditing include individual

accountability, reconstruction of higher-level events, intrusion detection, and problem analy-

sis. Additionally, the knowledge of the existence of an auditing mechanism can be a powerful

deterrence to potential attackers.

The ultimate goal of auditing is the reconstruction of high-level and human-understandable

events that attackers commit to breach security. Examples of these high-level events are user-

1
logon, buffer-overflow, privilege-escalation and password-stealing. However, currently there is

no mechanism that can be used to describe such high-level events. Instead, what is often used

is a record of system events that are at a very fine granularity. It is hoped that with such

a record, system administrators can understand the chain of high-level events that lead to

successful attacks and, more importantly, patch the very security vulnerability which made the

attacks possible in the first place. To this end, operating system vendors have implemented a

kernel-level auditing solution that records the arguments and return values of every system call

executed on the system.

With such a fine granularity of record generation and storage comes the fear of performance

overhead as a result of auditing. Many users and system administrators are loathe to install a

full featured auditing system, or to turn it on during normal daily activity because of the fear

of performance degradation on the host platform, in spite of the aforementioned benefits,

To better understand the cause of this performance degradation, and to investigate solutions

to mitigate the overhead, we proposed and implemented a system that significantly extends the

original LinuxBSM (Banford, 2001) (Ocelic, 2001) project. The improved LinuxBSM consists

of three components: (1) a set of kernel patches that augment the audit record generation of

the security-relevant system calls, (2) a kernel-space auditing device that provides a means for

user-space applications to interact with the kernel audit subsystem and, (3) a user-space audit

daemon that stores audit records into the local filesystem. These are described in detail in

Chapter 3.

1.2 Contributions

The contribution of our work is two fold. We have designed and implemented a kernel-level

security auditing system for the Linux operating system that significantly improves over the

most recent system (Banford, 2001) (Ocelic, 2001). Secondly, we have performed in-depth

benchmarking to quantify the actual performance overhead in practice. Based on these results,

we can say that a well designed and implemented auditing system incurs little overhead, so

system administrators can reap the full benefits of auditing without the fear of its performance

2
impact.

1.3 Report Organization

The report is organized as follows. We review the state-of-the-art in kernel-level auditing in

Chapter 2. Chapter 3 describes the design and implementation of our improved LinuxBSM.

Chapter 4 examines the performance impact of the auditing system, and we conclude and discuss

future work in Chapter 5.

3
Chapter 2

Related Work

2.1 Introduction

Given the benefits of auditing, many application-level software such as relational database

management systems and web servers provide comprehensive audit capabilities with extensive

information that can even be used in performance tuning. Even operating systems, such as the

recent Microsoft Windows Server 2003 (Microsoft, 2003) and Sun’s Solaris 10 (Sun, 2005) allow

comprehensive and highly configurable auditing policies.

Application-level auditing is commonly known as logging. They provide many benefits such

as system-health monitoring, application debugging and performance tuning. However, with

the specific aim of security auditing, we restrict our investigation to kernel-level auditing in the

following sections.

In this chapter, we survey the auditing tools available either as part of the operating system,

in section 2.2.1, or as a custom built solution in section 2.2.2. Then, in section 2.3, we look at

some audit benchmarks that were performed in the past and understand how it has become a

major obstacle to audit adoption.

4
2.2 Audit Tools

2.2.1 Built-in Logging Tool

At a basic level, the various Unix-like operating systems provide a logging utility called klogd

(Schwarz, 2000), the kernel logging daemon that intercepts and logs Linux kernel messages in

much the same way as the system logging daemon. The kernel can add an entry to the kernel log

by calling printk() that behaves almost identically to the C library printf() function. Invoking

the printk() function stores the new entry in a circular buffer and subsequently the user-space

klog daemon retrieves these messages from the buffer and feeds them into the log file via the

syslogd daemon.

Klogd is remarkable in that it aims to provide a unified logging framework to the kernel.

This enables critical kernel messages to be retrieved and reviewed easily by developers as well

as administrators and acted upon. However, klog entries are entirely customary and there

is no common standard on how, when and what to audit. Furthermore, the requirement for

security auditing mandates a custom auditing system that allows the effecient recording of

data-intensive audit trails. In recognizing this need, and in enabling rich and flexible auditing,

various operating system vendors, as well as the open source community, have implemented

their auditing solution.

2.2.2 Custom Audit Trails

The SunShield Basic Security Module (Sun, 2000) provides an auditing system for both SunOS

(Sunshield) and Solaris (Sunshield BSM). The BSM permits the actions of specific users to be

recorded and written to an audit file. Sun’s BSM is the originial inspiration to develop a high

quality audit tool for the open source Linux operating system, which is known as LinuxBSM.

LinuxBSM (Banford, 2001) is an initiative at University of California, Davis to provide a

comprehesive auditing package for Linux that is fully compliant with the U.S. Government’s C2

standards for security, defined in the standards document ’Trusted Computer System Evaluation

Criteria’ (DoD, 1985) or more commonly called the Orange Book. Specifically, the LinuxBSM

5
is an auditing tool that aims to bring the capabilities of Sun’s Solaris Basic Security Module

to Linux. LinuxBSM monitors and records system events on the kernel level, equipping system

adminstrators a mechanism to trace the activities in the system. The project’s development

stopped in November 2000 and its capabilities at that time included auditing a total of six

system calls for the Linux 2.2 kernel.

LinuxBSM2 (Ocelic, 2001) is the next evolution from LinuxBSM and it consists of several

architectural improvements over the original LinuxBSM, including a flexible system call audit

record generator. This tool allows an administrator to specify desired audit record fields for

new system calls in a plain text file. The file is used in inserting the relevant source codes to

enable auditing of these system calls automatically. In this way, kernel developers can avoid a

certain amount of manual duplication of identical code section in audit record construction. The

project is also targeted at the same Linux 2.2 kernel and the numbers of system calls audited

remains at six. Meanwhile, major kernel changes took place in 2001 and the first release of new

version 2.4 kernel was released at the same time. LinuxBSM2 cannot be used in the new kernel

due to these breaking changes. Similarly, development stopped towards the end of 2001.

Linux Auditing Subsystem (SuSE, 2003), LAuS, is a very similar auditing system imple-

mented by SuSE Inc. The system is composed of kernel-resident and user-space components

that facilitate highly-configurable and robust logging of system call invocations. It is currently

distributed as part of Red Hat Enterprise Linux 3 and SuSE Linux Enterprise Server 8 that are

based on the Linux 2.4 kernel. The design principle of LAuS is significantly different from the

improved LinuxBSM and we will describe its key features as follows.

• LAuS adds a new field to the task structure to store audit parameters to enable selective

auditing of processes. If a processes is marked to be audited, then relevant audit fields

will be recorded at the system call dispatch point. This has the disadvantage of the need

to modify individual system call handlers for each supported hardware architecture.

• LAuS stores the audit records in a memory buffer in the kernel-space. This buffer is like a

queue in that the audit records are read, one at a time, by invoking the read() system call

on the auditor device. Such a design necessitates frequent disk write operations and hence

6
 places a I/O load on host systems. Also, the rate at which audit records are generated

may outpace that of dequeuing from the memory buffer and this results in audit data loss.

The improved LinuxBSM solves these problems by avoiding an architecture-dependent de-

sign and employing a double-buffer strategy, as detailed in Chapter 3.

The abovementioned audit solutions, including the improved LinuxBSM, share a common

need for kernel patches. This is because the Linux kernel, by default, does not have any provision

for system call interception or monitoring. Hence, a kernel patch is the precursor for any system

call level auditing systems. However, the recent development in the Linux Security Modules

(Wright, Cowan, Smalley, Morris, & Kroah-Hartman, 2002) project introduces a lightweight,

general purpose access control framework as loadable kernel modules. The LSM framework,

itself a set of kernel patches, is on the verge of being integrated into the mainstream kernel. It

provides access to various kernel objects such as processes, IPCs and file system. Any access

control policy can be implemented as a LSM security kernel module. Currently there are some

problems with LSM, such as the lack of support for persistent security attribute binding without

resorting to external metadata storage. However, once LSM is stablized in the mainstream

kernel, the improved LinuxBSM can leverage the access to auditable kernel objects without

having to place system call hooks manually.

2.3 Auditing Benchmarks

Despite the well understood benefits of auditing, many system administrators are reluctant to

install an auditing system primarily due to two reasons.

1. Auditing tools have not been standardized and often the format of audit records are

customary to individual applications. As a result, audit records from different auditing

systems exist as information islands and there is no coherent view across these potentially

useful data. This is a classical data warehousing problem and solutions to it include

standardizing audit format for the various auditing sources, as proposed by (Bishop, 1995).

Alternatively, we can perform extraction, transform and load (ETL) by custom-built data

7
 adapters and store the result into a central repository. This method is favored by hybrid

intrusion detection systems such as Prelude (Mathieu Blanc, 2003).

2. The second major hurdle to widespread adoption of auditing is due to the fear of its

performance impact on the host system. To this end, we find surprisingly little literature

on documented performance benchmarks on auditing. (Nitzberg, 1994) performed an

early analysis of the performance impact of the audit component available in the HP-UX

version 8.0, a hybrid between AT&T System V and BSD Unix, and found that the worse

case impact is as high as 85% degradation. This might explain the fear of implementing an

auditing solution. More recently, (Veronique Abily, 2000), used the SunShield BSM (Sun,

2000) to measure the performance impact of an intrusion detection system and discovered

that the auditing component contributed to about 4% performance hit. This result is

much more encouraging than the earlier study. However, the result is buried as part of

benchmarking process of an intrusion detection system.

In chapter 4, we describe the benchmark results of our auditing system that provide con-

clusive evidence about the fact that auditing does not impact system performance in most

systems.

8
Chapter 3

System Design and Implementation

This chapter describes the design and implementation issues in the improved LinuxBSM. It

starts with a discussion on the obvious (and inefficient) approach to kernel auditing in section

3.1. Section 3.2 explains the approach employed by the improved LinuxBSM to what and how

to audit in the improved LinuxBSM.

3.1 Naive Approach

As argued in (Sekar, Bowen, & Segal, 1999), regardless of how an attack is delivered, any dam-

age to a host is effected via the system calls made by a process running on the host computer.

Therefore, auditing system calls is an efficient way of analyzing (potential) security breaches.

Note that this is different from auditing policies in the Microsoft Windows Server family (Mi-

crosoft, 2003) where a level of abstraction above system calls is provided.

A naive approach to system call auditing would involve ’capturing’ system calls in the ar-

chitecture dependent system call handler1 . As user-space programs invoke system calls, threads

of execution trap into the kernel from where the system call is dispatched. At this point, an

audit record containing the system call number and associated arguments can be generated and

written to the local file system. However, there are at least two problems with this approach.

1. Invoking system calls can have different post-conditions and return values. For example,
1
For example, in the Linux kernel, the system call handler resides in entry.s

9
 the read() function call can succeed or fail and when it succeeds, the file offset can have

different values based on how many bytes are read. As a result, if only a set of generic

parameters are recorded in the audit trail, then it will be remarkably more difficult to

distinguish one system call from another.

2. More seriously, if every system call results in a disk write operation to the file system, it

is clear that auditing will become a bottleneck to system performance.

3.2 Improved Approach

To overcome the abovementioned problems, the improved LinuxBSM consists of three compo-

nents, a set of kernel patches to the security sensitive system calls, a kernel device driver that

records audit events in memory buffers (shared with the user-space daemon), and a user-space

daemon that periodically writes to the local file system. This design enhances the availability

of the host on which it runs while providing the benefits of auditing. We describe the three

components in the following sections.

3.2.1 Kernel Patches

The main design policy of the system is to maximize its utility and minimize its load in a

production environment. To maximize its utility, we must recognize the ingenuity of attackers

and their pattern of intrusions. In an ideal situation, we will audit all system calls. A system

administrator will have a complete record of events to reconstruct the steps of an attack.

However, such an approach will not only overwhelm the system administrator with mostly

normal audit records, the system on which the audit trail runs will incur a significant perfor-

mance hit. Performance degradation due to auditing on Unix is reported (Nitzberg, 1994) to

range from negligible impact for the arithmetic component of the benchmark to 85% degradation

for system call overhead tests.

Therefore, we need to strike a balance between security and performance. (Bernaschi,

Gabrielli, & Mancini, 2002) divided the Linux system call set into nine groups, as shown in

Table 3.1 according to their intended functionalities. They assert that it is safe to disregard

10
 Table 3.1: System call group
Group Functionality

1 File system and device

2 Process

3 Loadable kernel module

4 Memory

5 Time and Timer

6 Interprocess Communication

7 System Information

8 Reserved

9 Not Implemented

system calls from group 4 to 9 as they cannot be used to subvert a system. For example,

unimplemented system calls (group 9) obviously cannot be used to subvert a system. So are

reserved (group 8) and system information related (group 7) calls. More interestingly, system

calls that deal with memory management (group 4), timers (group 5) and interprocess commu-

nication (group 6) are useless to an attacker who seeks to control either the running system or

the running process. We are left with system calls from group 1 to group 3 where either the file

system, processes or kernel modules are involved.

They also classifed the system calls, as in Table 3.2 according to four levels of security

threats, from allowing full control of the system to harmless. The result is that a total of 102

system calls are identified as security sensitive.

In our implementation, we apply a set of kernel patches to a stock Linux kernel version

2.4 as shown in Figure 3.1. These patches are placed at critical system call points to capture

security related information to a buffer. The set of system calls chosen are based on the study

done by (Bernaschi et al., 2002), with the exception of signals and real-time signals.

11
 Table 3.2: Threat level classification
Threat Level Description

1 Allows full control of the system

2 Used for a denial of service attack

3 Used for subverting the invoking process

4 Harmless

Figure 3.1: Audit records are generated at system calls

12
 As can be seen from Figure 3.2, the system calls are categorized into three threat levels.

• Level 1 refers to system calls that allow full control of the system.

• Level 2 identifies those calls that can be used for a denial of service attack on the host

machine

• Level 3 includes system calls that can be used for subverting the invoking process.

Note that by convention, the smaller the threat level number, the more dangerous are the

system calls. Also, there exists a total order across the threat levels; for if a system call allows

the attacker to take full control of a system, he can subsequently launch a denial-of-service

attack on the local machine trivially.

Together, these system calls deal with file systems, devices, processes and module manage-

ments that are the core functionality of the Linux kernel. By auditing these specific system

calls, we can discover actions that lead to successful or attempts of security breaches. For ex-

ample, various attempts have been made in the past to effectively examine audit trails in order

to discover important events that lead to a security breach. Such a forensic discovery process

can be greatly assisted by two types of tools.

The first type of log analyzers, such as Xlogmaster(Greve, 1998) and Swatch(Hansen &

Atkins, 1993), focus mainly on the effective filtering of audit events according to user-defined

rules. The problems of these tools soon become obvious to practioners that they require

highly experienced professionals who know what they are looking for from audit trails. Se-

curity breaches are often the result of successful exploitation of unprecedented vulnerabilities

in operating systems or application programs. This calls for a tool that can automatically trace

from compromised objects to the initial break-ins.

The answer to that is the second type of audit trail analyzers, such as Tudumi(Takada &

Koike, 2002) amd BackTracker(King & Chen, 2003). In short, they employ techniques from

infomation visualization to present the chain of events in a graphical user interface (GUI). In

this thesis, we merely state that auditing security related system calls can lead to a successful

identification of security vulnerabilities.

13










 


 



 
























 











 










 
 
























 























 










  
 










 
 









 







 










 
 






















 













 




































 





















 
























































 
 








 








 
 






















 






  
 










  


 








 










  









  












  









 
  










 
 
 




























 








Figure 3.2: Security sensitive system calls

14
3.2.2 Auditor Device

Since we have previously argued that write operations to the file system is expensive and should

be minimized, we create a character device in the kernel-space that writes audit records as they

are created by the hooks in the modified system calls to memory buffers. The reason that the

auditor is a character device and not a block device is that we do not require the nonsequential

access to the memory buffer. Nonsequential access to block devices such as floppy drives and

CD-ROM drives are not relevant to the auditor device. At run time, the auditor device will fill

up a memory buffer with audit records sequentially and a user-space audit daemon will clear

the audit records once the memory stream is full, also sequentially. Implementing it as a block

device would incur considerable effort with no apparent benefits. The other important task

of this auditor is to signal to the user-space audit daemon that the memory buffer is full and

the audit daemon should copy the contents in the buffer out. In this way, disk write is only

performed periodically and the periodicity can be adjusted via the size of the buffer based on

empirical results of performance impact on the system.

Storing audit records in a memory buffer solves the problem of frequent write but it creates

another problem. When the content of a full buffer is being copied to the user-space, some

audit records will be lost as the the buffer is in the read mode2 . At first thought, it is tempting

to solve this problem by simply implementing the memory buffer as a queue data structure. In

this way, as soon as the buffer is full, we can start dequeuing from the other end of the buffer

to accommodate the incoming audit records. However the rate of clearing the content into

user-space might not equal or exceed the rate at which new audit records are generated. Hence,

audit loss becomes inevitable in some circumstances. In the security sensitive environment, we

must assume that an attacker has access to the mechanism with which auditing is done. He

will exploit this vunerability so that the record of the key events are lost during the so-called

buffer jams.

In our improved LinuxBSM, we apply the double-buffer strategy, shown in Figure 3.3, to

avoid audit record loss. Specifically, the kernel auditor device associate two memory buffers
2
From the perspective of the user-space daemon

15
and label them as the read and write buffer respectively. From the user-space daemon’s point

of view, read buffer refers to the daemon’s reading from this buffer. Write buffer is the memory

region that audit records are appended as they are generated. Once the write buffer is full, the

device driver sends a signal to the user-space daemon and immediately swaps the two buffers.

In this way, while the audit daemon writes the content of read buffer to the local file system,

fresh audit records are written to the newly appointed write buffer. The double-buffer strategy

is reminiscent to graphics rendering devices in that while the front buffer is rasterized onto

screen display the back buffer can be used to store the content of the next frame to render.

This design is due to the original LinuxBSM.

It might be argued that circumstance might occur where audit records are generated at a

rate that is faster than that of being cleared by the user-space audit daemon and hence propose

a triple-buffering strategy. We believe that while triple-buffering certainly adds a higher level

of assurance against audit record loss and in fact, a variable number of buffers can be used

in an adaptive manner depending on the rate of audit growth, empirical results indicate that

double-buffering is sufficient even at high-load situations, as we can see from Chapter 4.

16
 17

Figure 3.3: Auditor device with double buffering

3.2.3 Audit Daemon

The communication between the kernel-space and user-space in the Linux operating system

is facilitated through the use of signals. Signals, in this case, are used as software interrupts

that allow asynchronous notifications from kernel-space to user-space about events being raised.

Specifically in our design, as soon as we reach the point where storing one more audit record

will overflow the write buffer, the kernel notifies a user-space audit daemon with a SIGIO signal

to swap the contents in the buffer out of the kernel-space. A daemon is similar to a service in

the Windows operating system that is an autonumous background process. The audit daemon

copies the content from the read buffer in the kernel and writes to the local file system partition

or sends the audit data via network to another computer.

Before the audit daemon is running, the kernel auditor device will not generate any audit

records. The audit daemon is started with a set of command line parameters including the range

of process numbers, user ids and group ids to audit. These allow the selective auditing of events

that are deemed useful by the system administrator. Besides setting the audit parameters, the

audit daemon also registers the signals and attaches signal handlers to them. In this way, it can

perform the necessary actions such as writing to filesystem and closing the audit device based

on the received signals. Once the initialization of opening the device, setting up the memory

buffers and registering signal handlers are done, the audit daemon informs the kernel with a

specific IOCTL command. As soon as the kernel is aware that the audit daemon is running, it

can start auditing predefined system calls.

18
Chapter 4

Evaluation

In this thesis, we benchmark the performance overhead of our improved LinuxBSM that audits

a total of 82 security related system calls. Following the standard practice of benchmarking

the kernel, for example as done for Linux Security Modules (LSM) (Wright et al., 2002), we

use lmbench (Larry W. McVoy, 1996) for microbenchmarking. lmbench is developed specifi-

cally to measure the performance of core kernel system calls and facilities such as file system

access, context switching and memory access. For macrobenchmarking, we follow the oft used

convention of measuring kernel compile times.

4.1 Test Environment

We compare the standard Linux 2.4.22 kernel against the 2.4.22 kernel incorporating the im-

proved LinuxBSM with the audit daemon running. The tests are run on a single-processor 1.6

GHz Pentium 4 computer with 256MB of SDRAM and one 20GB Ultra ATA/133 hard disk

drive. In most tests, the performance overhead of the improved LinuxBSM is within the bound-

ary of the inevitable experimental errors. In certain cases we even have better results with

auditing and that is almost certainly due to random errors that are part of any experiment.

In addition, the size of each buffer, which is used by the auditor as described in Chapter 3, is

configured to be 250 kilobytes. The results are shown in section 4.2 and 4.3.

19
4.2 Microbenmarks

4.2.1 Arithmetic Operations

Figure 4.1 shows that, for basic arithmetic opreations, besides empirical disrepancies, there is

no difference between the unpatched kernel and improved LinuxBSM. This is to be expected as

these CPU-bound tasks do not invoke any system calls that we audit.

20
 
!
!

!

 


 
 
 
 

    

"!    


"!   



   
 
#
 

 
!
!

!

 

   


  

"! 
    
"! 
   


    
#
 

 
"
!

!

 

   
   


"!      
"! 
    

    
#
 

Figure 4.1: Integer, float and double opreations

21
4.2.2 Memory and File System Operations

Just as auditing has no performance impact on arithmetic operations, so is its lack of overhead

on memory latencies, as shown in Figure 4.2. In our test machine we have both level 1 and

level 2 cache and access to these memory is effected through architecture specific assembly code

where no auditing is performed. Furthermore, main and random memory access are not audited

too because so far attackers are unable to manipulate kernel memory directly without crashing

the system. Since no auditing is done to memory operation, no consistent performance impact

could possibly be observed.

On the other hand, file system operations are fully audited due to their sensitive nature. In

the Linux operating system, most objects, such are directories and devices, can be considered

as part of the file system. As a result, we expect and observe a large proportion of filesystem

related audit records. Therefore, this is the most likely bottleneck of our auditing mechanism.

However, Figure 4.2 shows encouraging results as we fail to observe significant performance

differences.

22
!&
#"

""


 

 



$ # 
 


$#  

 

!    
%!
 




"&"#
#"

!""

 

   
       
      

$ # 
          

$# 
         

  

    
!

%! 



Figure 4.2: Memory and file system results

23
!%!
$!

!

 

     







"!       

"!         
       


# 


 
 

!

 

" " !!  !   
# 
       !    
"!       
   

"!      

  

    
      


# 



Figure 4.3: Process related results

4.2.3 Process Related Benchmarks

The performance overhead of auditing context switch, which occurs when the scheduler calls

context switch() to run the next process in the run queue, is shown in Figure 4.3. Benchmarks

are performed on 2, 8 and 16 processes with 0, 16 and 64KB of data associated with the switching

processes. Again, except for random errors, we fail to observe audit-related performance impact.

Also shown in Figure 4.3 is process related benchmarks such as process creation fork() and

overlay exec() system calls. These system calls, among others, are audited but we do not see

any discernible performance difference.

24
4.2.4 Communication Benchmarks

Finally, we measure the bandwitdth and latency of memory related operations and interprocess

communications. Again, we fail to observe, as shown in Figure 4.4, any significant difference

between the unpatched kernel and the modified kernel with the auditor running.

25
"
" '!&"!
&!%
!
$"%"!%
  
      
!   


'!#& 
     
 

'& 
    
   

    
 
 
$
!
($




"
" '!&"!
!)&%
!
%
 

     
  " "  
 


 
'!#&  
   
   


'&     
  
  


  
          
$
!
($  



Figure 4.4: Communication results

26
 Table 4.1: Linux kernel compilation macrobenchmarks, time in seconds
unpatched auditd running % Overhead with auditd

313 316 1.0

4.3 Macrobenchmarks

We perform the widely used kernel compilation macrobenmark, measuring the time to build

the Linux kernel. As before, we run the compilations on a single-processor 1.6 GHz Pentium 4

computer with 256MB of SDRAM and 20GB Ultra ATA/100 disk.The command make bzImage

is executed with the results shown in Table 4.1.

4.4 Analysis

Our observation of the negligible performance overhead resulting from kernel auditing requires

some explanation. We present an approximate calculation of the performance impact here which

is in line with the observed results and therefore provides credence to it.

The memory size, B, used by the auditor is configured to be 250 kilobytes for each buffer.

The buffers are described in detail in Chapter 3. Empirical observation shows that the auditor-

induced disk write occurs at 5 second frequency during intensive I/O operations.

The hard disk drive running on the test PC has the following relevant parameters.

• Number of sectors per track, denoted by P: 63

• Number of bytes per sector, denoted by Q: 512

• Track-Track seek time, denoted by s: 0.8 ms

• Average seek time, denoted by S: 8.5 ms

• Rotational Delay, denoted by R: 4.2 ms

• Data Transfer time, denoted by D: 0.595 ms per track

27
4.4.1 Ideal Scenario

In the ideal scenario where the partition to which audit data is written is unfragmented, the

audit daemon approximately writes to consecutive free blocks and little time is lost because

of seek delay. Under such an assumption, we have T, the total time, required for the write

operation as
B
T = ∗ (S + R + D)
PQ
where B is the buffer size, P is the number of sectors per track, Q is the number of bytes per

sector, S is the average seek time, R is the rotational delay and D is data transfer time. The
B
first multiplier PQ calculates the number of tracks written. The second multiplier, (S + R + D)

calculates the time required to write one full track. The result of this multiplication is the total

time, T, required for the write operation.

Hence, at every 5 second intervals we spend

250000
∗ (8.5 + 4.2 + 0.595) ≈ 106ms
63 ∗ 512

in write operations.

This is about 2% (106 ∗ 10−3 /5 ≈ 0.0212) and that is close to the performance overhead

observed in our benchmarks.

4.4.2 Worse Case Scenario

In the worse case scenario, the partition is heavily fragmented due to frequent read-write op-

erations. In this case, the disk access incurs a seek on every sector that is written. In this

scenario,

B
T = ∗ (S + R + D + s)
PQ
Hence, at every 5 second intervals we spend

250000
∗ (8.5 + 4.2 + 0.595 + 0.8) ≈ 109ms
63 ∗ 512

in write operations.

This is again about 2% (109 ∗ 10−3 /5 ≈ 0.0218) performance overhead.

28
4.4.3 Remarks

With these hardware parameters, we understand that the negligible audit performance overhead

is due to the rapid advances in hardware technology. It is also worth noting that lmbench

benchmarking suite emulates both I/O intensive and typical interactive sessions in its testing.

The result of it therefore covers both scenarios in which auditing is highly desirable.

While lmbench provides us with a set of performance measurements of typical system oper-

ations on the Linux kernel, we aim to measure the worst case scenario with the audit subsystem

turned on. In order to do that, we create a program that repeatedly calls open() and close()

on a file in the local file system 10,000,000 times. The process that executes this program does

virtually nothing except invoking audited system calls and it provides us with a satisfactory

upper-bound of the performance overhead of the audit subsystem.

int main()

int fd;

int counter = 0;

while (counter < 10000000)

fd = open("abc");

close(fd);

count++;

return 0;

Executing the above program on the same test machine takes about 74.2 and 45.7 seconds with

auditing turned on and off repsectively. This indicates an upper bound of 62.4% performance

overhead. In reality, processes invoke system calls in a manner that is more akin to lmbench’s

simulation than to our explicitly constructed upper bound. Across this spectrum of system call

29
intensities, system administrators will have to understand the opreating environment and make

a judgment call on whether to turn on kernel-level auditing.

30
Chapter 5

Conclusions and Future Work

5.1 Future Work

5.1.1 Tamper Proof Audit Trail

One requirement for forensic analysis on compromised computers is that the audit trails be

tamper proof. Such a framework is described in (Schneier & Kelsey, 1999), and we plan to

study and implement the framework to our improved LinuxBSM. This will remarkably increase

the reliability to LinuxBSM as a forensic tool.

5.1.2 LSM Integration

As described in earlier, we intend to port our auditing mechanism to the LSM framework. This

has two advantages.

1. As part of a standardized effort to provide enhanced security to the Linux operating

system, our auditing system can be more easily accepted by system administrators.

2. The current improved LinuxBSM is based on the Linux 2.4 kernel. By porting it to the

LSM framework, it automatically works with the latest Linux 2.6 kernel.

31
5.2 Conclusions

In this thesis, we provide a much more fine-grained auditing system for the Linux operating

system to the Linux 2.4 kernel. Also, we discover that given a well implementd auditing system,

performance overhead is negligible as compared to an unpatched kernel. With these, we believe

the scale of auditing adoption by system administrators will improve considerably and security

vulnerabilitiy exploits can be discovered and rectified in a much more timely fashion.

32
References

Banford, J. (2001). Linuxbsm. http://linuxbsm.sourceforge.net/, , July, 2001.

Bernaschi, M., Gabrielli, E., & Mancini, L. V. (2002). Remus: A security-enhanced operating
system. ACM Transactions on Information and System Security, 5 , February, 2002, 36–
61.

Bishop, M. (1995). A standard audit trail format. Proceedings of the 18th NIST-NCSC National
Information Systems Security Conference (pp. 136–145), Baltimore, Maryland, October,
1995.

CERT (2003). Cert advisory ca-2003-23 rpcss vulnerabilities in microsoft windows.

http://www.cert.org/advisories/CA-2003-23.html, , September, 2003.

DoD (1985). Trusted computer system evaluation criteria. Washington, DC 20301-1000: De-
partment of Defense.

Greve, G. C. F. (1998). The xlogmaster. http://www.gnu.org/software/xlogmaster/xlogmaster.html,

, June, 1998.

Hansen, S. E., & Atkins, E. T. (1993). Automated system monitoring and notification
with swatch. Proceedings of the Seventh Systems Administration Conference (LISA VII)
(USENIX Association: Berkeley, CA), , November, 1993, 145.

King, S. T., & Chen, P. M. (2003). Backtracking intrusions. Proceedings of International

Conference on Graph Theory, Ann Arbor, MI 48109-2122, October, 2003.

Larry W. McVoy, C. S. (1996). lmbench: Portable tools for performance analysis. USENIX
Annual Technical Conference (pp. 279–294), San Diego, CA, January, 1996: USENIX
Association.

Leyden, J. (2004). Students find 44 unix flaws as homework.

http://www.theregister.co.uk/2004/12/16/unix flaw homework/, , December, 2004.

Mathieu Blanc, Laurent Oudot, V. G. (2003). Global intrusion detection: Prelude hybrid ids
(Technical report). Villeurbanne, France: ExaProbe.

Microsoft (2003). Windows server 2003 product documentation. Redmond,WA, 98052: Mi-
crosoft Corporation.

NIST (1995). Introduction to computer security: The nist handbook. Gaithersburg, MD: NIST.

Nitzberg, S. (1994). Performance benchmarking of unix system auditing. Master’s thesis,

Monmouth college, Monmouth, IL.

33
Ocelic, D. (2001). Linuxbsm2. http://linuxbsm2.sourceforge.net/, , December, 2001.

Schneier, B., & Kelsey, J. (1999). Secure audit logs to support computer forensics. ACM
Transactions on Information and System Security, 2 (2), May, 1999, 159–176.

Schwarz, M. A. (2000). Take command: klogd: The kernel logging daemon. Linux Journal, ,
August, 2000, 20.

Sekar, R., Bowen, T., & Segal, M. (1999). On preventing intrusions by process behavior
monitoring. Proceedings of the Workshop on Intrusion Detection and Network Monitoring
(pp. 29–40), Santa Clara, CA, April, 1999.

Sun (2000). Sunshield basic security module guide. Palo Alto, CA: Sun Microsystems.

Sun (2005). Solaris 10 product documentation. Santa Clara, CA 95054: Sun Microsystems.

SuSE (2003). Linux audit-subsystem design documentation. Waltham, MA 02451: Suse Inc.

Takada, T., & Koike, H. (2002). Tudumi: Information visualization system for monitoring and
auditing computer logs. Proceedings of Sixth International Conference on Information
Visualisation, London, England, July, 2002.

Veronique Abily, M. D. (2000). Benchmarking a distributed intrusion detection system based

on asax: Preliminary results. Lecture Notes in Computer Science, 1907 , April, 2000.

Wright, C., Cowan, C., Smalley, S., Morris, J., & Kroah-Hartman, G. (2002). Linux security
modules: General security support for the linux kernel. Proceedings of the 11th USENIX
Security Symposium (pp. 17–31), San Francisco, CA, August, 2002: USENIX Association.

34
Project Information Page
Project Type : Honours Year Project

Project Area : Software System

Project Title : Assessing the Performance Impact of the Linux

Kernel Audit Trail

Project No : H089020

Students Name : Qiu Jiang

Project Advisor : Prof. Sandeep Kumar

Date of Completion : April 2005

Deliverables : Report 1 Volume

Implementation

Software and Hardware : Linux Kernel 2.4

Abstract
Security breach into information systems is almost always caused by unpatched or unknown
vulnerabilities. To counter this threat, system administrators often rely on effective auditing
to discover such security violations. The benefits of auditing include accountability,
reconstruction of higher-level events, intrusion detection, and problem analysis. Furthermore,
the mere presence of auditing is a powerful deterrence to potential attackers. Despite these
well-understood benefits, system administrators are reluctant to turn on auditing on their
systems because of the fear of performance overhead. In this thesis, we design, implement and
benchmark a low overhead auditing sub-system for the Linux 2.4 kernel.

Subject Descriptors:
C5 Computer System Implementation
D.4.6 Security and Protection

Keywords:
audit, kernel

35