Trends in Database Log Management

Anton Chuvakin, Ph.D.

WRITTEN: 2007

DISCLAIMER:
Security is a rapidly changing field of human endeavor. Threats we face literally
change every day; moreover, many security professionals consider the rate of
change to be accelerating. On top of that, to be able to stay in touch with such
ever-changing reality, one has to evolve with the space as well. Thus, even
though I hope that this document will be useful for to my readers, please keep in
mind that is was possibly written years ago. Also, keep in mind that some of the
URL might have gone 404, please Google around.

Introduction: Why Database Logs Fall In The Realm of Database Security

Buried deep within enterprise IT infrastructures, databases can be said to

hold the “crown jewels” of an organization. Unfortunately, database security is
often lacking, leaving sensitive, business-critical information such as customer
data, financial details, and more, vulnerable to hackers. Dept of VA, TJ Maxx, TD
Ameritrade – these are just a few of the many organizations that have driven the
media wild over data security breaches in the last year.
It is common that database administrators (DBAs) are assigned the task of
database security, but this is an issue that should be of utmost importance to any
business that wants to stay in business. TJ Maxx reported at least 45.7 million
credit and debit card numbers stolen over a period of several years, costing the
company an estimated $168 million [or whatever other large random number ].
Proper security measures may not have stopped the initial hack-in, but perpetual
data theft could have been avoided through careful log collection and analysis.
This article will not only discuss the importance, challenges and benefits to
database logging, but will also offer a few forward-looking trends to managing
your database logs.

About Logs and Database Logging

Databases are now becoming one of the most voluminous log generators
in the enterprise – rivaling firewalls for the top spot. Most databases (ie: Oracle,
Microsoft SQL Server, IBM DB2, MySQL, etc.) will log system starts, stops and
restarts by default, but database logging isn’t merely about “keeping the system
running,” particularly when your databases contain sensitive, private information.
Security and compliance requirements must therefore be considered when
configuring your database and managing your logs. In fact, regulations such as
PCI, HIPAA, and FISMA all mandate log monitoring, with SOX strongly
recommending it as a best practice.
Database logging thereby becomes an essential (and required)
component of database security – and it makes sense to not only focus on
“keeping the bad guys out,” but also take a “what’s going on in here?” approach.
After all, you may not know who the “bad guys” are. Logs can provide a
continuous fingerprint of everything that happens in your IT systems and with
your data and will point you to the “who, what, when, where” information of any
breach – whether the malicious behavior comes from outside hackers, a
disgruntled employee, or another source.
Database security is a task often assigned to DBAs, not because they’re
security experts, but because they know the in’s and out’s of databases. If
configured properly, databases may be logging overwhelming amounts of files,
perhaps up to gigabytes of data per day. Typical database log events may
include:

• User logins and logouts

• Database system starts, stops and restarts
• Various system failures and errors
• User privilege changes
• Database structure (metadata) changes
• Most other DBA actions
• Select or all database data access (if configured to be so)

As we know, hackers are always looking for new ways to break through
security barriers to access your sensitive information and all preventative security
measures fail at some point. Thus, since you are not able to guard against every
malicious hacker, logs will at least allow you to detect such security breaches as
well as actually figure out how it was done during the incident investigation. At a
minimal level, logs must be collected and archived, but log analysis does make
the data significantly more useful. In more explicit terms, log monitoring and
management should include:

• Collection: Gathering log data where it is being generated via an agent or

remotely
• Transfer: Securely transmitting log data to a central server for analysis
and storage
• Alerts: Issuing real-time alerts to database administrators if needed
• Reporting and Analysis: Providing reports and analytics based on log data
• Storage: Securely storing logs as long as prescribed by your retention
policy and then, just as safely, destroying them

The above examples for managing your log data will help you keep tabs
on the activities occurring in your business. Regularly collecting log data is a best
practice for incident response and can save you during crunch time after a server
crash, data theft, or surprise visit by your friendly auditor. Alternatively, if
someone is downloading an entire table or changing a database schema while
being logged on from a remote connection, a real-time alert will catch your
attention. Further, reports may help you track and analyze login failures and
successes, or after hours access, to better evaluate insider privilege abuse. In
other words, database logs can help you catch unusual behavior before a
problem gets out of hand and into headline news.

Database Log Management: Where to Begin

If you are just beginning to set up a method for managing your database
log data, be ready for a large volume of log records as well as issues pertaining
to log availability and log format complexity. Log formats can be verbose and
obscure, particularly in cases where a single message spans multiple lines,
making it difficult to extract useful, actionable information via automated tools.
Other challenges to database logging include decreased performance and
storage restrictions. Unlike other situations where logging has a minimal impact
on system performance, database audit logging slows down the database,
sometimes significantly. High-performance databases are built to provide
thousands of data transactions per second, logging all of these presents a
challenge to system IO as well as CPU and disk storage resources. Since many
regulations specify a 3-12 month period for log retention, plus a longer period for
log retention on tape or another dedicated storage tool, database logging is
typically getting a bad rap among DBAs already spread thin for time and
resources.
Because the difficulties associated with database log management can
seem overwhelming, it’s best to take things one step at a time. Start slowly and
build up your system from there. You’ll want to collect logs from multiple servers
at one central location to facilitate incident analysis and response. This will also
help prevent loss of log data during routine log rotations. To gain insight into
“What’s going on” internally, conduct periodic reviews of DBA activity logs – you
can then keep tabs on people entrusted with sensitive and/or private information
such as customer data or product inventory information.
When beginning to organize and manage database log data, also keep in
mind that manual log analysis can cost a lot in terms of time, human resources,
etc. Popular database solutions such as Oracle, Microsoft SQL Server, MySQL,
and more, tend to offer various basic logging options, but none comprehensive
enough to really capture a continuous feed of database activity. By contrast, an
automated log management tool will not only free up DBA time for other
important database performance and security tasks, but can also be more
reliable and efficient than manually managing log data.

Further, with an automated LMI tool, you can schedule log collection to occur at
off hours when other database service operations, such as backups, are
happening so that database performance is undeterred during the workday.
Trends in Database Log Management

Database logging often presents a new frontier for many security

practitioners – one that must be conquered. Given that historically many
databases were running without any data access and data change logging
disabled (as by default), the key trend is that this is finally beginning to change.
Why is it happening? There are two main drivers for this trend: PCI DSS
compliance requirements that apply to those who handle credit card data and the
proliferation of data breaches and data loss discussed above. The cost of data
loss investigations in the absence of detailed access logs is absurdly high!

What would happen next? As more people enable logging, the challenge of “what
to do with all that data?” will emerge. Handling log storage and controlling log
retention so that logs will be there when needed for the investigations will be the
next trend.

After that, database log analysis will become all the rage. Analyzing logs for
anomalies, suspicious user activities, unsafe administrator actions, privilege
abuse as well as good old hacking will require deployment of log management
tools with database-specific intelligence.

Further, logging guidance from IT “best practices” such as ITIL and ISO will
become the norm and we will reach the database logging nirvana, when logging
is enabled because “it is the right thing” and not only due to compliance
pressures or the latest data breach. Log collection and automated analysis will
become the norm as new a log management and intelligence (LMI) technologies
simplify managing the log flood as well as “making sense” of logs.

So, enabling logging is a good start, and taking small progressive steps towards
more in-depth log analysis can be greatly enhanced by deploying LMI platform.

LMI tools are becoming increasingly advanced and are now typically able to take
a deep dive in to log analysis and management. A good log management
solution will not only automate log data analysis, but also the whole lifecycle of
log management.

There is also an increasing trend in logging across the entire IT infrastructure –

firewalls, servers, network devices, applications, operating systems, other
sources of log data all produce logs! – all of which can be managed by an LMI
platform. In other words, jump on the logging bandwagon with your other system
administrators and balance your IT infrastructure with a log management system
that will work across multiple database servers, across various database types,
and with all other log producers in your organization. You will not only improve
performance and business continuity, but also be able to put database log data in
the context of other organizational log data to correlate IT activities with events
occurring in your business.

Conclusion

Database log management is becoming a “best practice” for database

security – you should be aware of who is accessing or changing your data, when
they are accessing it, and where they are accessing it. Luckily, you can combine
database log management with other similar projects (such as firewall or Unix
server syslog management) and use a single automated LMI platform to enable
efficient and reliable log collection, reporting analysis, and retention. As long as
you grow your LMI deployment in phases, rather than trying to cover all logs on
day one, you will be on the path to overall greater IT security within your
organization.

ABOUT THE AUTHOR:

This is an updated author bio, added to the paper at the time of reposting in
2009.

Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in

the field of log management and PCI DSS compliance. He is an author of books
"Security Warrior" and "PCI Compliance" and a contributor to "Know Your Enemy
II", "Information Security Management Handbook" and others. Anton has
published dozens of papers on log management, correlation, data analysis, PCI
DSS, security management (see list www.info-secure.org) . His blog
http://www.securitywarrior.org is one of the most popular in the industry.

In addition, Anton teaches classes and presents at many security conferences

across the world; he recently addressed audiences in United States, UK,
Singapore, Spain, Russia and other countries. He works on emerging security
standards and serves on the advisory boards of several security start-ups.

Currently, Anton is developing his security consulting practice, focusing on

logging and PCI DSS compliance for security vendors and Fortune 500
organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance
Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging
Evangelist, tasked with educating the world about the importance of logging for
security, compliance and operations. Before LogLogic, Anton was employed by a
security vendor in a strategic product management role. Anton earned his Ph.D.
degree from Stony Brook University.