Splunk Operational Intelligence Cookbook

Table of Contents

Splunk Operational Intelligence Cookbook

Credits

About the Authors

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Play Time – Getting Data In

Introduction

Indexing files and directories

Getting ready

How to do it...

How it works...

There's more...

Adding a file or directory data input via the CLI

Adding a file or directory input via inputs.conf

One-time indexing of data files via the Splunk CLI

Indexing the Windows event logs

See also

Getting data through network ports

Getting ready

How to do it...

How it works...

There's more...

Adding a network input via the CLI

Adding a network input via inputs.conf

See also

Using scripted inputs

Getting ready

How to do it...

How it works...

See also

Using modular inputs

Getting ready

How to do it...

How it works...

There's more...

See also

Using the Universal Forwarder to gather data

Getting ready

How to do it...

How it works...

There's more...

Add the receiving indexer via outputs.conf

Loading the sample data for this book

Getting ready

How to do it...

How it works...

See also

Defining field extractions

Getting ready

How to do it...

How it works...

See also

Defining event types and tags

Getting ready

How to do it...

How it works...

There's more...

Adding event types and tags via eventtypes.conf and tags.conf

See also

Summary

2. Diving into Data – Search and Report

Introduction

Making raw event data readable

Getting ready

How to do it...

How it works...

There's more...

Tabulating every field

Removing fields, then tabulating everything else

Finding the most accessed web pages

Getting ready

How to do it...

How it works...

There's more...

Searching for the top 10 accessed web pages

Searching for the most accessed pages by user

See also

Finding the most used web browsers

Getting ready

How to do it...

How it works...

There's more…

Searching the web browser data for the most used OS types

See also

Identifying the top-referring websites

Getting ready

How to do it...

How it works...

There's more…

Searching for the top 10 referring websites using stats instead of top

See also

Charting web page response codes

Getting ready

How to do it...

How it works...

There's more...

Totaling success and error web page response codes

See also

Displaying web page response time statistics

Getting ready

How to do it...

How it works...

There's more...

Displaying web page response time by action

See also

Listing the top viewed products

Getting ready

How to do it...

How it works...

There's more...

Searching for the percentage of cart additions from product views

See also

Charting the application's functional performance

Getting ready

How to do it...

How it works...

There's more...

See also

Charting the application's memory usage

Getting ready

How to do it...

How it works...

See also

Counting the total number of database connections

Getting ready

How to do it...

How it works...

See also

Summary

3. Dashboards and Visualizations – Make Data Shine

Introduction

Creating an Operational Intelligence dashboard

Getting ready

How to do it...

How it works...

There's more...

Changing dashboard permissions

Using a pie chart to show the most accessed web pages

Getting ready

How to do it...

How it works...

There's more...

Searching for the top 10 accessed web pages

See also

Displaying the unique number of visitors

Getting ready

How to do it...

How it works...

There's more…

Adding labels to a single value panel

Coloring the value based on ranges

See also

Using a gauge to display the number of errors

Getting ready

How to do it...

How it works...

There's more…

See also

Charting the number of method requests by type and host

Getting ready

How to do it...

How it works...

See also

Creating a timechart of method requests, views, and response times

Getting ready

How to do it...

How it works...

There's more...

Method requests, views, and response times by host

See also

Using a scatter chart to identify discrete requests by size and response time

Getting ready

How to do it...

How it works...

There's more...

Using time series data points with a scatter chart

See also

Creating an area chart of the application's functional statistics

Getting ready

How to do it...

How it works...

See also

Using a bar chart to show the average amount spent by category

Getting ready

How to do it...

How it works...

See also

Creating a line chart of item views and purchases over time

Getting ready

How to do it…

How it works...

See also

Summary

4. Building an Operational Intelligence Application

Introduction

Creating an Operational Intelligence application

Getting ready

How to do it...

How it works...

There's more...

Creating an application from another application

Downloading and installing a Splunk app

See also

Adding dashboards and reports

Getting ready

How to do it...

How it works...

There's more…

Changing the permissions of saved reports

See also

Organizing the dashboards more efficiently

Getting ready

How to do it...

How it works...

There's more…

Modifying the SimpleXML directly

See also

Dynamically drilling down on activity reports

Getting ready

How to do it...

How it works...

There's more…

Disabling the drilldown feature in tables and charts

See also

Creating a form to search web activities

Getting ready

How to do it...

How it works...

There's more...

Adding a Submit button to your form

See also

Linking web page activity reports to the form

Getting ready

How to do it...

How it works...

There's more...

Adding an overlay to the Sessions Over Time chart

See also

Displaying a geographical map of visitors

Getting ready

How to do it...

How it works...

There's more...

Adding a map panel using SimpleXML

Mapping different distributions by area

See also

Scheduling the PDF delivery of a dashboard

Getting ready

How to do it...

How it works...

See also

Summary

5. Extending Intelligence – Data Models and Pivoting

Introduction

Creating a data model for web access logs

Getting ready

How to do it...

How it works...

There's more...

Searching data models using the search interface

See also

Creating a data model for application logs

Getting ready

How to do it...

How it works...

See also

Accelerating data models

Getting ready

How to do it...

How it works...

There's more...

Viewing data model and acceleration summary information

Advanced configuration of data model acceleration

See also

Pivoting total sales transactions

Getting ready

How to do it...

How it works...

There's more...

Pivot searching using the pivot command and search interface

See also

Pivoting purchases by geographical location

Getting ready

How to do it...

How it works...

See also

Pivoting slowest responding web pages

Getting ready

How to do it...

How it works…

See also

Pivot charting top error codes

Getting ready

How to do it...

How it works...

See also

Summary

6. Diving Deeper – Advanced Searching

Introduction

Calculating the average session time on a website

Getting ready

How to do it...

How it works...

There's more...

Starts with a website visit, ends with a checkout

Defining maximum pause, span, and events in a transaction

See also

Calculating the average execution time for multi-tier web requests

Getting ready

How to do it...

How it works...

There's more…

Calculating the average execution time without using a join

See also

Displaying the maximum concurrent checkouts

Getting ready

How to do it...

How it works...

See also

Analyzing the relationship of web requests

Getting ready

How to do it...

How it works...

There's more…

Analyzing relationships of DB actions to memory utilization

See also

Predicting website-traffic volumes

Getting ready

How to do it...

How it works...

There's more…

Predicting the total number of items purchased

Predicting the average response time of function calls

See also

Finding abnormally sized web requests

Getting ready

How to do it...

How it works...

There's more...

The anomalies command

The anomalousvalues command

The cluster command

See also

Identifying potential session spoofing

Getting ready

How to do it...

How it works...

There's more...

Creating logic for urgency

See also

Summary

7. Enriching Data – Lookups and Workflows

Introduction

Lookups

Looking up product code descriptions

Getting ready

How to do it...

How it works...

There's more...

Manually adding the lookup to Splunk

See also

Flagging suspicious IP addresses

Getting ready

How to do it...

How it works...

There's more...

Modifying an existing saved search to populate a lookup table

See also

Creating a session state table

Getting ready

How to do it...

How it works...

See also

Adding hostnames to IP addresses

Getting ready

How to do it...

How it works...

There's more…

Enabling automatic external field lookups

See also

Searching ARIN for a given IP address

Getting ready

How to do it...

How it works...

There's more...

Limiting workflow actions by event types

See also

Triggering a Google search for a given error

Getting ready

How to do it...

How it works...

There's more...

Triggering a Google search from the chart drilldown options

See also

Creating a ticket for application errors

Getting ready

How to do it...

How it works...

There's more...

Adding a workflow action manually in Splunk

See also

Looking up inventory from an external database

Getting ready

How to do it…

How it works...

There's more...

Use DB Connect for direct external DB lookups

See also

Summary

8. Being Proactive – Creating Alerts

Introduction

Alerting on abnormal web page response times

Getting ready

How to do it...

How it works...

There's more...

Viewing triggered alerts in Splunk's Alert manager

See also

Alerting on errors during checkout in real time

Getting ready

How to do it...

How it works...

There's more...

Building alerts via a configuration file

Identify the real-time searches that are running

See also

Alerting on abnormal user behavior

Getting ready

How to do it...

How it works...

There's more...

Alerting on abnormal user purchases without checkouts

See also

Alerting on failure and triggering a scripted response

Getting ready

How to do it...

How it works...

There's more…

See also

Alerting when predicted sales exceed inventory

Getting ready

How to do it...

How it works...

There's more…

Adding an RSS feed notification action to an alert

See also

Summary

9. Speed Up Intelligence – Data Summarization

Introduction

Calculating an hourly count of sessions versus completed transactions

Getting ready

How to do it...

How it works...

There's more...

Generating the summary more frequently

Avoiding summary index overlaps and gaps

See also

Backfilling the number of purchases by city

Getting ready

How to do it...

How it works...

There's more...

Backfilling a summary index from within a search directly

See also

Displaying the maximum number of concurrent sessions over time

Getting ready

How to do it...

How it works...

There's more...

Viewing the status of an accelerated report

See also

Summary

10. Above and Beyond – Customization, Web Framework, REST API, and SDKs

Introduction

Customizing the application's navigation

Getting ready

How to do it...

How it works...

There's more…

Adding a force-directed graph of web hits

Getting ready

How to do it...

How it works...

There's more…

Changing the time range on the search manager

See also

Adding a calendar heatmap of product purchases

Getting ready

How to do it...

How it works...

See also

Remotely querying Splunk's REST API for unique page views

Getting ready

How to do it...

How it works...

There's more…

Authenticating with a session token

See also

Creating a Python application to return unique IP addresses

Getting ready

How to do it...

How it works...

There's more...

Paginating the results of your search

See also

Creating a custom search command to format product names

Getting ready

How to do it...

How it works...

See also

Summary

Index

Splunk Operational Intelligence Cookbook

Splunk Operational Intelligence Cookbook

Copyright © 2014 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: October 2014

Production reference: 1241014

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-84969-784-2

www.packtpub.com

Cover image by Paul R Johnson (<paul@discoveredintelligence.ca>)

Credits

Authors

Josh Diakun

Paul R Johnson

Derek Mock

Reviewers

Mika Borner

Amit Mund

Jon Webster

Commissioning Editor

Kartikey Pandey

Acquisition Editor

Rebecca Youé

Content Development Editor

Anila Vincent

Technical Editor

Veronica Fernandes

Copy Editors

Janbal Dharmaraj

Sayanee Mukherjee

Karuna Narayanan

Project Coordinator

Neha Bhatnagar

Proofreaders

Simran Bhogal

Mario Cecere

Bernadette Watkins

Indexer

Monica Ajmera Mehta

Production Coordinators

Kyle Albuquerque

Arvindkumar Gupta

Conidon Miranda

Alwin Roy

Cover Work

Conidon Miranda

About the Authors

Josh Diakun is an IT operations and security specialist with a focus on creating data-driven operational processes. He has over 10 years of experience in managing and architecting enterprise grade IT environments. For the past 5 years, he was managing a Splunk deployment that saw Splunk used as the platform for security and operational intelligence. Most recently, Josh has partnered in setting up a business venture, Discovered Intelligence, which provides data intelligence solutions and services to the marketplace. He is also a cofounder of the Splunk Toronto User Group.

I would first like to thank my co-authors, Derek Mock and Paul R Johnson, for their support, endless efforts, and those many late nights that led to this book becoming a reality. To my partner, Rachel—an endless thank you for being my biggest supporter and making sure I always remembered to take a break. To my mother, Denyce, and sister, Jessika—thank you for being the two most amazing people in my life and cheering me on as I wrote this book. Finally, to my late father, John, who was always an inspiration and brought the best out of me; without him, I would not be where I am today.

Paul R Johnson has over 10 years of data intelligence experience in the areas of information security, operations, and compliance. He is a partner at Discovered Intelligence—a company that specializes in data intelligence services and solutions. He previously worked for a Fortune 10 company, leading IT risk intelligence initiatives and managing a global Splunk deployment. Paul cofounded the Splunk Toronto User Group and lives and works in Toronto, Canada.

I would like to thank my fellow authors, Josh Diakun and Derek Mock, for their support and collaborative efforts in writing this book. Thanks guys for giving up nights, days, and weekends to get it completed! I would also like to thank my wife, Stacey, for her continuous support, for keeping me focused, and for her great feedback and patience.

Derek Mock is a software developer and architect, specializing in unified communications and cloud technologies. Derek has over 15 years of experience in developing and operating large enterprise-grade deployments and SaaS applications. For the past 4 years, he has been leveraging Splunk as the core tool to deliver key operational intelligence. Derek is a cofounder of the Splunk Toronto User Group and lives and works in Toronto, Canada.

I could not have asked for better co-authors than Josh Diakun and Paul R Johnson, whose tireless efforts over many late nights brought this book into being. I would also like to thank my mentor, Dave Penny, for all his support in my professional life. Finally, thanks to my partner, Alison, and my children, Sarah and James, for cheering me on as I wrote this book and for always making sure I had enough coffee.

About the Reviewers

Mika Borner is a management consultant for data analytics at LC Systems based in Switzerland, Germany, and Austria.

Drawing on his years of experience, he provides Splunk consulting in the telecommunications/ISP, financial, retail, and other industries. During the course of his career, he has held numerous positions in systems engineering in IT, with service providers, telecommunications/ISP companies, and financial institutions.

Mika was one of the first Splunk users in Europe and was later running one of the largest Splunk environments worldwide. He is also a regular speaker at the Splunk User Conference.

Amit Mund has been working on Linux and other technologies on automation and infrastructure monitoring since 2004. He is currently associated with Akamai Technologies and has previously worked for the website-hosting teams at Amazon and Yahoo!.

I would like to thank my wife, Rajashree, for always supporting me and my colleagues for helping me in my learning and development throughout my professional career.

Jon Webster has been fascinated with computers since he met his first mainframe at Hewlett-Packard at the age of 11 and played chess and Qubic on it.

In his roles from an ERP Developer through APM Product Manager and Splunk Architect, Jon has always sought to apply the maximum leverage that technology offers for his customers' benefit.

I'd like to thank my parents for encouraging me to explore these strange things they didn't understand, David Kleber and Kennon Ward for helping me learn how to optimize my code and my career, PeopleSoft for the amazing playgrounds and opportunities, Alan Habib for dragging me into APM (just attend one meeting!), and finally, Splunk for the most amazing people, tools, and opportunities I've ever had the pleasure of working with. The Aha! moments keep coming!

www.PacktPub.com

Support files, eBooks, discount offers, and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Preface

In a technology-centric world, where machines generate a vast amount of data at an incredibly high volume, Splunk has come up with its industry-leading big data intelligence platform—Splunk Enterprise. This powerful platform enables anyone to turn machine data into actionable and very valuable intelligence.

Splunk Operational Intelligence Cookbook is a collection of recipes that aim to provide you, the reader, with the guidance and practical knowledge to harness the endless features of Splunk Enterprise 6 for the purpose of deriving extremely powerful and valuable operational intelligence from your data.

Using easy-to-follow, step-by-step recipes, this book will teach you how to effectively gather, analyze, and create a report on the operational data available in your environment. The recipes provided will demonstrate methods to expedite the delivery of intelligent reports and empower you to present data in a meaningful way through dashboards and by applying many of the visualizations available in Splunk Enterprise. By the end of this book, you will have built a powerful Operational Intelligence application and applied many of the key features found in the Splunk Enterprise platform.

This book and its easy-to-follow recipes can also be extended to act as a teaching tool for you as you introduce others to the Splunk Enterprise platform and to your new found ability to provide promotion-worthy operational intelligence.