Using Splunk 6 Lab Exercises

Lab typographical conventions

{student name} indicates you should replace this with your student user name.
{server-name} indicates you should substitute the server name assigned to this class.

Lab Exercise 1 Log into Splunk

Description
Learn how to log into Splunk and review the data summary window.

Steps
Task:

Log into Splunk on the classroom server.

1.

Direct your web browser to the class lab system, for example:
http://{server-name}.splunk.com

2.

Log in with the credentials assigned by your instructor.

3.

On the Home view, select Search under the Search & Reporting app box.

4.

Take a moment to examine the How to Search and What to Search sections.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Lab Exercise 2 Customize your User Settings

Description
Customize your Splunk user account settings.

Steps
Task:

Explore the basic Splunk navigation.

1.

Task:

Explore some of the menu items of interest to you to familiarize yourself with Splunk navigation.

Change your account settings to reflect your name and local time zone.
2.

Click your user name next to the Messages menu option in the top right corner.

3.

Click Edit Account.

4.

In the Full Name field, modify the existing name and enter your name.

5.

From the Time zone menu, select your local time zone.

6.

Under Default app, select search and then click Save.

7.

Next to the Splunk logo in the upper left, click Apps.

8.

Click Search & Reporting.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Lab Exercise 3 Run Basic Searches

Description
Use the Search app to investigate failed login attempts.

Steps
Task:

Task:

Perform a basic search.

1.

Return to the Search & Reporting app, if you are not already there.

2.

Select Search & Reporting from the App menu in the top left of the main navigation bar, also called
the Splunk bar.

3.

In the search bar, type the search: error OR fail*

NOTE: As you type, the Search Assistant provides suggestions. If you were to press the enter key,
the search would begin with the default time range of All time (a very large, slow search).

4.

Use the time range picker to set the time range to Last 24 hours (located in the Relative section).
When you select a time range, the search begins as if you had pressed the enter key.

5.

Mouse over search results and notice that your search terms are highlighted and that you could page
through to see more results.

Narrow your results.

6.

Search for password fail* over the Week to Date.

We are only interested in events with the sourcetype linux_secure.
You should see some events with sourcetype=linux_secure. (You may need to page through
the search results to find a matching event.) There also see several events with sourcetype=ps.

Task:

7.

Use the NOT Boolean to remove the ps events. Add NOT sourcetype=ps to your search string.
NOTE: Your search should now be password fail* NOT sourcetype=ps

8.

Click the Search button or press Enter to run the search.

9.

Page through the results. There are many login failures.

NOTE: Above the events there is a menu item that allows you to change the number of events that
display on a page. It is usually set to 20 Per Page but you can click the down arrow next to
it to increase or decrease that number.

Use the timeline to look for patterns in the results.

10. Look at the pattern of these events. Do these events happen routinely over time?
11. Drill down on one of the "spikes" of events by clicking the bar in the timeline. Look at these events.
12. Drill down on another spike. Do the events look similar? It looks like your system may be the target of
an attack.
13. Click Deselect above the time line to see all the events again.

Task:

Use the output of your search to refine the results.

14. Pick one of the user names in the search results and click the name. Notice what is added to the
search bar. Also, note that the user name is now highlighted in each event.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Task:

Save and share results. (Extend the default save time and expand default viewing permissions to all.)
15. In the Search Bar, from the Job menu, select Edit Job Settings.
16. Change the Read Permissions of the job. The default is Private. Click Everyone. For important
searches, this allows others to leverage your work. Extend the Lifetime of your search. The default is
10 minutes. Click 7 days.
17. Click Save.
18. To retrieve your search, from the Activity menu, click Jobs and find your search in the list. (It is found
at the top of your browser view.)
19. Delete all of your displayed jobs except the one that you changed to 7 days.
20. Return to the Search view.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Lab Exercise 4 Using Fields and Tags

Description
You now continue investigating failed login attempts and then investigate customer purchase patterns. You explore
how fields can help you with your investigations.

Steps
Task:

Task:

Demonstrate that Splunk fields are related to search results.

1.

Scroll up and click Search in the navigation menu to clear the previous search.

2.

Search for sourcetype=sales_entries over the Last 4 hours.

3.

Examine the Fields sidebar. There are some selected fields and a number of interesting fields. How
many fields are not displayed in the fields sidebar? ____
HINT:
Look at the bottom of the fields sidebar.

4.

Search for the sourcetype=access_combined. How many fields are not displayed in the fields
sidebar? _____ Notice the difference in the names of fields associated with the search results.

Examine search modes.

5.

Notice that the search, by default, ran in Smart mode. Record the number of Selected fields and
Fields not displayed.
Smart mode:
_____ Selected fields _____ Fields not displayed

6.

Rerun the same search in Fast mode and record the results.
Fast mode:

7.

_____ Selected fields _____ Fields not displayed

Re-run the same search in Verbose mode and record the results.
Verbose mode: _____ Selected fields _____ Fields not displayed
NOTE: The difference in the speed of queries might not be noticeable on your lab system. However,
you will see the impact the search modes have on reports later today.

Task:

Use fields to examine search results.

8.

Use Smart mode to search for action=purchase. Keep the time range at the Last 4 hours.
Examine the Fields sidebar Interesting Fields list. Notice that product_name is one of the fields
extracted by Splunk. Click product_name in the Fields sidebar. Notice the pop-up window shows the
top ten best selling products.

9.

Another interesting field is price. This field tells you the prices at which most purchases occur. In the
Fields sidebar, click price.
NOTE: You may need to open the Fields window to find and select it.

10. In order to quickly see values of the price field in your events, click Yes in the upper right corner next
to Selected and close the window. Notice price is now a selected field in the Fields sidebar and is
displayed below each event that contains the field.
11. Back in the Fields sidebar, examine the most frequently occurring value for the price field under
Selected Fields. Click the price field and in the window that opens, click the value with the highest
number of purchases (listed at the top). Notice the field and value have been added to the search bar.
Also, this selection causes a new search to be executed using the new search criteria.
12. Remove the price field from the search and re-run the search.
13. In the Fields Sidebar, click categoryId to see which types of games account for the most
purchases.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Scenario: As a seasoned Splunk power user, you are going to build some knowledge into your Splunk environment.
Hosts named www1, www2, and www3 serve an external e-commerce store in the DMZ. The web team is
specifically responsible for the store hosts. Two teams are interested in these servers, the DMZ team and
web team.
Task:

Create tags to identify hosts.

14. Run a search over the Last 24 hours that returns events from all hosts with names that begin with
www.
Hint: Use the wildcard (*).
15. In the Fields sidebar, click the host field and note all the hosts that are returned from the search. You
should see three servers that begin with www.
16. Run a search for host=www1.
17. Click the arrow under the

icon in the first event.

18. Find the row for the host field. Click the down arrow under the Actions column and select Edit
Tags.
19. Tag host www1 with the values dmz and webteam.
20. Repeat steps 16-19, but change the search to host=www2.
21. Repeat steps 16-19 again, this time using the search host=www3.
Task:

Use tags in a search.

22. Search for sourcetype=linux_secure over the last 24 hours. Use the fields sidebar to determine
how many hosts are returned from your search.
23. Modify the search to limit results to only hosts in the dmz.
HINT: tag=dmz. Also note that tags are case sensitive. A search for tag=DMZ produces no results.
24. Note the difference in the number of hosts in the returned results.

Task:

View your recent searches using the Jobs page.

25. Click Activity in the Splunk bar, then click Jobs.
26. Take a look at your recent searches (jobs) to make sure there arent any errors in search structure.
Some jobs may still be running. If you already have enough data, you can Finalize them to stop the
search job. You can inspect any job to verify its contents are what you intended.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Lab Exercise 5 - Creating Alerts

Description
You learn to create an alert.
(NOTE TO INSTRUCTORS: Please rerun the live_datagen script before beginning this lab.)

Steps
Scenario: For security reasons, you need to monitor failed login attempts into our servers in the DMZ. We are only
interested in failed logins from known user accounts.
Task:

Create a search to identify specific types of failed logins.

1.

Task:

Using tags, search for the Linux secure logs on all web servers in the Last 60 minutes.
HINT: Search for tag=dmz sourcetype=linux_secure

Add keywords to identify only these types of events.

2.

Add the keywords failed AND password NOT invalid. Re-run the search.

Scenario: This search identifies login attempts to existing user accounts on the servers. You need to track these
because they can be more dangerous than unknown users. To gain access, attackers need a user name
and a password. With a valid user name, they are partially there! Create an alert that triggers when there
are more than one failed login attempts within one minute.
Task:

Create and view and alert.

3.

From the Save As menu, select Alert.

4.

Name the alert {student name} - Login Attempts.

5.

Click the Real Time button.

6.

Next to Trigger condition, click the drop down button that says Per-Result and select Number of
Results.

7.

The Number of Results is button is already set to Greater than. In the field next to it is already set
where you want it, at 0. (This setting triggers the alert for every failed login.)

8.

The in field is already set to 1 minute.

9.

Click Next.

10. Select List in Triggered Alerts.

11. Set the Severity to High.
12. For When triggered, execute actions, select For each result.
13. Select the Throttle checkbox. The default is already set to Suppress triggering for 60 second(s),
14. Suppress results that share the same host field value.
15. For Permissions, select Shared in App.
16. Click Save.
17. Click the Permissions link to examine details about the permissions you set.
18. Click Cancel.
19. You should see an overview screen describing your new alert.
20. From the Splunk bar, click Activity and click Triggered Alerts.
21. Select your student ID from the Owner menu and view the triggered alerts.
NOTE:
It may take a few minutes for your alert to trigger.
22. Click the View results link on a triggered alert to view the event(s) that caused the alert.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Task:

Disable the alert.

23. In the App Navigation bar, click Alerts.
24. Click Edit and click Disable.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Lab Exercise 6 Reporting and Visualizations

Description
First, you will save a search as a report. Then, you explore the differences between search modes for reporting. You
then use the reports you created to build dashboards.
Task:

Task:

Save a search as a report.

1.

Return to the Search & Reporting app.

HINT: You can use the Apps menu in the top-left nav bar or click the Splunk logo in the upper left
since you set your default app to be Search and Reporting.

2.

Search for password fail* root NOT sourcetype=ps over the last 24 hours.

3.

From the Save As menu, select Report.

4.

Name the report {user name} Failed Logins for Root Last 24 hours

5.

Select No for the time range picker option, then click Save.

6.

Click View to view the report.

Explore the impact of search mode on reporting.

7.

From the Edit menu, select Open in Search. Explore search modes and visualizations.
NOTE: When you run a saved report, it runs in Smart Mode.

8.

In the Fields sidebar, click the host field and select the report type: Top values by time and click
Save.
Notice In the Events tab that the timeline and fields sidebar do not display. You also see an error
message notifying you that your search did not return any events because you are in Smart Mode.
Since the search string includes the tImechart command, you must change search modes to see
events.

9.

Change the search mode to Fast and re-run the search.

10. Select the Events tab. Neither Smart nor Fast mode return events in the events tab when a
reporting command is present.
11. Change the search mode to Verbose and re-run the search. Switch to the Events tab.
NOTE: Now in the Events tab, you see the timeline and fields sidebar.
Task:

Create a report using the Fields sidebar, view it in statistics and visualization tabs, and save it as a
dashboard.
12. Search for status>=400 AND status<=600 (action=purchase OR action=addtocart) in
Smart mode over the Last 7 days.
13. Click the host field in the fields sidebar, then select the chart Top values by time. A timechart
displays in the Visualization tab.
14. Click the Statistics tab to see another view of your results.
15. Click the Visualization tab to return to the timechart.
16. From the Save As menu, select Dashboard panel.
17. In the Dashboard Title field, enter a name for the entire dashboard {student name} Ops
Dashboard.
18. Select Shared in App.
19. In the Panel Title field, enter a name for your panel: Incomplete Sales - Previous 7 Days and click
Save.
20. In the confirmation dialog, click View Dashboard to display the dashboard you created.
21. Click the Edit button and click Edit Panels.
22. In the dashboard panel, click the middle of the three upper right dropdowns. Click Done.

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

Task:

Add a panel to the dashboard.

23. From the Edit menu, select Edit Panels.
24. Click the +Add Panel button.
25. In the Content Title field, type a name the new panel: Failed Logins for Root Last 24 hours
rd

26. Select the Report Content Type icon. This is the 3 icon in the row.
27. Click the report title to display the reports you can use in this panel. Select your Failed Logins for
Root report from the list.
28. Click Add Panel to add the panel to the dashboard.
29. Drag the Failed Logins for Root panel and position it to the right of the top panel. The panels should
display side-by-side.
30. Change the visualization to Line.
31. In the upper right, click Done to save your changes. Your dashboard may look something like this:

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

10

Lab Exercise 7 Pivot

Description
Now you can build a pivot table to track customer failed requests, a possible cause of lost revenue, and save it as a
report.
Task:

Create a pivot from an existing data model and save it as a dashboard panel.
1.

In the App Navigation Bar, click Pivot.

2.

Click Buttercup Games Online Sales.

NOTE: For each object listed, you can click the arrow to show/hide its constraints and the attributes
associated with it.

3.

Select the object: failed request. The Pivot interface opens with a count of failed requests.
NOTE: These are events where the http status returned was an error code.

4.

Change the Time filter from All Time to Week to date.

5.

From the Split Rows selector, add action as a Split Rows field. Give the action field a label of
Customer Action. Keep the defaults, then click Add to Table.

6.

From the Split Columns selector, add the host field. Keep the defaults, then click Add to Table.

7.

From the visualization selector along the left, select Bar Chart.

8.

Filter the report to exclude accessories. From the Filter section, select Add Filter.

9.

Select the category field.

10. From the Match menu, choose is not, then select ACCESSORIES.
11. Save the pivot as a Dashboard panel.
12. From the Dashboard selector, choose Existing, then select your dashboard.
13. Name the panel Errors on Customer Action Games Only Week to Date.
14. Click Save, and then click View Dashboard to view your dashboard.
15. From the Edit menu, select Edit Panels.
16. Drag the new panel to the top right, then move the Failed Logins panel to the bottom so that it spans
both top panels.
17. Click Done and admire your work! Your pivot may look something like this:

2013 Splunk Inc. All rights reserved.

Using Splunk 6

March 17, 2014

11