Searching in Splunk

It would be negligent for a book on mastering Splunk searching to not mention the dashboard of version 6.0.

The search dashboard

If you take a look at the Splunk search dashboard (and you should), you can break it down into four general areas. They are given as follows:

• The search bar: The search bar is a long textbox into which you can enter your searches when you use Splunk Web.
• Range picker: Using the (time) range picker, you can set the period over which to apply your search. You are provided with a good supply of preset time ranges that you can select from, but you can also enter a custom time range.
• How-To (panel): This is a Splunk panel that contains links that you can use to access the Search Tutorial and Search Manual pages.
• What-To (panel): This is another Splunk panel that displays a summary of the data that is installed on the current Splunk instance.

The new search dashboard

After you run a new search, you're taken to the New Search page. The search bar and time range picker are still available in this view, but the dashboard updates many more elements, including search action buttons, a search mode selector, counts of events, a job status bar, and the results tabs for events, statistics, and visualizations.

The Splunk search mechanism

All searches in Splunk take advantage of the indexes that are set up on the data that you are searching. Indexes exist in every database, and Splunk is not an exception. Database indexes and Splunk indexes might differ physically, but in concept, they are the same—both are used to optimize performance. Splunk's indexes organize words or phrases in the data over time. Successful Splunk searches (those that yield results) return records (events) that meet your search criteria. The more matches you find in your data, the more events returned by Splunk. This will impact the overall searching performance, so it is important to be as specific in your searches as you can.

Before we start, the following are a few things that you need to keep in mind:

• Search terms are case insensitive
• Search terms are additive
• Only the specified time frame is queried

The Splunk quick reference guide

To all of us future Splunk masters, Splunk has a Splunk Language Quick Reference Guide (updated for version 6.0) available for download in the PDF format from the company's website at http://www.splunk.com/web_assets/pdfs/secure/Splunk_Quick_Reference_Guide.pdf. I recommend that you take a look.

Please assist me, let me go

To master Splunk, you need to master Splunk's search language, which includes an almost endless array of commands, arguments, and functions. To help you with this, Splunk offers a search assistant.

The Splunk searching assistant uses typeahead to suggest search commands and arguments as you type into the search bar. These suggestions are based on the content of the datasource you are searching and are updated as you continue to type. In addition, the search assistant will also display the number of matches for the search term, giving you an idea of how many search results Splunk will return.

The screenshot in the next section shows the Splunk search assistant in action. I've typed TM1 into the search bar, and Splunk has displayed every occurrence of these letters that it found within my datasource (various Cognos TM1 server logs) along with the hit count.

Some information for future reference: the search assistant uses Python to perform a reverse URL lookup in order to return the description and syntax information as you type.

Note

You can control the behavior of the search assistant with UI settings in the SearchBar module, but it is recommended that (if possible) you keep the default settings and use the search assistant as a reference. Keep in mind that this assistance might impact the performance in some environments (typically in those environments that include excessive volumes of raw data).

Basic optimization

Searching in Splunk can be done from Splunk Web, the command-line interface (CLI), or the REST API. When you are searching using the web interface, you can (and should) optimize the search by setting the search mode (fast, verbose, or smart). The search mode selector is in the upper right-hand corner of the search bar. The available modes are smart (default), fast, and verbose. This is shown in the following screenshot:

Depending on the search mode, Splunk automatically discovers and extracts fields other than the default fields, returns results as an events list or table, and runs the calculations required to generate the event timeline. This "additional work" can affect the performance; therefore, the recommended approach will be to utilize Splunk's fast mode during which you can conduct your initial search discovery (with the help of the search assistant), after which you can move to either the verbose or the smart mode (depending on specific requirements and the outcome of your search discovery).

Fast, verbose, or smart?

Splunk adjusts the search method it uses based on the selected search mode. At a high-level, the fast mode is, as the name suggests, fast (typically, the fastest method) because it tells Splunk to disable field discovery and just use its default fields, while the verbose mode will take the time to discover all the fields it can. The smart mode will take an approach (enable or disable field discovery) based on the search command's specifics.

The breakdown of commands

Some Splunk search processing language (SPL) searching commands have specific functions, arguments, and clauses associated with them. These specify how your search commands will act on search results and/or which fields they act on. In addition, search commands fall into one of the three forms, as follows:

• Streaming: Streaming commands apply a transformation to each event returned by a search. For example, when the regex command is streaming, it extracts fields and adds them to events at search time.
• Reporting: Reporting commands transform the search result's data into the data structures required for visualizations such as columns, bars, lines, area, and pie charts.
• Nonstreaming: Nonstreaming commands analyze the entire set of data available at a given level, and then derive the search result's output from that set.

Understanding the difference between sparse and dense

Always consider what you are asking Splunk to do. Based on your search objectives, you need to consider whether what you are searching for is sparse or dense. Searches that attempt to analyze large volumes of data with the expectation of yielding a few events or a single event are considered to be sparse searches. Searches that intend to summarize many occurrences of events are dense searches.

With an understanding of what type of search you are interested in performing and how Splunk will process the search, you can consider various means of knowledgeable optimizations (KO).

Knowledgeable optimizations can include a simple recoding of the search pipeline (the structure of a Splunk search in which consecutive commands are chained together using a pipe character [|]), using more relevant search commands or operators, applying simplified logic, or configuring Splunk to recognize certain key information that you identify within your search results as you index new data.

Searching for operators, command formats, and tags

Every Splunk search will begin with search terms. These are keywords, phrases, Boolean expressions, key-value pairs, and so on, that specify which events you want to retrieve with the search.

Splunk commands can be stacked by delimiting them with the pipe character (|). When you stack commands in this way, Splunk will use the results (or the output) of the command on the left as an input to the command on the right, further filtering or refining the final result.

A simple example of command stacking might be to use commands in order to further filter retrieved events, unwanted information, extract additional event information, evaluate new fields, calculate statistics, sort results, or create a visualization (such as a chart).

The search example in the next section examines Cognos TM1 server log files for the phrase Shutdown in an attempt to determine how many times the TM1 server was shut down. Next, I've added a search field to only see the matching events that occurred (so far) in the year 2014. Finally, I want to produce a visualization of the results (to show on which days the server was shut down and how many times), so I stack the search command using the pipe delimiter to feed the results of my search into the Splunk chart command (along with the arguments I need to create a "count by day" chart).

The process flow

Before creating more complex Splunk queries or attempting any knowledgeable optimizations, it is important to understand the separate steps that occur when Splunk processes your search command pipeline. The concept of separate steps (rather than a single large query) allows Splunk to be efficient in processing your request, in much the same way as separate, smaller-sized SQL queries are more efficient than one large complicated query.

Consider the following search query example:

Shutdown date_year=2014 | chart count by date_mday 

The following process will occur:

• All the indexed data (for this installation of Splunk and which version the user is configured for) is used as an input for the Splunk search
• An intermediate result table is created, containing all the events in the data that matched the search criteria (the term Shutdown is found in an event that occurs in the year 2014)
• The intermediate result table is then read into the chart command, and a visualization is created by summarizing the matching events into a count by day

Boolean expressions

The Boolean data type (a data type with only two possible values: true and false) is supported within Splunk search. The following operators are currently supported:

• AND
• OR
• NOT

Splunk Boolean searches can be simple or compound, meaning that you can have a single Boolean expression, such as the following:

Shutdown OR Closing

You can also have a compound Boolean expression, such as the following:

(shutdown OR Closing) AND (date_mday=3 OR date_mday=4)

Splunk, like any programming language, evaluates Boolean expressions using a predetermined precedence. What this means is that your Splunk search will be evaluated as follows:

1. Evaluate the expressions within the parentheses.
2. Evaluate the OR clauses.
3. Evaluate the AND or NOT clauses.

As a Splunk master, the following are some key points that you need to remember when designing your Splunk searches:

• All Boolean operators must be capitalized (or Splunk will not evaluate them as an operator)
• The AND operator is always implied between terms, that is, shutdownclosing is the same as shutdown AND closing
• You should always use parentheses to group your Boolean expressions (this helps with readability, among other things)
• Do not write searches based on exclusion, rather strive for inclusion (error instead of NOT successful)

You can quote me, I'm escaping

All but the simplest search commands will include white spaces, commas, pipes, quotes, and/or brackets. In addition, in most use cases, you won't want to search for the actual meaning of Splunk keywords and phrases.

To make sure that Splunk interprets your search pipelines correctly, you will need to use quotes and escapes.

Generally, you should always use quotes to ensure that your searches are interpreted correctly, both by Splunk as well as by other readers. Keep in mind that the following Splunk searches are completely different searches:

Server shutdown
"Server shutdown"

In the first search, Splunk implies the Boolean operator AND, so events with the words Server and shutdown in them will be returned. In the second search, only those events that have an occurrence of the phrase Server shutdown will be returned, obviously yielding potentially different results.

Furthermore, if you do want to search for events that contain the actual (raw) values of Splunk keywords or operators, you'll need to wrap the events in quotes. The rules for using quotes in Splunk search pipelines are given as follows:

• Use quotes around phrases and field values that include white spaces, commas, pipes, quotes, and/or brackets
• Quotes must be balanced (an opening quote must be followed by an unescaped closing quote)
• Use quotes around keywords and phrases if you don't want to search for their default meaning, such as Boolean operators and field-value pairs

As the quote character is used to correctly qualify your search logic, this makes it difficult to search for the actual value of a quote. To resolve this issue, you need to use the backslash character (\) to create an escape sequence.

The backslash character (\) can be used to escape quotes, pipes, and itself. Backslash escape sequences are still expanded inside quotes.

Consider the following examples:

• The sequence \| as part of a search will send a pipe character to the command, instead of having the pipe split between commands
• The sequence \" will send a literal quote to the command, for example, searching for a literal quotation mark or inserting a literal quotation mark into a field using the rex command
• The \\ sequence will be available as a literal backslash in the command

A simple example would be if you wanted to look for events that actually contain a quote character. If you use the simple search of a single quote or even wrap a quote within quotes, you will receive a syntax error.

If you use a backslash to escape the quote, you'll get better results.

One more thing to make a note of: asterisks, *, cannot be searched for using a backslash to escape the character. Splunk treats the asterisk character as a major breaker (more on this later).

Tag me Splunk!

Based on the search pipeline you construct, Splunk will effectively dissect and search through all of its indexed data. Having said that, there might be occasions where you would want to add additional intelligence to the searching—things that you know, but Splunk might not. This might be relevant information about how your organization is structured or a specific way in which you use areas of data. Examples might be host names or server names. Instead of requiring your users to retype this information (into the search pipeline each time), you can create a knowledge object in the form of a Splunk search tag.

Assigning a search tag

To assist your users (hopefully, to make searching more effective) with particular groups of event data, you can assign tags (one or multiple) to any field/value combinations (including eventtype, host, source, or sourcetype) and then perform your searches based on those tags.

Tagging field-value pairs

Let's take a look at an example. Wherever I go, it seems that Cognos TM1 servers are logging message data. This (machine-generated) data can be monitored and inputted by a Splunk server, where it will be indexed and made available for searching. This data is made up of logs generated from multiple Cognos TM1 admin servers and indexed by a single Splunk server.

If I wanted to have the capacity to search an individual server source, without having to qualify it in each of my searches, I could create a tag for this server.

So, in a typical search result (using Splunk Web), you can locate an event (that has the field value pair that you want to tag) and then perform the following steps:

1. First, locate the arrow graphic (next to the event) and click on it.
2. Again, locate the arrow graphic, this time under Actions, and click on it (next to your field value).
3. Select Edit Tags.
4. Now, you can construct your tag and click on Save (to actually add this tag).

In this example, a tag named TM1-2 was created to specify an individual Cognos TM1 server source. Now, in the future, this tag can be used to narrow down searches and separate events that occurred only in that server log.

The syntax to narrow down a search (as shown in the preceding example) is as follows:

tag=<tagname>

Taking this a bit further, you can narrow down a search by associating a tag with a specific field using the following syntax:

tag::<field>=<tagname>

Wild tags!

You, as a Splunk master, can use the asterisk (*) as a wildcard when you are searching using Splunk tags. For example, if you have multiple sourcetype tags for various types of TM1 servers, such as TM1-1 all the way through TM1-99, you have the ability to search for all of them simply using the following code:

tag::eventtype=TM1-*

What if you wanted to locate all the hosts whose tags contain 44? No problem, you can search for the tag as follows:

tag::host=*44*

Although you'll find the following example in several places in the Splunk documentation, I have yet to find a way to use it. If you want to search for all the events with event types that have no tags associated with them, you can search for the Boolean expression as follows:

NOT tag::eventtype=*

Wildcards – generally speaking

Yes, Splunk does support wildcards, and this extends the flexibility of your search efforts. It is, however, vital to recognize that the more flexible (or less specific) your Splunk searches are, the less efficient they will be. Proceed with caution when implementing wildcards within your searches (especially complex or clever ones).

Disabling and deleting tags

Once you have established a tag, you can manage it—delete it or disable it—by going to Settings and then selecting Tags.

From there, you can select all unique tag objects to view your tags (some tags might not be public). Finally, from there, you can change the status (to disable) or select the action to be deleted.

Transactional searching

transaction [<field-list>] [name=<transaction-name>] <txn_definition-opt>* <memcontrol-opt>* <rendering-opt>*