#SPLUNK COMMANDS SERIES#

#SPLUNK COMMANDS SERIES#

sortby : A series of sort options exist, which are excellent if your dedup takes place at the end of the command.Bonus points if you can tell me why this exists. The fields for lang and group are not present for those events. Note how each event has lang and group fields under the events. Then add lang and group to the selected fields. This option is weird enough to try: | windbag | eval group = (random() % 2) + 1 | dedup keepevents=true lang, group keepevents : Keep all events, but remove the selected fields from events after the field event containing that particular combination.The problem this solves may be easier to rectify using fillnull, filldown, or autoregress. keepempty : Allows keeping events where one or more fields have a null value.

By default, it’s false, but you can probably see how it’s helpful to trim repeating values. consecutive : This argument only removes events with duplicate combinations of values that are consecutive.Note: It may be better to use other SPL commands to meet these requirements, and often dedup works with additional SPL commands to create combinations. There are several options available for dedup that affect how it operates. In case anyone is wondering, ~80 of that data is the component Metrics (apparently, we need to use this cloud stack more) Other dedup Comand Options and Considerations Result: Returned 500 events, then stats counted. Stats counts based upon fields in the data? Why not? index=_internal | dedup 100 host | stats count by component | sort - count What can we do with this? Anything you can picture in SPL. Great, so we can reduce our count of events. Result: Each of the twenty-five lang entries returned two events. This time we’ll add an integer behind dedup to give us more results per search. Step 5: What if we want more than one event per field? It ranged from seventy-five events to eighty-six in the ten runs I let it try. | windbag | eval group = (random() % 2) + 1 | bin span=12h _time | dedup lang, _time, group

This time, we’re going to randomly assign (using random and modulo arithmetic) each event a 1 or 2 for the group, and then use that in a dedup along with the span of 12 hours. The above exercise was one way to divide the data up. Step 4: Add a random 1 or 2 to the mix, and dedup off of those three fields. Twenty-five different lang fields, with the highest event count at 3. | windbag | bin span=12h _time | dedup lang, _time Since I ran this at 21:45, I wound up with four buckets (Who said this was perfect?), with the middle two buckets having forty-two events each. Using bin like this is one way to split the data. Taking advantage of this, we can create another usable field by using bin to set the time into 12-hour buckets. The windbag data is spread out over the past 24 hours (because I’m running 24-hour time). Step 3: Cast time into a bin, then reduce fields with lang and time bin We can also reduce by a combination of fields and even create fields before using dedup. Lang still has 25 unique values, but there is only one event for each language specified this time. Now, let’s limit that to 1 event for each of those values in lang. Step 2: Using Dedup to reduce events returned Twenty-five unique values for the field lang, with the highest value having eight events. I will use the windbag command for these examples since it creates a usable dataset (windbag exists to test UTF-8 in Splunk, but I’ve also found it helpful in debugging data). Let’s run through an example scenario and explore options and alternatives. If the dedup runs in real-time, the first events received are searched, which does not guarantee that they are the most recent (data doesn’t always arrive in a tidy order). When run as a historic search (e.g., against past data), the most recent events are searched first. Outputting events is useful when you want to see the results of several fields or the raw data, but only a limited number for each specified field. The first thing to note is the dedup command returns events, which contrasts with stats commands which return counts about the data. We’ll focus on the standard mode, which is a streaming search command (it operates on each event as a search returns the event). Typical examples of a dedup produce a single event for each host or a pair of events for each sourcetype.ĭedup has a pair of modes. The Splunk dedup command, short for “deduplication”, is an SPL command that eliminates duplicate values in fields, thereby reducing the number of events returned from a search.

Splunk commands