Digging through the logs

*
Accepted Session
Short Form
Beginner
Scheduled: Tuesday, June 21, 2016 from 2:30 – 3:15pm in B302/303

Excerpt

Okay, so now it's time for the really fun part: We've removed the duplicate rows from the log, now we need to only show the rows that contain something that *looks like* an IP address.
To do this we'll use a search pattern. These patterns are written in Regular Expressions or RegEx. Like so many other tools in Linux they're immensely powerful but either don't work at all or go haywire with a single incorrect character. Let's write one that looks for a cluster of numbers, then a period, then another cluster of numbers.

Description

“Pop Quiz hot shot: One user or a small group of users has been hammering your service. Problem is, their requests are logged along with everyone else’s in a log file that’s 2 million lines long. You need to find the IP address that shows up most frequently in the log file. What do you do?”

In 45 minutes I’ll show you how to do one of the most common and critical tasks in server maintenance: using Linux tools to sort, filter, search, and wrangle giant logfiles.

Tags

grep, regex, awk, uniq, linux, tutorial, beginner, operations

Speaking experience

I have given a similar talk in tutorial format once before, and by the time of OSBridge will have done it once more at LinuxFest NW.
I spoke at Indie Game Con last year and regularly present to local game dev meetup groups.

Speaker

  • Toby Fee

    Web Dev

    Biography

    Self-taught programmer working as a web dev for Vacasa, currently looking for my next job.

    Sessions

      • Title: Digging through the logs
      • Track: Practice
      • Room: B302/303
      • Time: 2:303:15pm
      • Excerpt:

        Okay, so now it’s time for the really fun part: We’ve removed the duplicate rows from the log, now we need to only show the rows that contain something that looks like an IP address.
        To do this we’ll use a search pattern. These patterns are written in Regular Expressions or RegEx. Like so many other tools in Linux they’re immensely powerful but either don’t work at all or go haywire with a single incorrect character. Let’s write one that looks for a cluster of numbers, then a period, then another cluster of numbers.

      • Speakers: Toby Fee

Leave a private comment to organizers about this proposal