Talk structure

Hi everyone, I'm Sam and I'll be talking about a few quick command line tools you can use to work a lot faster.

Before I get into details, I just want to remind everyone that the best way to find out what a tool can do is to check out its man page. 

$ man man

In ten minutes, there's only so much that I can talk about, but all of these tools have a ton of options built in.

$ clear

I'll start off with the ps command.

$ ps

ps displays information about the processes currently running on the system. By default it only shows your current processes. We can see every process by running `ps aux`.

$ ps aux

This is a crazy huge list. What if we just want to see one user?

$ ps -u scgruber

But I have multiple sessions going, so wouldn't it be great if I could see how they are related? Perhaps in some sort of tree?

$ ps f -u scgruber

Now all of this is great, but let's say I want to select processes by name. ps isn't so great at that.

$ clear

Enter grep.

$ ps aux | grep 'sshd'

All I've done here is take the output of ps and send it to grep. grep's sole calling in life it to look through lines of text and report back matches. So `grep sshd` shows me all of the active SSH sessions on this server.

But maybe I want to see who's editing text files.

$ ps aux | grep 'vi'

That's okay, but we can do better... regular expressions!

$ ps aux | grep -E 'vi|emacs'

And with color!

$ ps aux | grep -E 'vi|emacs' --color

Right now this is all sorted by process id, which is nice, but not really useful for us humans. What if we want to group things together by user? The easiest way to do that is just to sort them.

$ ps aux | grep -E 'vi|emacs' | sort

By default this sorts in dictionary order, we can also sort randomly.

$ ps aux | grep -E 'vi|emacs' | sort -R

Let's say we want to order by memory usage. This is the fourth field in ps, so we can select that with the -k flag. -n is numeric, -b means ignore blank spaces at the beginning, and -r (that's little r) means reverse, because by default sort puts small things up top and big things at the bottom.

$ ps aux | grep -E 'vi|emacs' | sort -k 4 -nbr

Ok so now we've got this data, and we can sort it, but a lot of it isn't really that valuable to us. There are a couple of different ways that we can make this simpler.

$ clear

The first one is cut.

$ ps aux | grep -E 'vi|emacs' | cut -b 1-9

-b means bytes, so it extracts the characters on the line in a given range. This can be great if I only want, say, the first four letters of each username for some reason.

$ ps aux | grep -E 'vi|emacs' | cut -b 1-4

But it's not so great if I want the last four

$ ps aux | grep -E 'vi|emacs' | cut -b 6-9

Cut has some ability to select by field.

$ ps aux | grep -E 'vi|emacs' | cut -sf 1

But it depends on a nicely-structured output (meaning tab-separated). We can change the delimiter to a space, but this causes a different problem.

$ ps aux | grep -E 'vi|emacs' | cut -d " " -sf 2

$ clear

Fortunately, cut isn't the only way to pull data out of lines of text.

$ ps aux | grep -E 'vi|emacs' | awk '{print $1,$2,$3,$4}'

awk is a much better tool for getting fields. As you can see, it's smarter than cut about delimiters. We can easily insert tabs in our output.

$ ps aux | grep -E 'vi|emacs' | awk '{print $1,"   \t",$2,"\t",$3,"\t",$4,"\t",$11}'

awk can process fairly complex scripts, and you don't generally want to type that all on a command line. the -f option allows you to run awk scripts from a file.

$ ps aux | grep -E 'vi|emacs' | awk -f awkscript

//----awkscript----
{print $1,"   \t"}
{print $2}
//-----------------

$ clear

So let's see what we can do with this info. Maybe I don't like emacs because everyone should be using vim?

$ ps aux | grep -E 'vi|emacs' | sed s/emacs/vim/

sed is a great tool for manipulating text. I'm just going to scratch the surface in this talk, but it's got a lot of complexity to it. If you know regular expressions, sed is your friend. If you don't, well you should really go out and learn regular expressions.

We can use parentheses and storage variables to change part of the selected string, say the year. Wrap what you want in parentheses (with escapes) and then use \1 to put that into the output.

$ ps aux | grep -E 'vi|emacs' | sed 's/\(20\)12/\199/'

Maybe I don't like the number 1 and I want to change all 1's to 7's. I use the g flag to make the edit global along the whole string.

$ ps aux | grep -E 'vi|emacs' | sed 's/1/7/g'

Remember if you're doing any complicated sed commands to use quotes so that the bash shell doesn't try and interpret special symbols for you.

We can decide to only print the lines we change.

$ ps aux | grep -E 'vi|emacs' | sed -n s/emacs/vim/p

$ clear

So another thing we notice about the output of ps is that it has a lot of numbers. Well, what are numbers good for? Putting in calculators of course!

$ bc

bc is a basic calculator. It works pretty much like your calculator from high school: type in some math, get an answer

> 1 + 1
> last * 5
> a = 42
> a^2
> quit

You can also pipe output into bc. Let's say for some reason we want to add up the cpu and memory usage of each ssh session

$ ps aux | grep 'sshd' | awk '{print $3,"+",$4}' | bc

This is nifty. Now what if we want to add up all of these totals?

$ clear

We use a tool called xargs to get them on the same line. Bear with me, this involves a long command.

$ ps aux | grep 'sshd' | awk '{print $3,"+",$4}' | bc | xargs echo

xargs takes a series of lines, and feeds them in as arguments to the command you give it. In the case of echo, this puts all of the numbers on one line.

$ ps aux | grep 'sshd' | awk '{print $3,"+",$4}' | bc | xargs echo | sed 's/\s/\+/g'

Now we use sed to replace the spaces with plus symbols.

$ ps aux | grep 'sshd' | awk '{print $3,"+",$4}' | bc | xargs echo | sed 's/\([0-9]*\)\s/\1\+/g' | bc

And finally bc adds them up for us.

$ clear