One of my uglier scripty bits

So – sed and awk versions on the Mac are a little behind. Turns out if I write an expression to wrap a forward slash into braces that it doesn’t exactly translate when moving from Linux to a Mac. So this happened (make fun, it’s cool):

#!/bin/bash #Extracts occurances of a file path from a text file #Then sorts and counts up each echo "Please make sure to run script from same Original_file PATH" read -p "Enter filename: " original_file cp $original_file tempfile.txt && sed 's/ /\n/g' tempfile.txt > tempfile1.txt && sed -i '.bak' '/^$/d' tempfile1.txt && cat tempfile1.txt | grep '^/' | sort |uniq -c | sort -k2nr |sort -r| awk '{printf("%s,%s\n",$1,$2)}' > filtered-sorted-file.txt && rm -rf tempfile*.txt

The script extracts all file paths from a text file and sorts/counts them. I thought it would take 5 minutes… Famous last words…

A Bit About Python Module Imports

The first lines of practically every python script ever begin with the word import. Per the python docs at https://docs.python.org/2/tutorial/modules.html a module is “a file containing Python definitions and statements.” To open a python environment, simply open Terminal and type Python

python

To import a module just type import followed by the name of the module. A common one is os, so let’s import that:

import os

Now let’s use str to see where it is:

str(os)

You’ll then see what the script was that is actually the os module and where it can be found/edited/viewed. Now let’s do a couple of basic tasks with os. First, let’s grab the user who’s running the script’s id (still from a standard python interpreter):

os.getuid()

Here, we’re using a function that’s built into that os script, and the () is the parameters we’re passing to the function. When run in that interactive mode, you can use os.environ to see what environment variables your python script has access to (e.g. if you’re shelling out a command).

os.environ

Now let’s use that os to actually shell out a bash command. Here, we’ll set a variable in the python interpreter:

bashCommand="cat /test.log"

Basically, we just set a variable called bashCommand to contain a simple cat of a log file (quoting it since it has special characters and all that). Next, we’ll use os.system with the variable of the command as the parameter we’re sending into the command:

os.system(bashCommand)

Now I can clear the contents of that bashCommand variable using the del command, still from within that python console:

del bashCommand

When a module is imported, the interpreter first searches for a built-in module with the name you supply. If the interpreter can’t find a module, it will then search through the current working directory, then the PYTHONPATH wet in sys.path and . If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. Now, let’s import urllib and check out what functions it has:

import urllib dir(urllib)

Then let’s ask for help for one of them:

help(urllib.basejoin)

The response will look something like this (ymmv):

urljoin(base, url, allow_fragments=True)

    Join a base URL and a possibly relative URL to form an absolute

    interpretation of the latter.

Now you know what goes in the parenthesis when you actually call the function from within your scripts.

Looping cURLy Braces

No, this article is not about my daughter’s curly hair and that expander thingie she had in her mouth for awhile (that sucked for her, booo). Instead we’re talking about expanding braces in shell scripts so we can expand braces in the script.

What’s a brace? As with grammar, a brace is a type of bracket meant to denote a series of characters. The bash interpreter will then allow for the expansion to include a series, such as 1..100 or 1 to 100 or a..z or a through z, useful in loops where the variable name (num in the below example) when invoked in the loop as $num will expand to be the number in the series in the loop. In the below example, we start with a set command and use -B to expand the curly braces:

set -B
for num in {1..100}; do
echo `curl -s -k 'GET' -H 'header info' 'http://krypted.com/log='$num`
echo 'http://krypted.com/log='$num
done


Note: In the above example, I did an echo just as a debugging line. I’d remove that before actually running the script (or more to the point automating it to run).

If we hadn’t of used the brace, we’d just be typing a lot more to get a 1 to 100 series to expand (e.g. for (( i = 0; i < 20; i++ ))) You can also use curly braces to do some much cooler expansion than a series of numbers. For example, {a,b{1..100},c,d{1…100},e,f{1…100},g} would do a then b1 through b100 then c, etc. In the above e script, once we’ve enabled brace expansion with that -B for set, we can then simply echo a curl onto the screen that loops through a series of 100 log files, using the $i within the loop.

A simpler example might be to use the following , which uses a series of letters instead:

set -B
for letter in {a..z}
do
echo $letter
done


If you actually wanted to learn more about palatal expanders then I’m sorry to bore you with everything else – here’s a better place for learning about that: https://en.wikipedia.org/wiki/Palatal_expansion

Super-Simple Bash Graphs

The sparkr gem is installed by default in macOS. To use it to produce simple graphs, simply run it followed by a series of integers:

sparkr 12 110 250 110 12

The result would be as follows:

This is useful for a quick and dirty visualization in scripts. For example, a series of 5, 10, 200 numbers that don’t have that much range where you’re just looking for a simple pattern. Like number of lines in logs, etc. Obviously, you can pay a lot of money for graphing frameworks and very fancy-schmancy tools. This is really just for me in small scripts. 

Note: sparkr isn’t installed on all Mac systems. to install it manually use:

sudo gem install sparkr

Thanks to Armin Briegel for pointing out that sparkr isn’t installed by default on the latest OSen.

Use colrm to remove columns from a text file in bash

The colrm command is a simple little command that removes columns from standard input before displaying them on the screen (or piping the text into another file). To use, simply cat a file and then pipe it to colrm followed by the start and then stop in $1 and $2. For example, the following would only list the first column of a text file called testfile:

cat testfile | colrm 2

Not providing a second column in the above command caused only the first column to be displayed to the screen. You could pipe all but the second and third columns of a file to another file called testfile2 using the following:

cat testfile | colrm 2 3 > testfile2

Install and Use Homebrew on macOS

Homebrew is a package manager for macOS. You can use Homebrew to install command line packages on a Mac, provided someone has written a formulae, which is a simple Ruby script that walks through the process for installing all the little bits required for a piece of software.

Installing Homebrew is simple. Run the following command which is listed on the Homebrew homepage (not as root):
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
This will install the macOS Command Line Tools from Xcode as well as create the following directories (if they’re not already present):
  • /usr/local/Cellar
  • /usr/local/Homebrew
  • /usr/local/Frameworks
  • /usr/local/opt
  • /usr/local/sbin
  • /usr/local/share/zsh
  • /usr/local/share/zsh/site-functions
  • /usr/local/var
Then the script will move all the required bits from https://github.com/Homebrew/brew to the correct locations. Once done, you can easily install a package if you know the name. For example, I do this on practically every new machine I configure for development:

brew install wget

This one is nice because the dependencies that get installed. And you get the latest versions. Let’s look at the version for wget:

wget -V

Next, let’s use brew to search for something: radius

brew search radius

You’ll see that there’s one item on the local taps: freeradius-server

Let’s install that:

brew install freeradius-server
Now, you’ll find that the bits that make freeradius work are located in /usr/local/Cellar/freeradius-server/3.0.16. If you later need to upgrade that package, use the upgrade verb.

brew upgrade freeradius-server

And finally, to update Homebrew to the latest version, run the update verb:

brew update

Bash: if, then, else, and expressions

One of the most important skills in any language is to know how a basic if then statement works. You set a variable, you say if that variable matches a given pattern (or doesn’t) then do something or else do something different. If/then/else. Other than that, it’s just syntax, and the annoyance of figuring out the operators and syntax. 

So let’s go through the below script (which checks a Mac to see if grayscale is enabled in Universal Access), line-by-line:
  • Line 1: we’re first going to create a variable called grayscale. We’ll fill it with a 0 or 1, the output of a simple “defaults read com.apple.universalaccess grayscale” command
  • If that grayscale variable matches 0 execute whatever is after the then
  • One line just for then followed by the command in the next line (which will optionally allow for an else)
  • Use echo to display the text on the screen that indicates Grayscale is disabled to stdout (which could be redirected if you want to do something after we’re done processing)
  • One line just for the else (optional)
  • The echo to display it’s enabled
  • fi ends the if/then so you can move on to the rest of the script (even if you have nothing after the if/then you’ll need the fi or the script won’t run)
grayscale=`defaults read com.apple.universalaccess grayscale`
if [[ $grayscale = 0 ]]
then
echo "Grayscale is disabled"
else
echo "User doesn't have a token"
fi

In the above if, we used = as an expression. That’s the most common one you’ll likely use, but there are a lot, including the following:
  • [ -a FILE ] True if FILE exists
  • [ -b FILE ] True if FILE exists and is a block-special file
  • [ -c FILE ] True if FILE exists and is a character-special file
  • [ -d FILE ] True if FILE exists and is a directory
  • [ -e FILE ] True if FILE exists
  • [ -f FILE ] True if FILE exists and is a regular file
  • [ -g FILE ] True if FILE exists and its SGID bit is set
  • [ -h FILE ] True if FILE exists and is a symbolic link
  • [ -k FILE ] True if FILE exists and its sticky bit is set
  • [ -p FILE ] True if FILE exists and is a named pipe
  • [ -r FILE ] True if FILE exists and is readable
  • [ -s FILE ] True if FILE exists and has a size greater than zero
  • [ -t FD ] True if file descriptor FD is open and refers to a terminal
  • [ -u FILE ] True if FILE exists and its SUID (set user ID) bit is set
  • [ -w FILE ] True if FILE exists and is writable
  • [ -x FILE ] True if FILE exists and is executable
  • [ -O FILE ] True if FILE exists and is owned by the effective user ID
  • [ -G FILE ] True if FILE exists and is owned by the effective group ID
  • [ -L FILE ] True if FILE exists and is a symbolic link
  • [ -N FILE ] True if FILE exists and has been modified since it was last read
  • [ -S FILE ] True if FILE exists and is a socket
  • [ FILE1 -nt FILE2 ] True if FILE1 has been changed more recently than FILE2, or if FILE1 exists and FILE2 does not
  • [ FILE1 -ot FILE2 ] True if FILE1 is older than FILE2, or is FILE2 exists and FILE1 does not
  • [ FILE1 -ef FILE2 ] True if FILE1 and FILE2 refer to the same device and inode numbers
  • [ -o OPTIONNAME ] True if shell option “OPTIONNAME” is enabled
  • [ -z STRING ] True of the length if “STRING” is zero
  • [ -n STRING ] or [ STRING ] True if the length of “STRING” is non-zero
  • [ STRING1 == STRING2 ] True if the strings are equal. “=” may be used instead of “==” for strict POSIX compliance
  • [ STRING1 != STRING2 ] True if the strings are not equal
  • [ STRING1 < STRING2 ] True if “STRING1” sorts before “STRING2”
  • [ STRING1 > STRING2 ] True if “STRING1” sorts after “STRING2” 
  • [ ARG1 OP ARG2 ] “OP” is one of -eq, -ne, -lt, -le, -gt or -ge. These arithmetic binary operators return true if “ARG1” is equal to, not equal to, less than, less than or equal to, greater than, or greater than or equal to “ARG2”, respectively. “ARG1” and “ARG2” are integers
Attribution: The above list of expressions was originally borrowed from http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html.

This New Years Day, Learn The Jot Command

This New Years Day, Learn The Jot Command The jot command is one I haven’t used in awhile. But it’s still useful. Let’s take a look at a few of the things you can do with it. First, let’s just print lines into a new file called “century.txt” which we can do by running with the number of increments followed by the starting number, and then redirecting the output into the file name:

jot 100 1 > ~/Desktop/century.txt

Or to do integers instead, simply put the decimals:

jot 100 1.00 > ~/Desktop/century.txt

Or in descending order,

jot – 100 1 > ~/Desktop/century.txt

Now we could change the output to be just 50 to 100, by incrementing 50 (the first position) and starting at 50 (the second):

jot 50 50

The jot command is able to print sequential data, as we’ve seen. But we can also print random data, using the -r option. Following that option we have three important positions, the first is the number of iterations, but the next two are the lower and upper boundaries for the numbers, respectively. So in the below command we’ll grab 10 iterations (or ten random numbers) that are between 1 and 1000:

jot -r 10 1 1000

Now if we were to add a -c in there and use a and z as the upper and lower bounds, we’d get… letters (this time we’re just gonna’ ask for one letter)!

jot -r -c 1 a z

Something I find useful is just to shove random data into a file infinitely. And by useful I mean hopefully not left running overnight on my own computer (been there, done that). To do this, just use a 0 for the number of iterations:

jot -r -c 0

Something that is actually useful is the basic ASCII set:

jot -c 128 0

We can also append data to a word using -w. So let’s say we want to print the characters aa followed by a through z. In the below we’ll define that with -w and then we’ll list those two characters followed by %c which is where the character substitution goes and then the number of iterations followed by the lower bound:

jot -w aa%c 26 a

You can also do stuttering sequences, useful for the occasional tango dancer, so here we’ll do a 5/3 countdown:

jot – 100 0 -.5

Or we could create a one meg file by creating 1,024 bytes:

jot -b 0 1024 > onemegfile.txt

Oh wait, that file’s two megs. Get it? 😉

And running strings teaches you that you can’t bound random (a good lesson for the New Year). Anything you use jot for?

Happy New Years!

Quick and dirty: Pull a list of all filevault encrypted users on a Mac

In the following example script, I’m going to pull a list of just the usernames from fdesetup. sudo fdesetup list The output would be as follows:
charlesedge,F4D8B61D-1234-1234-98F4-103470EE1234 emerald,2E1203EA-1234-4E0D-1234-717D27221234 admin,50058FCF-88DF-1234-1234-91FCF28C0488
I’ll then pipe them into sed and use the , as a delimiter, pulling * or everything before it: sudo fdesetup list | sed 's;,.*;;' As follows:
charlesedge emerald admin

Is This Bash Command A Builtin?

Builtin commands are always kinda’ interesting. At first glance, it’s hard to know which commands are builtins. Luckily, there’s a command that I rarely use, called… command. If you run command with the -V flag it will tell you if the command is a builtin: command -V cd
cd is a shell builtin
If you run a command that isn’t a builtin command -V ls
ls is /bin/ls
Some builtins are in /bin (like echo). But not all builtins are in /bin. Some are in /usr/bin (like cd). Information about how to use builtins is built into the help command rather than standalone man pages. So, if you do help followed by the name of a command, you’ll get information about the command, and sometimes how to use the command: help cd
cd: cd [-L|-P] [dir] Change the current directory to DIR. The variable $HOME is the default DIR. The variable CDPATH defines the search path for the directory containing DIR. Alternative directory names in CDPATH are separated by a colon (:). A null directory name is the same as the current directory, i.e. `.’. If DIR begins with a slash (/), then CDPATH is not used. If the directory is not found, and the shell option `cdable_vars’ is set, then try the word as a variable name. If that variable has a value, then cd to the value of that variable. The -P option says to use the physical directory structure instead of following symbolic links; the -L option forces symbolic links to be followed.
There are also commands not in a path, which can be found using the which command: which dsconfigad
/usr/sbin/dsconfigad