krypted.com

Tiny Deathstars of Foulness

Earlier, we looked at creating thousands of empty directories. Today, we’re going to get rid of them. But we need to get rid of only empty directories. To do so, we’ll use the find command:

find . -depth -type d -empty -exec rmdir {} \;

Now, we can put both into a script:

mkdir $(printf '%05d\n' {1..10000})
find . -depth -type d -empty -exec rmdir {} \;

July 29th, 2015

Posted In: Mac OS X, Ubuntu, Unix

Tags: , , , ,

You see a lot of entries for various things in log files. Here, we’re going to print out the number of entries with backupd in them:

awk '/backupd/{print NR}' /var/log/system.log

July 11th, 2015

Posted In: Mac OS X, Ubuntu, Unix

Tags: , , , , , ,

The wc command is used to count words, characters and lines. Here, we’ll run it a few different ways. -l shows the number of lines in a file. For example, in my home directory, I can use it to see how many lines are in my .gitconfig file:

wc -l .gitconfig

This would output something like the following:

11 .gitconfig

Or count the number of characters with -c:

wc -c .gitconfig

Or check the number of words:

wc -w .gitconfig

You can also run it against multiple files. For example, here I’ll check the number of lines in both my .gitconfig file and my .gitignore_global files:

wc -l .gitconfig .gitignore_global

Let’s say I have a list of numbers and I want to take an average of them. I can use this to quickly figure out how many numbers I have (and so will divide by) before tallying them up.

July 4th, 2015

Posted In: Mac OS X, Ubuntu, Unix

Tags: , , , , , , ,

There are a number of ways to see information about what version of Linux that you’re running on different

cat /etc/lsb-release

Which returns the distribution information, parsed as follows:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04.5
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu Precise Pangolin (LTS)"

LSB_release can also be run as a command, as follows:

lsb_release -a

Which returns the following:

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Precise Pangolin (LTS)
Release: 12.04.5
Codename: precise

lab_release can be used as a command as well:

cat /etc/issue.net

Which returns:

Ubuntu Precise Pangolin
(development branch)

In Debian, you can simply look at the version file:

cat /etc/debian_version

Which returns the following:

wheezy/sid

Or Red Hat Enterprise can also be located with /etc/issue.net:

cat /etc/issue.net

With many variants, including OS X, you can also use uname to determine kernel extensions, etc:

uname -a

The thing I’ve learned about Linux is that there’s always a better way to do things. So feel free to comment on your better way or favorite variant!

March 5th, 2015

Posted In: Ubuntu, Unix

Tags: , , , , , ,

Popped in a list of Linux bash commands here: http://krypted.com/commands/linux-bash-commands/

January 24th, 2015

Posted In: Ubuntu

Tags: , , , , ,

The Terminal application defaults have a problem passing colors with Ubuntu and other types of Linux machines with properly formed .bashrc files. This is because those systems do not know how to interpret the Lion xterm-color256 terminal declaration.

The fix is to change this setting to xterm-color. This needs to be done for each Terminal default. Click on each (Basic, Grass, Homebrew, etc) and then click on the Advanced tab. From there, just set the Declare terminal as: to xterm-color and close.

This can also be done through the command line. These settings are stored in the com.apple.Terminal.plist per user, in their ~/Library/Preferences. The key for each is in TerminalType, which should read:

TerminalType = "xterm-color";

This small issue pales in comparison (especially because it is easily remedied) to how great the ability to take Terminal into full screen mode, have it state fully open and show the last run commands, assign background images and other little cool tricks are. There are also lots of new commands in Lion, such as tmutil that you’re sure to love if you haven’t made the upgrade yet!

August 7th, 2011

Posted In: Mac OS X

Tags: , , , , , , , , , , ,

There are two commands that can be really helpful when scripting operations that involve filenames and paths. The first of these is dirname: dirname can be used to return the directory portion of a path. The second is basename: basename can be used to output the file name portion of a path.

For our first example, let’s say that we have an output of /var/db/shadow/hash/850F62CD-966C-43A7-9C66-9F9E6799A955, which we know contains the encrypted password for a given user. To just see the UUID here would be done using the following extremely basic incantation of basename:

basename /var/db/shadow/hash/850F62CD-966C-43A7-9C66-9F9E6799A955

Basename can also be used to trim output. For example, let’s say we didn’t need the final portion of the above filename in our output. We could run basename using the -s option, followed by the string at the end that we do not want to see to output of just the first 4 sections:

basename -s -9F9E6799A955 /var/db/shadow/hash/850F62CD-966C-43A7-9C66-9F9E6799A955

The dirname command is even more basic. It outputs the directory portion of the file’s path. For example, based on the same string, the following would tell you what directory the UUID files with the passwords are stored in:

dirname /var/db/shadow/hash/850F62CD-966C-43A7-9C66-9F9E6799A955

A great example of when this gets more useful is keying off of currently active data. For example, if we’re scripting a make operation, we can use the which command to get an output that just contains the path to the make binary:

which make

We can then wrap that for expansion and grab just the place that the active make binary is stored:

dirname `which make`

This allows us to key other operations off the path of an object. A couple of notable example of this is home or homeDirectory paths and then breaking up data coming into a script via a positional parameter (e.g. $1).

May 27th, 2011

Posted In: Mac OS X, Mass Deployment, Ubuntu, Unix

Tags: , , , , , , , ,

Ifenslave is an open source package that can be used to bond interfaces in Ubuntu 10. To install ifenslave, we can use apt-get:

apt-get install ifenslave

Once installed, we will need to take down our existing eth interfaces. Presumably these are eth0 and eth1, but you can use ifconfig to verify:

ifconfig eth0
ifconfig eth1

Once you’ve verified the interfaces you want to bond, bring them down:

ifdown eth0
ifdown eth1

Next, locate the entries in /etc/network/interfaces and comment out the corresponding lines:

vi /etc/network/interfaces

You will then need to add information for the link aggregated bond. Bond levels in ifenslave include:

  • bond0: Round Robin with all interfaces active (likely the most common).
  • bond1: Fault Tolerance with only one slave to the bond active at a time. The backup kicks in if the active slave fails.
  • bond2: XOR with bond policy defined using xmit_hash_policy.
  • bond3: Transmits all traffic on all slaves.
  • bond4: 802.3ad or dynamic link aggregation. Similar but more robust to bond0, although has more prerequisites.
  • bond5: Doesn’t require special switch configuration, but has incoming traffic on one slave and balanced outgoing traffic.
  • bond6: Also doesn’t require special switch configuration (although you may need to configure updelay on the switch), otherwise similar to bond4 but more friendly to more members.

Let’s assume we’re going to go with a vanilla round robin bond (bond0). Let’s add the following (which includes the IP address, netmask and gateway as well as the bonding information):

auto bond0
iface bond0 inet static
address 192.168.210.100
netmask 255.255.255.0
gateway 192.168.210.1
slaves all
bond-mode 0
bond-miimon 100

These variables:

  • iface: defines the interface (similar to a standard interface)
  • address: IP address
  • netmask: Defines the subnet/netmask
  • gateway: Defines the router, or default gateway
  • slaves: Defines which adapters will be a part of the bond (also can use master/primary in addition to slaves according to bond type)
  • bond-mode: mode of the bond (corresponds to bond*)
  • bond-miimon: An integer from 0 to 100 that defines the number of milliseconds between link monitors
  • updelay and downdelay: milliseconds to delay bringing links up and down when failures and additions are detected
  • arp_interval and arp_target: The number of milliseconds to test arp connectivity and the address to use for testing

Finally, we can bring the bond online using ifup followed by the bond name:

ifup bond0

You should then be able to ping, use iftop, netperf, etc to test the performance of the bond.

April 19th, 2011

Posted In: Ubuntu, Unix

Tags: , , , , , ,

There are a number of ways that you can interact with Google Apps: there is the website, the new Google Cloud Connect and an API that allows you to integrate Google Apps with your own solutions. The API is available for python and java and can take some time to get used to, even though Google has done a good job with making it pretty straight forward (comparably). Therefore, there are a couple of tools that ease the learning curve a bit.

GoogleCL on Ubuntu

The first, and easiest is GoogleCL. GoogleCL is a command line version of Google Apps that will allow you to interact with YouTube, Picasa, Blogger and of course Google Docs. To use GoogleCL you’re going to need python-gdata. If you’re using Ubuntu, you would do an apt-get and install python-gdata:

apt-get install python-gdata

Once installed, you’ll want to then download the deb package from Google Code:

wget http://googlecl.googlecode.com/files/googlecl_0.9.11-1_all.deb

Once downloaded, install it using dpkg with the -i option (assuming you’re still using the same working directory:

dpkg -i googlecl_0.9.11-1_all.deb

GoogleCL on Mac OS X

GoogleCL is also available for the Mac. First, download the gdata-python-client from http://code.google.com/p/gdata-python-client/downloads/list and then extract the file (ie – unzip gdata-2.0.13). Next, install it using Python (2.0.13 is the latest version) with your working directory set to the previously extracted folder:

python setup.py install

Next up, let’s grab GoogleCL from the GoogleCL Google Code page:

wget http://googlecl.googlecode.com/files/googlecl-0.9.11.tar.gz

Then hop into the newly extracted directory and run the python installer:

python setup.py install

Using GoogleCL on Mac and Linux

Once GoogleCL has been installed, the use is the same between Mac OS X and Linux. Simply use the newly acquired google command (this is actually a Python front-end to the API at /usr/bin/google) followed by a service and then a verb. Verbs are based on services (not all services offer the same features and therefore do not have the same verbs). A list of services with their verbs includes the following.

docs – Allows for interaction with Google Docs, with verbs that include the following:

  • edit – Allows you to indicate an application to use as an editor for the given document (ie – vi).
  • delete – Delete a document on Google Docs.
  • list – List documents on Google Docs.
  • upload – Uploads the specified document (options include title, folder and format of the document being uploaded).
  • get – Downloads the specified document in the format specified using the format option.

blogger – Manage content stored using the blogger service.

  • post – Allows you to post content (which is then known as blog).
  • tag – Requires a title (for blog entries) and the tags that you would like to use with the post in question.
  • list – Shows posts (can use blog entry, title and owner as a delimiter, useful when used w/ grep to constrain output).
  • delete – Removes a post specified.

picasa – Allows you to interact with the picasa service for posting and obtaining images used with Google Apps.

  • get – Download specified albums.
  • create – Create an album.
  • list – List images.
  • list-albums – List albums.
  • tag – Tag images
  • post – Add a photo to an album.
  • delete – Delete a photo or an album.

contacts – Manage contacts (given the lack of an edit option, use an add and then a delete to impart an edit).

  • list – Show contacts (can specify fields to constrain output).
  • list-groups – Show the groups for a user.
  • add – Add a contact.
  • add-groups – Create a group of contacts.
  • delete-groups – Remove a group of contacts
  • delete – Remove a single contact

calendar – Manage calendars.

  • add – Create a calendar entry
  • list – Show all events on a given calendar.
  • today – Show calendar events over the next 24 hour period.
  • delete – Remove calendar events.

Beyond GoogleCL

Let’s put this into perspective. Let’s say I have an application, and that application can run a simple shell command. Then, let’s say I create a calendar event in that application. The application could send a command to the shell with a variable. If I had calendar information to create such as “Meeting with KK tomorrow at 9am” then I could send a command as follows:

google calendar add “Meeting with KK tomorrow at 9am”

This would cause the event to appear on my calendar and sync to any devices that were then configured to work with my calendar. But, if I were to issue this command on the server-side then it would attempt to create all events for the same users, which is likely not very helpful for most organizations that have more than one calendar and/or user.

As mentioned /usr/sbin/google is a python script. It makes use of python-gdata and provides a more direct access to the Google Apps API. As such, it allows for far more complex logic than the GoogleCL front-end does. The google script does give savvy developers a look at how Google intends for many of their methods to be used and even allows you to borrow a line or two of code here and there. Simple logic can be parlayed into code quickly using GoogleCL, but you will quickly outgrow what can be done with GoogleCL and move into using the API more directly if you have any projects of substance!

November 28th, 2010

Posted In: cloud, Mac OS X, Mac OS X Server, Ubuntu, Unix

Tags: , , , , , , , , ,

Using the firewall in Ubuntu can be as easy or as hard as you want to make it. BSD variants all basically use the ipfw command whereas most of the rest of the *nix world will use netfilter. Netfilter has a number of front ends; the one that comes pre-installed in Ubuntu is ufw, short for ‘uncomplicated firewall’. Ufw is good for basic port management: allow and deny type of stuff. It’s not going to have the divert or throttling options. So let’s look at some basic incantations of ufw (you need to have elevated privileges to do all of this btw).

Initial Configuration

First you need to enable ufw, which is done using the ufw command (no need to apt-get this to install it or build it from source) followed by the enable option:

ufw enable

You can also use the disable option to turn the firewall back off:

ufw disable

And to see rules and the status of the firewall, use the status option:

ufw status

The ufw Configuration File

The ufw configuration file is /etc/default/ufw. Here, you can manage some basic options of ufw. These include:

  • IPV6 – Set to YES to enable
  • DEFAULT_INPUT_POLICY – Policy for how to handle incoming traffic not otherwise defined by a rule. Defaults at DROP but can be changed to ACCEPT or REJECT
  • DEFAULT_OUTPUT_POLICY – Same as above but for handling outgoing traffic not otherwise defined by a rule.
  • DEFAULT_FORWARD_POLICY – Same as above but for forwarding packets (routing).
  • DEFAULT_APPLICATION_POLICY – I’d just leave this as the default, SKIP.
  • MANAGE_BUILTINS – when set to yes, allows ufw to manage default iptables chains as well.
  • IPT_MODULES – An array of iptables modules that can be added

To restart ufw after you make changes to the configuration file, use the services command:

service ufw reload

Or:

service ufw restart

Creating Rules

The first thing most people will want to do is enable a port. And of the ports, 22 is going to be pretty common, since without it you can’t ssh back into the box. For this, you’ll use the allow option followed by the name of the service (application profile):

ufw allow ssh

You can use numbers instead (since ufw isn’t going to know every possible combination and you might be running some on custom ports):

ufw allow 22

You can also deny traffic using the same structure, just swapping allow with deny:

ufw deny http

Beyond a basic allow and deny, you can also specify what IP addresses are able to access each port. This is done using ufw followed by the proto option, which is the followed by the actual protocol (tcp vs udp, etc) which is then followed by the from option and then the source then the to option then the IP to accept traffic (or the any option for all IPs on your box) and finally the port option followed by the actual port. Sounds like a lot until you see it in action. Let’s say you actually want to allow traffic for port 10000 but only from 192.168.210.2. In that case, your rule would be:

ufw allow proto tcp from 192.168.210.2 to any port 10000

Or if you only wanted 10000 to be accessible on one IP of your system (theoretically you have two in this scenario) that has an address of 192.168.210.254:

ufw allow proto tcp from 192.168.210.2 to 192.168.210.254 port 10000

Using ufw

Once you have your rules configured, you are invariably going to have to troubleshoot issues with the service. Obviously, start with log review to perform a hypothesis of what the problem is. To enable logging use the logging option and specify the on parameter for it:

ufw logging on

Once enabled I usually like to view both /var/log/messages and /var/log/syslog for entries:

cat /var/log/syslog | grep UFW ; cat /var/log/messages | grep UFW

One of the best troubleshooting tools to prove any hypothesis that has to do with a rule is to simply delete the rule. To delete the deny http rule that we made earlier, just use the ufw command along with the delete option specifying the deny 22 rule as the rule to remove:

ufw delete deny http

Additionally, just disabling ufw will usually tell you definitively whether you are looking at a problem with a rule, allowing you to later look into disabling each rule until you find the offending rule.

iptables

Ubuntu also comes with iptables by default. iptables is the ipchains replacement introduced a number of years ago and is much more complicated and therefore flexible than ufw, although using one does not mean you cannot use the other. To get started, let’s look at the rules:

iptables -L

You will then see all of the rules on your host. If you have been enabling rules with ufw these will be listed here. You can then configure practically anything for how each chain (a chain is a series of rules for handling packets) functions. You can still do basic tasks, such as enabling ssh, but iptables will need much more information about what specifically you are trying to do. For example, to accept incoming traffic you would need to define that the chain will add an input (by appending it to the chain using -A INPUT) for tcp packets (-p tcp) on port 22 (–dport ssh) and accepting those packets (-j ACCEPT):

iptables -A INPUT -p tcp –dport ssh -j ACCEPT

That is about as simple as iptables get. I’ll try and write up more on dealing with it later but for now you should have enough information to get a little wacky with some basic firewall functionality on Linux. Enjoy.

November 24th, 2010

Posted In: Ubuntu, Unix

Tags: , , , , , , , , , ,

Next Page »