krypted.com

Tiny Deathstars of Foulness

Over the users I’ve written a good bit about pushing a workload off to a virtual machine sitting in a data center somewhere. The Google CloudPlatform has matured a lot and I haven’t really gotten around to writing about it. So… It’s worth going into their SDK and what it looks like from a shell using some quick examples.

For starters, you’ll need an account with Google Cloud Platform, at cloud.google.com and you’ll want to go ahead and login to the interface, which is pretty self-explanatory (although at first you might have to hunt a little for some of the more finely grained features, like zoning virtual instances.

The SDK

The SDK will include the gcloud command, which you’ll use to perform most tasks in the Google CloudPlatform. To install the SDK, go to https://cloud.google.com/sdk/downloads and download the appropriate version for your computer. If you’re on a mac, most likely the x86_64 version.

Next, move the downloaded folder to a permanent location and run the install.sh inside it, which will kindly offer to add gcloud to your path.

./install.sh

Welcome to the Google Cloud SDK!
To help improve the quality of this product, we collect anonymized usage data
and anonymized stacktraces when crashes are encountered; additional information
is available at <https://cloud.google.com/sdk/usage-statistics>. You may choose
to opt out of this collection now (by choosing ‘N’ at the below prompt), or at
any time in the future by running the following command:
gcloud config set disable_usage_reporting true
Do you want to help improve the Google Cloud SDK (Y/n)?  y
Modify profile to update your $PATH and enable shell command
completion?
Do you want to continue (Y/n)?  y
The Google Cloud SDK installer will now prompt you to update an rc
file to bring the Google Cloud CLIs into your environment.
Enter a path to an rc file to update, or leave blank to use
[/Users/charlesedge/.bash_profile]:
Backing up [/Users/charlesedge/.bash_profile] to [/Users/charlesedge/.bash_profile.backup].
[/Users/charlesedge/.bash_profile] has been updated.
==> Start a new shell for the changes to take effect.
For more information on how to get started, please visit:
https://cloud.google.com/sdk/docs/quickstarts

Inside that bin folder, you’ll find the gcloud python script, which once installed, you can then run. Next, you’ll need to run the init, which links it to your CloudPlatform account via oauth. To do so, run gcloud with the init verb, which will step you through the process:

gcloud init

Welcome! This command will take you through the configuration of gcloud.
Your current configuration has been set to: [default]
You can skip diagnostics next time by using the following flag:
gcloud init –skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection…done.
Reachability Check passed.
Network diagnostic (1/1 checks) passed.

You must log in to continue. Would you like to log in (Y/n)? y

If you say yes in the above screen, your browser will then prompt you with a standard Google oauth screen where you’ll need to click Allow.

Now go back to Terminal and pick a “Project” (when you set up billing the default was created for you):

Pick cloud project to use:
[1] seventh-capsule-138123
[2] Create a new project
Please enter numeric choice or text value (must exactly match list
item):
1

The Command Line

Next, we’re gonna’ create a VM. There are several tables that lay out machine types. Let’s start by listing any instances we might have:

gcloud compute instances list

Listed 0 items.

Note: If you have a lot of these you can use  --regexp to filter them quickly.

Then let’s pick a machine type. A description of machine types can be found at https://cloud.google.com/compute/docs/machine-types. And an image. Images can be seen using the compute command with images and then list, as follows:

gcloud compute images list

Now, let’s use that table from earlier and make a custom machine using an ubuntu uri, a –custom-cpu and a –custom-memory:

gcloud compute instances create krypted1 –image https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-1610-yakkety-v20170502 –custom-cpu 2 –custom-memory 5

You’ll then see that your VM is up, running, and… has an IP:

Created [https://www.googleapis.com/compute/v1/projects/seventh-capsule-138523/zones/us-central1-a/instances/krypted1].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
krypted1 us-central1-a custom (2 vCPU, 5.00 GiB) 10.128.0.2 104.154.169.65 RUNNING

Now let’s SSH in:

gcloud compute ssh krypted1

This creates ssh keys, adds you to the hosts and SSH’s you into a machine. So viola. You’re done. Oh wait, you don’t want to leave it running forever. After all, you’re paying by the minute… So let’s list your instances:

gcloud compute instances list

Then let’s stop the one we just created:

gcloud compute instances stop krypted1

And if you’d like, tear it down:

gcloud compute instances delete krypted1

Overall, super logical, very easy to use, and lovely command line environment. Fast, highly configurable VMs. Fun times!

May 18th, 2017

Posted In: cloud, Mac OS X, Ubuntu, Unix

Tags: , , , , , , ,

The xxd is a bash command in Linux and macOS that is used to take a hexdump (convert a string to hex), or convert hex back to a string. To use xxd, just call it with a couple of options. Below, we’ll use the -p option to export into plain hexdump, and we’ll quote it and the <<< is to take input rather than a file name to convert (the default behavior), as follows:

xxd -p <<< "hey it's a string"

The output would be a hex string, as follows:

6865792069742773206120737472696e670a

Then use the -r option to revert your hex back to text. Since xxd doesn’t allow for a positional parameter to revert, we’ll simply echo the hex string and pipe it back into xxd, as follows:

echo 6865792069742773206120737472696e670a | xxd -r -p

And the output would be (is):

hey it's a string

Other useful options:

  • -b: Perform a binary dump instead of a hex dump
  • -e: what it looks like when a little endian takes a hex dump
  • -h: get help with the command
  • -len: stop after the defined number of characters
  • -u: use uppercase in the hex, instead of the default lower-case (doesn’t seem to actually work on macOS)
  • -v: grab the version of xxd

April 2nd, 2017

Posted In: Mac OS X, Mac OS X Server, Mac Security, Ubuntu, Unix

Tags: , , , , , ,

One of my favorite things about grabbing things with scripts is just how many ways (and sometimes how needfully or needlessly convoluted you can make them) to grab the same pieces of information. For example, something as simple as what hosts you use to resolve names on a Mac. There are a number of ways to grab what DNS server a device is using in macOS. So when you’re running a script you might choose to grab DNS information one way or another, according to what you’re after. Some of this might seem more complicated than it should be. And that’s correct…

resolv.conf

The /etc/resolv.conf file is updated automatically to look at what servers are used to resolve names used for DNS. The easiest way to see theses to simply cat it and grep for nameserver:

cat /etc/resolv.conf | grep nameserver

scutil

The next way we’ll grab DNS information is using scutil. Here, we use the –dns option, which outputs a lot of DNS stuffs, including all the built-in resolvers:

scutil --dns

To just grab the name servers:

scutil --dns | grep nameserver

We can also simplify the output to just the servers with awk:

scutil --dns | grep nameserver | awk '{print$3}'

networksetup

The second way is using networksetup. This command has an option to get a DNS server in (shocker) -getdnsservers. However, you have to list the interface for each. So below we’ll dump all interfaces into an array using -listallhardwareports and then read them in using a for loop and querying the name servers.

interfaces=( "$(networksetup -listallhardwareports | grep Hardware | cut -c 16-900)" )
for i in "${interfaces[@]}"
do
networksetup -getdnsservers $i
done

The one tricky thing in this one is I initially forgot to quote the interfaces as they went into the array, which meant each word of the interface was an item in the array and therefore the -getdnsservers option failed. Once I quoted, it was all happy. The other thing I can point out is I used cut instead of sed because it was easier to quote; however, it seems unlikely the name can be more than 890 characters, so I think it’s fine…

dig

You can also use dig. Here, you’ll query for a name without using an @ option, but omit everything but the line with the server that responded:

dig google.com | grep SERVER:

The output is kinda’ fug:

;; SERVER: 4.2.2.2#53(4.2.2.2)

For simpler output, we’ll use sed to constrain the output to just what’s between the parenthesis:

dig google.com | grep SERVER: | sed 's/^.*(//;s/)$//'

nslookup

nslookup is a tool similar to dig, used for querying names. We’ll basically do the same thing as above, just using awk as it’s just a standard position in a line:

nslookup google.com | grep Server: | awk '{print$2}'

system_profiler

Then there’s system_profiler, the command line interface for System Profiler. Here, we can query the SPNetworkDataType. This is going to produce a lot of output, so we can limit it to just the DNS servers using grep to constrain to just the lines we want and awk for just the columns in those lines, as follows:

system_profiler SPNetworkDataType | grep "Domain Name Servers:" | awk '{print$4}'

hosts

@knapjack added to use hosts. I had to use verbose mode to pull the local name server as follows:

host -v -t ns google.com | grep Received | awk '{print $5}'

ipconfig

Thanks to the lovely Allister (@sacrilicious), we also have ipconfig to add to the list:

/usr/sbin/ipconfig getpacket en0 2> /dev/null | grep name_ | cut -d' ' -f3-

There are tons of ways to find things in macOS. Do you have a way to find a DNS server that I didn’t think of here?

March 6th, 2017

Posted In: bash, Mac OS X, Mac OS X Server, Mac Security, Ubuntu

Tags: , , , , , , ,

There’s a great site out there called spellcheck.net that will check your bash scripts to verify that they are syntactically correct and offer some tips on fixing issues you may encounter. In the example below, I single quoted at the end of a quoted string, and… error
screen-shot-2016-12-14-at-9-44-00-am
Hat tip to Daniel MacLaughlin for posting this site.

December 14th, 2016

Posted In: Programming, Ubuntu

Given the increased reliance on XML in scripts and exchanging data, a number of different solutions leverage XML traversal options to get all the things done. We frequently use path to bring a file into a script or program, or accept input from STDIN. The most basic task that we then perform is simply selecting an item from that file or STDIN and then variabalizing it. One common tool that we use here is Path. XPath calls these objects nodes, and uses path expressions to select these nodes. A path expression is the path along the xml input that is followed to find a piece of data.

There are some pretty standard wildcards the can be used with xpath, where node() watches any node, * matches any element node, @* matches any attribute node, helping to constrain output.

Supported expressions include:

  • node: This is a text input that identifies the name of a node to start a relative search from – for example site would select all nodes in a structure with the name site
  • . Identifies the current node (kinda’ like pwd in a shell)
  • .. Starts at the parent of the current node – for example,
  • / Starts traversal from the root node – for example, /computer would select any nodes that falls underneath
  • /computer meaning that these are absolute paths
  • // Identifies the nodes in an XML structure that match a selection wherever they may be – for example
  • //computer would select all nodes that contain //computer and search for other expressions below those that you may identify such as: ‘xpath //computer/general/mac_address’
  • //* Selects everything
  • //computer/* Selects all the child element nodes of everything that starts with computer
  • @ Select an attribute in an XML structure – for example ‘xpath //computer/general/@’
  • [1] This predicate selects the first item (or whatever number is identified, so xpath
  • //computer[3]/general/mac_address would return with the mac address of the third computer
  • [@PATTERN] Constrains found sets, so ‘xpath //computer/general/[@mac_address]’ identifies all computers with an actual mac_address attribute
  • //[@PATTERN=VALUE] Constrains a found set to all items where the attribute contains the value, so ‘xpath //computer/general/[@mac_address=00]’ identifies all computers with an actual mac_address attribute that has the value of 00
  • //[@*] Selects only items with something in an attribute (non-null), so ‘xpath //computer/general/[@mac_address=@*] (btw, rather than use an =, you can use > or <)
  • | creates compound matches. So ‘xpath //computer/general/mac_address | //computer/general/name’ would grab the mac_address and name of every computer
  • [last()] Identifies the last item, so ‘xpath //computer[last()]/general/mac_address’ would return the last computer’s mac address
  • [last()-2] placing a negative number after the parenthesis identifies descending orders from the end of a found set – for example, //computer[last()-2] Selects the second to last computer

Overall, as you can see xpath really makes traversing XML structures simple. Other tools and languages have their own ways, but most are similar in syntax.

November 15th, 2016

Posted In: Mac OS X, Mac OS X Server, Ubuntu

Tags: , , , , ,

There are a lot of scripts stored on github. And you can run them directly by curling them into bash. To do so, you’ll need a link to the raw script (using the github page with the URL of the script brings in all the cruft, so you’ll need to find the raw text). To grab that, click on the page with the script and then right-click  on Raw, as seen here:

Screen Shot 2016-04-16 at 11.21.48 PM

Then, throw out a bash command followed by < and then the URL you just copied into your clipboard in parenthesis:

bash <(curl -Ls https://github.com/krypted/resetsoftwareupdate/raw/master/resetsoftwareupdate.sh)

April 20th, 2016

Posted In: Mac OS X, Mac OS X Server, Mac Security, Ubuntu, Unix

Tags: , , , ,

August 29th, 2015

Posted In: Mac OS X, Ubuntu, Unix

By default, the Chromium OS rootfs is read-only. If you boot the system in developer mode, you will be able to disable rootfs verification and modify existing files or write new files into the file system. Before you do this, note that your file system will no longer be verifiable (won’t checksum properly) and you’ll end up needing to restore a recovery image in order to get back to normal mode. So this might be a bit dangerous if you’re not using the device for something like regression analysis (why I needed to do this).

To make the file system writeable, first fire up a command prompt via crosh, by using Control-Alt-T and then running the shell command at the shell prompt:

shell

Then run the following shell script to remote rootfs verification and make the file system writeable:

sudo /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification

 

Then you can reboot and do whatever it is you need to do. For example, install a vine server and run automation scripts to do regression testing. Enjoy.

August 18th, 2015

Posted In: ChromeOS, Ubuntu, Unix

Tags: , , , ,

We tend to use a lot of commands in the Terminal app. That is, after all, what it’s there fore. And there’s a nice history of what we do. There are also a number of ways to view and manage the bash history. The simplest of which is the history command, which will show the previous commands run. Here, we’ll simply run it:

history

Keep in mind that this shows the history based on context, so if you sudo bash, you’ll potentially see a different history. You can also use the bash built-in fc command, which has the additional awesomeness of being able to edit and re-run commands from the history. To start, we’ll simply look at showing the last 16 commands using the -l option:

fc -l

You can also constraint entries in the output by specific line numbers. For example, to see lines 12 through 18, simply use them as the first two positions of the command after fc:

fc 12 18

You can load the history into an editor and remove or add entries using fc without any options:

fc

To exit the editor, hit control-z. I’ve written in the past about using substitution. For example, sudo !! to run the last command. fc can do some basic substitution as well. For example, use the -s to start substation and then enter a string, which will append whatever you like before a command. So the following would put sudo in front and re-run the previous command:

fc -s sudo

And let’s say that you were doing a find for a string of krypted. To then swap that string with charles:

fc -s krypted=charles

Overall, the bash history can be incredibly useful. I frequently pipe the output of a series of lines into a new file with a .sh at the end as a starting point for scripts and use these substitution options to save myself a bunch of time not retyping longer commands. Enjoy.

August 14th, 2015

Posted In: Mac OS X, Mac OS X Server, Mac Security, Mass Deployment, Ubuntu, Unix

Tags: , , , , , , , ,

Earlier, we looked at creating thousands of empty directories. Today, we’re going to get rid of them. But we need to get rid of only empty directories. To do so, we’ll use the find command:

find . -depth -type d -empty -exec rmdir {} \;

Now, we can put both into a script:

mkdir $(printf '%05d\n' {1..10000})
find . -depth -type d -empty -exec rmdir {} \;

July 29th, 2015

Posted In: Mac OS X, Ubuntu, Unix

Tags: , , , ,

Next Page »