Hey there,
It’s Robin from CFD Engine & this week I’m combining my comfy place (the command line) & my CFD place (AWS) with an overview of the AWS command-line interface.
If you’ve experimented with AWS, you might’ve done everything via the web console. It’s vast, but relatively intuitive & you can get most things done pretty easily.
So why, if there’s a perfectly serviceable web portal, would you want to do anything from the command-line?
- you’re already in the terminal doing OpenFOAM bits;
- you can use the commands to automate some repetitive jobs;
- you can use the commands in your run scripts to give your remote machines access to extra services;
It’s super-easy to install, (kind-of) easy to use & overall well worth exploring if you’re doing more of your CFD on AWS.
Here’s an overview of what I do with it, how to get it & a couple of tips if you’re planning on taking it for a spin.
What is it?
The name pretty much covers it – it’s a command-line interface to Amazon Web Services – almost anything you need to do on AWS can be done with this toolkit.
It’s open-source, written in Python & made available for Mac, Linux & Windows. It ships with it’s own Python runtime, has no external dependencies & requires no Python skillz to use.
What I do with it
I stick to doing simple, everyday stuff:
- requesting new instances (on-demand & spot)
- finding the address of new machines
- auto-tagging instances for cost reporting
- transferring/syncing files to/from storage (S3) both locally & on EC2
- checking on the status of instances (see below)
- what’s currently running?
- what’s stopped / hibernating?
- what’s recently been terminated?
- what project / job is that instance is assigned to?
Above, an example of using the ec2 describe-instances command with queries & table output.
Everything else I do in the web console, but these everyday tasks are nice to fire off from the command line.
How to get it
The install instructions are here – cut-&-paste a single line & you’re pretty much done.
Follow the configuration steps to tell it who you are, what credentials to use, which region you’re in, plus a couple of other little bits & you’re set.
Its also worth installing it on your remote instances. Really useful for transferring files to & from deep storage (S3) and for tagging your resources.
If you do want to use it on EC2 instances, I have two notes:
- include it in your custom CFD machine image to avoid having to install & configure it every time.
- don’t store your credentials (secret key etc) in your machine image. There’s an alternative that uses IAM to control what the CLI tools can (& can’t) do from your EC2 instance. Not super-straightforward, but worth it, promise.
The documentation is comprehensive (as in VERY long), so maybe start by trying out some high-level storage commands and build up from there.
NB
A couple of other observations having used it for a while:
- it doesn’t include the tools for managing clusters of instances. That use-case has it’s own excellent command-line tool called Parallel Cluster.
- the output can be a bit clunky (especially in JSON format) but it’s easily manipulated by other command-line tools.
- it’s not quite so slick if you’re working across multiple AWS regions (but neither is the web console) & I guess if you’re using multiple regions you’re probably an AWS pro anyway.
Wrap up
If you’re just starting out on AWS, then the web console has you covered. But, as you expand your AWS usage, take note of the things you do often (starting instances, uploading/downloading etc) and give them a try in the CLI. The easy ones can be wrapped into your scripts & the tricky ones can wait for another day.
Do you already use the CLI? What nifty commands do you use that I’m missing out on? Or did you try it & found it more trouble than it was worth?
Let me know, I’m always keen to hear your thoughts & spark a discussion.
Until next week, stay safe,
