Issue 047 – May 22, 2021

Custom Machine Images for CFD

Hey there,

It’s Robin from CFD Engine and I thought I’d give you an overview of building your own custom machine image for AWS.

I’ve had a few emails about it, it’s not hard & it’s worth doing if you’re going to run more than just the odd instance.

This isn’t a deep-dive, but it should give you an overview of why it’s a good idea & how to do it.

Machine Image?

We’ve talked about virtual machines before & cloud computing relies heavily on them – a layer of insulation between what you experience as your machine and the actual hardware it’s running on.

Your virtual hardware also needs some software, and that lives in a machine image. It’s often a snapshot of an operating system, but it can be much more than that.

When your virtual machine (the hardware) and your machine image (the software) are combined (behind the scenes) you get something that looks (to you) like your very own server, running your choice of software.

That’s about as complicated as we need to get:

Virtual Hardware + Machine Image = Instance

Why build your own?

You’re probably quite particular about the software in your CFD toolkit.

You probably need particular versions of OpenFOAM, or ParaView, or Python to make your CFD-magic happen.

Your particular versions might not always be the most recent ones 🤫

You might also have in-house codes, or bespoke solvers that are super-custom.

So there’s a good chance that you won’t find your unique mix of tools pre-built on the AWS marketplace.

Unless you want to install your toolkit every time you start a machine (I don’t) you’ll probably want to create a custom machine image, with everything you need, primed & ready to go.

It’s not difficult, if you’ve installed codes locally, you can probably do it in the cloud.

Steps To Build-Your-Own

There’s not too much to it – here’s how you can build your first custom machine image on AWS…

  • Launch a new instance running a recent official Ubuntu AMI, e.g. 20.04 LTS
    • It can be a small instance if you’re just installing binaries, or a bigger one if you need some crunch power for compiling things.
    • Make sure to request enough storage for everything you’re going to install / compile.
  • Connect to the instance and install the software you need. For me that includes:
    • the OpenFOAM.com Ubuntu binaries (installed using apt-get);
    • headless ParaView binaries (downloaded from ParaView.org);
    • AWS command-line tools (using an install script);
    • zip & ffmpeg binaries (installed using apt-get);
  • Test everything works as expected (maybe run a tutorial?);
  • Stop your instance – it’s not required, but the next step will kick you out anyway;
  • From the EC2 management console, select your instance, then from the Actions drop-down menu, select Image & Templates - Create Image
  • Give your image a name (you can’t change this later, so be sensible) & add a description (list what’s installed – it’s optional, but recommended)
  • Click the Create Image button & go get a coffee (it takes a few mins)
    • You can check on progress by selecting AMIs from the sidebar & waiting for the status to become available

Now you can launch a new instance based on your custom AMI with everything installed & ready to go – it’s CFD time.

Don’t forget to terminate instances you no longer need.

Parallel Cluster

Here’s a top tip I picked up from AWS.

Base your image on one of the AWS parallel cluster images (as opposed to the official Ubuntu image I chose). These images are set up with all the tools you need to deploy a cluster of instances. So, if you customise one of them, you’ll not only have all of the tools you need, but they’ll also be ready to be deployed in a cluster. Bonus 👍

You will need to compile OpenFOAM in this case, in order to take advantage of the MPI libraries that AWS have configured for you. Compiling OpenFOAM isn’t my preference, but it’s not difficult & (in this case) it’s worth the extra effort.

I should probably adopt this method myself, but I don’t run clusters very often 🤔

EC2 Image Builder

AWS has a tool for building custom machine images (they have a tool for almost everything). I’ve played with it & it’s good but, for me, it’s not worth automating something I only do once every six months.

Costs

Your AMIs are stored as snapshots in your AWS account & you’re charged something like 5¢ per GB per month for storing them. It’s probably not a problem, but bear it in mind if you go crazy & make LOADS of custom AMIs or you have big root volumes.

🎵 “I like big roots & I cannot lie” 🎵

Give it spin

If you want full control over the software you have on your cloud instances, and you don’t want to have to install it every time you start a new machine, then you should give this a go.

What unusual tools would be in your custom image? What do you use that you would never find in a pre-built CFD machine image? Hit reply & let me know.

I’m always keen to hear from you, especially if you have questions or comments on anything you read in these issues, or you just want to chat CFD.

Until next week, stay safe,

Signed Robin K