Hey there CFD friend,
Time flies when you’re writing a weekly email - who knew? Thanks for being one of the first subscribers, you are appreciated 🙏
When I was thinking about how to make these emails helpful for both of us, I stumbled across the idea of having a revolving theme. I can’t share the client work I’m doing, but I can work on something for us & report back.
The idea is this; I’ll pick a CFD theme & spend a couple of months working on it in the background. And when the time’s up, I’ll come back to you with something useful for your CFD toolbox 🤞
I won’t write about it every week though. I’ll give you a heads-up at the start & a summary at the end, the choice cuts.
Before I outline the first theme, I realised that I mentioned a couple of things in last week’s email which I should’ve elaborated on. Let me fix that.
Spot. The difference.
I mentioned “spot instances” last week, quite possibly the best thing about AWS. But what if you’ve never heard of them?
Spot instances are cheap compute. The spot market is AWS’ clearing house for unused instances. Rather than let their hardware sit idle, they sell it off at a deep, but fluctuating, discount. The discount changes by the minute, but you often can find CFD-capable instances at more than 60% off retail. Nice.
There’s always a catch. In this case, if someone’s willing to pay full price for your instance, then you can be kicked off with just 2mins notice. Not the end of the world for little batch jobs. But pretty painful if it happens just before the end of your overnight job 🤦
Definitely worth digging into if you aren’t already using them.
“One Job, One Instance”
The other thing I should have explained better was the idea of “running single jobs on single instances.” It’s not exactly complicated, the name pretty much covers it. But, if you’re used to running CFD on a cluster with a queuing system, then it’s the opposite of that.
In this way, each CFD job is run on a freshly-booted single instance. No queue. No waiting for a machine to finish another job. The instance is then discarded as soon as the simulation has finished. No machine ever runs two jobs.
It’s easy to manage, even for multiple clients & projects. Everything is sandboxed. Cost reporting is a doddle. And different projects can run different code versions if needed. But…
…run times are longer. I’m using less cores than you would with a cluster. There’s also the chance that, if you have a big model, it might not even fit on a single machine. It’s horses for courses, but that’s how I do it.
It’s the way I’ve been doing it since I started running CFD on AWS in 2011. I even produced a course a couple of years ago to show others how to do it.
It showed AWS beginners how to build a custom machine image loaded with their essential CFD tools. Plus, how to add storage and how to start (& connect to) instances. A simple setup that could do useful CFD, not just tutorials.
I don’t sell the course any more. Changes to the AWS web console, meant that the walkthrough videos stopped making sense. The concepts are still good, but the tools have moved on. Which brings me to our first theme…
OpenFOAM on AWS: The Basics (2020)
I’m going to take the concept –“one job, one instance”– and update the methods & tools I taught in the course to be…a bit more 2020.
What does a simple OpenFOAM on AWS workflow look like now? How do the pieces fit together? Does it still make sense?
I’ll let you know.
Let me know what you think about the “themes” idea & definitely drop me a note if you have any ideas for future themes.
I’ll be back next week with my take on OpenFOAM v2006 & the trouble with 6-month release cycles.
Until then, stay safe,
