Hey there,
It’s Robin from CFD Engine & I’ve got an under-rated, AWS-related YouTube recommendation for you this week. But, instead of just linking to the channel, I watched a whole bunch of their videos & summarised my takeaways 🥡
The channel in question is HPC Tech Shorts where the AWS engineering team chat about all sorts of HPC-geekery, not just AWS stuff.
There are case studies, CFD benchmarks, open-source projects from adjacent fields & plenty of how-to’s – all presented as a water-cooler chat between peers.
Here are those executive summaries to give you a flavour of what you’re missing…
CFD benchmarks
AWS are pretty serious about running CFD on their HPC-friendly instances & they’re keen to demonstrate it with lots of cost-per-run & speed-up charts:
📺 : OpenFOAM & StarCCM+
IMHO: the OpenFOAM scaling didn’t look great in this video (esp. next to Fluent & StarCCM+). They were all running different cases & different model sizes, but the OpenFOAM ones strayed a bit far from ideal scaling lines for my liking.
📺 : Fluent & Under-Populated Nodes
Under-populating your nodes (running on half the cores, but with twice as many nodes) looks like a speed-up (in Fluent at least). It also looks (to me) like you’re running on the wrong instances.
The chart showing a flat “run-cost vs node-count” relationship was pretty compelling though – run your jobs as fast as you like without it costing you any more money 🤔
NICE things
Back in 2016, AWS bought an Italian software company called NICE & made their tools free to use on AWS. However, they didn’t really bother to tell us what those tools were, or why we should care.
Turns out, they’re pretty useful, but I only found that out thanks to these conversations with their developers:
📺 : Supercomputing Visualization good enough for the most demanding gamers
If streaming 4K @ 60fps, on a secure protocol, across a ropey internet connection, with support for 4 monitors, webcams & printers is what passes for a remote desktop these days, then NICE DCV has it covered.
Sounds like it should be enough for a remote ParaView session or editing a few dictionaries in gedit 😉
📺 Make your HPC users highly productive using EnginFrame with AWS HPC Connector
NICE EnginFrame is a single web portal that can manage your jobs (& compute) in the cloud and on-premise. It works with your existing job schedulers & has some nifty tools like automatic uploads for remote jobs & rules that control which jobs run where (& when).
It uses Parallel Cluster to do the heavy lifting on AWS so, if you’re already using pcluster, you can copy over your configs & have browser-based job management, without having to build it for yourself.
Systems geekery
These videos are especially for those of us that find talk of containers & interconnects riveting 🤓
📺 : How EFA works and why we don’t use infiniband in the cloud
AWS wrote their own network protocol to take advantage of their big fat network (as opposed to just infiniband-ing some instances). It turns out that for our use case (& CFD was one of their design points) it’s not just how fast you can bounce a single packet over the wire that matters. It’s much more about how slow the slowest packet goes & how things hold up under high load or when something goes wrong.
📺 : Containers in HPC - what they fix & what they break (and how to fix that, too)
Software containers (like Docker) are much more flexible than I’d previously thought & it’s that flexibility that causes problems when using them for HPC. For example, I did not know that it’s a hassle to get containers to play nicely with shared storage.
📺 : Containers, Episode II - the Runtimes Strike Back
It seems that running containerised MPI jobs across multiple nodes might be easier than I’d been led to believe, thanks to some clever HPC runtimes. There’s loads more runtime info here but beware, this rabbit hole runs deep.
Check out the rest
In brief, I watched ~5 hours of HPC Tech Shorts (so you didn’t have to) and I reckon it’s well worth a sub (or at least a browse) even if you don’t run your CFD on AWS.
Let me know if you check out any of their other videos & if you pick up some interesting nuggets along the way.
Do you have any CFD-related YouTube recommendations I should check out while waiting for the next HPC Tech Short?
Until next week, stay safe,
