Hey there,
It’s Robin from CFD Engine & this week’s note is a cautionary tale about the day that spot ran out 😰
Back in Issue 002 I mentioned the wonders of the AWS spot market & how you can often grab compute at 60%-off retail. It’s a compelling discount that comes with a caveat – you will be kicked off your instance if someone is willing to pay more for it than you.
I’m pretty sure spot wasn’t built with long-running CFD jobs in mind, but it can work really well, and you can pocket a steep discount for nearly all of your runs.
What you might not know (& what might seem strange in the seemingly limitless world of AWS compute) is that sometimes there just aren’t any spare machines.
And on those days, with no discount to be had, full-price looks really expensive.
A little history
AWS don’t tell you how much spot capacity there is (or capacity in general, for that matter). But you used to be able to infer the state of the spot inventory, using the pricing history as a proxy.
In the olden days of spot you had to specify the maximum you were prepared to pay before starting your instance. People often bid way over the retail price to make sure they weren’t interrupted 🤷♂️ The spot price would then bounce around between the many different bids in the spot auction “book”.
If the price hadn’t been above 60¢ for the past month, then you could be pretty confident that instances had been sitting idle. If the graph was locked at $10 (or if the price history looked like a DES force trace) then you were in for a rough ride. Prepare to be kicked off your instance.
Not any more
A couple of years ago AWS adopted a new spot pricing model which is a little opaque & it’s not really an auction any more. Prices are updated less frequently (no more spiky pricing history) and the stated aim is to reduce the number of interruptions.
It seems to have had the desired effect, spot prices stay low & you seem to get biffed less frequently. However, our proxy into the spot capacity has gone. The price history graphs are locked at low-low prices, giving the illusion of spare (cheap) compute even when there is none. So, when spot runs out, it comes as a complete surprise.
This happened to me (again) this week. The instance price history was (& still is) flatlined at 60% off, but I couldn’t get (or when I could get, I couldn’t keep) any spot instances.
Full price time 💸
Key Takeaway
I’m writing this for me, as much as for you:
Pro Tip: Do not base your expected compute costs on the idea that you’ll be able to grab a 60+% discount at all times.
Any saving is a bonus. It’s one worth grabbing whenever possible, but it’s not one to bank on.
If you get accustomed to paying the spot price (like I am) then it makes the retail price feel expensive, even though you’ve been netting a discount for most of your spend.
There’s no easy way to avoid this. But, if it becomes more frequent, then you could try…
Different instance types
Not all instances (even similar ones) are subject to the same demand. The more flexible you can be with your choice of instance, the more likely you are to be able to find some spare capacity.
Different region
Not all geographical regions have the same demand or the same capacity. You could try moving your workloads to another region, although I’m not sure how to identify the quieter ones. Maybe try the regions where your usual instances are cheapest &/or the spot discounts are the deepest?
Different architecture
I’ve not tried this one (yet), but you might be able to make significant savings by switching to ARM instances. Not something you’ll do on a whim, but not as difficult as it might sound (apparently).
A heads up
I just wanted to let you know that #NoSpot is a real possibility. Whilst AWS is, for our purposes, essentially bottomless, spot instances are not.
If you’ve been eyeing spot prices (& their histories) and you’re tempted to conclude that it’s the de-facto compute cost. It’s not. It’s a welcome bonus. But, sometimes there’s no spot & you’ll have to pay retail.
What’s been your experience using spot instances? Are they worth the extra effort? Do you get interrupted often?
Or are you happy to pay retail to avoid all of this nonsense? Let me know, especially if you’ve discovered a secret stash of spot capacity in Oregon – your secret will be safe with me 🤐
Until next week, may the spot be with you,
