Reply To: ECS and spot instances – Spot instance draining

[juano1985]

Hi Ricardo,

Thanks for your response. I was just reading a bit more on this and came across this article that explains the behaviour, it’s pretty outdated (2018) so I guess maybe there have been quite some fixes/updates since then. I understand then that just like you say, there could be a case where that instance type is not available in that AZ and new spot instances wouldn’t be able to spin up, leading to an outage. Let me know what you think:

https://medium.com/@kevintruckenmiller/aws-spot-instances-and-ecs-b61c5802b375

“Don’t Take an Outage

One problem remains with Spot Fleets, when a spot instance gets outbid in a particular Availability Zone, you have to handle how the instance reacts to the underlying change. If you don’t, you could take an outage because the instance will be terminated and the container riding on top of the instance won’t be notified. On ECS you have the distinct advantage to set your instance to DRAINING. When you set your AWS instance to drain, you tell the host underlying the containers to send the containers a SIGTERM, and you also tell the load balancers target groups to drain the connection of the container so that new traffic doesn’t flow to the containers on that instance. This allows the scheduler to then spin up containers on the other instances within the cluster, and not take an outage on the actual instance that’s been outbid.”

Thanks!

Juan