Race between status update and instance creation can cause resource leak

[cg505]

The cloud may not immediately reflect instance creation after sending the request. This can lead to the following race:

1. [process 1] sky launch a new cluster
2. [process 1] sky launch grabs the cluster lock
3. [process 1] sky launch adds cluster to the state database as INIT
4. [process 2] sky status -r is launched and tries to grab the cluster lock
5. [process 1] sky launch makes the cloud API calls to create the instances
6. [process 1] sky launch immediately dies (e.g. is SIGKILLed), maybe even before getting a response from the cloud API. The lock is released.
7. [process 2] sky status -r obtains the lock
8. [process 2] sky status -r queries the instances, which the cloud API does not show yet
9. The cloud creates the instances. Future queries will show them.
10. [process 2] Since no instances were found, sky status -r thinks the cluster is terminated and deletes it from the state database.

Apparently AWS takes some very small amount of time to reflect instance state after instance creation.
I was able to reproduce in AWS by crashing sky launch as soon as the create instance request is sent (step 6 in the above race description).
Other clouds may have similar behaviors. We really can't ensure the request is received before the lock is released in this case. For instance, the create instance request may still be on the wire and could be racing with the list instance request.

This is pretty hard to fix reliably. However, I think we can reasonably assume there's very limited set of cases where this could arise:

• We are terminating or checking the status of a cluster that was recently created. (Say, in the past 60s.)
  • Typically there is not a lot of time between cluster "creation" (adding to state database) and actually creating the instances, but we may want to tune this.
• The cluster is INIT, but we don't see any instances in the cloud. Also, we don't see any previously terminated instances since the cluster was created.

In this case, we can just wait a very short amount of time (maybe as short as 1 second, but hard to know) and double check that the instances do not exist.

[cg505]

This is possibly the cause of #4410.