Droplet re-creation fails with 404 error

[karlkfi]

Bug Report

Describe the bug

GPU droplet re-creation doesn't tolerate 404s.

Affected Resource(s)

• digitalocean_droplet

Expected Behavior

Re-creation should destroy, wait for destruction to complete, then re-create, without erroring.

Actual Behavior

$ terraform apply .tfplan
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-300"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-300"]: Still creating... [00m10s elapsed]
╷
│ Error: Error waiting for droplet (536572909) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536572909: 404 (request "59af37fd-bb3b-4926-a107-1faa751bbe88") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-300"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│

Steps to Reproduce

1. Create a GPU droplet
2. Change the config in tf to cause re-creation
3. terraform apply

Terraform version
v1.14.2

Provider version
v2.71.0

[karlkfi]

Another example log:

$ terraform apply .tfplan
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-317"]: Destroying... [id=532460776]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-312"]: Destroying... [id=532460766]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-315"]: Destroying... [id=532460772]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-310"]: Destroying... [id=532460747]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-313"]: Destroying... [id=532460769]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-316"]: Destroying... [id=532460773]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-319"]: Destroying... [id=532460783]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-318"]: Destroying... [id=532460781]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-314"]: Destroying... [id=532460771]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-311"]: Destroying... [id=532460764]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-312"]: Still destroying... [id=532460766, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-317"]: Still destroying... [id=532460776, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-315"]: Still destroying... [id=532460772, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-316"]: Still destroying... [id=532460773, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-310"]: Still destroying... [id=532460747, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-314"]: Still destroying... [id=532460771, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-313"]: Still destroying... [id=532460769, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-319"]: Still destroying... [id=532460783, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-311"]: Still destroying... [id=532460764, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-318"]: Still destroying... [id=532460781, 00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-317"]: Still destroying... [id=532460776, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-312"]: Still destroying... [id=532460766, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-315"]: Still destroying... [id=532460772, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-318"]: Still destroying... [id=532460781, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-316"]: Still destroying... [id=532460773, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-313"]: Still destroying... [id=532460769, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-310"]: Still destroying... [id=532460747, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-314"]: Still destroying... [id=532460771, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-319"]: Still destroying... [id=532460783, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-311"]: Still destroying... [id=532460764, 00m20s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-319"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-319"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-314"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-314"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-313"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-313"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-312"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-310"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-312"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-310"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-316"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-316"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-311"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-311"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-317"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-317"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-315"]: Destruction complete after 22s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-315"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-318"]: Destruction complete after 23s
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-318"]: Creating...
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-319"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-314"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-313"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-310"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-312"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-316"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-311"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-317"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-315"]: Still creating... [00m10s elapsed]
module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-318"]: Still creating... [00m10s elapsed]
╷
│ Error: Error waiting for droplet (536588011) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536588011: 404 (request "f5ea1f49-86de-41d7-88b8-09d29880669a") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-312"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536588008) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536588008: 404 (request "1fedd42e-0386-9a5d-b434-924eb2e7ff03") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-310"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536588012) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536588012: 404 (request "4300de8a-04b8-4cd2-9e49-0c64f7f2bfca") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-315"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536587999) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536587999: 404 (request "56a2b858-0ce1-4265-8216-09267c21f98f") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-313"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536588010) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536588010: 404 (request "b1654e27-58eb-4cb9-8d64-4b47c83c3373") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-316"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536588013) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536588013: 404 (request "02fe8400-4495-406f-832d-3f74ce0314bb") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-311"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536588018) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536588018: 404 (request "47baceb3-2e46-4483-b969-5fcde8aa3dfe") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-317"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536588017) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536588017: 404 (request "5fdc08c8-ed27-4bc9-ab31-45e5c5e66b3a") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-318"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536587997) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536587997: 404 (request "486149a7-63f3-43ed-b365-bb3663185954") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-319"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵
╷
│ Error: Error waiting for droplet (536588000) to become ready: Error retrieving droplet: GET https://api.digitalocean.com/v2/droplets/536588000: 404 (request "fe883092-0146-449a-b1da-e6f47ff7953f") The resource you were accessing could not be found.
│
│   with module.gpu_compute.digitalocean_droplet.gpu_droplet["mi325x8-314"],
│   on ../../../modules/do/gpu_compute/main.tf line 1, in resource "digitalocean_droplet" "gpu_droplet":
│    1: resource "digitalocean_droplet" "gpu_droplet" {
│
╵

[karlkfi]

After some debugging with DO support, it seems like the problem is that the terraform provider creates and polls GET for status, when it should be reading the "action" ID from the create call and reading the response from the GET to see the status of the create action, not just erroring 404.

The problem from DO's side is that these are droplets on contracted dedicated nodes, so if the create happens too son after the delete, there's no capacity to bring up the droplet. So it fails placement asynchronously, causing subsequent GETs to 404 (with a response payload of action status).