CI Failure (key symptom) in ConfigProfileVerifyTest.test_config_profile_verify

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/13644

Module: rptest.redpanda_cloud_tests.config_profile_verify_test
Class: ConfigProfileVerifyTest
Method: test_config_profile_verify

test_id:    ConfigProfileVerifyTest.test_config_profile_verify
status:     FAIL
run time:   139.927 seconds

RpkException('command /opt/redpanda/bin/rpk cloud byoc gcp apply --redpanda-id=covdb27ek221n77eaavg --project-id=devprod-cicd-infra -v returned 1, output: ', '02:18:55.266  DEBUG  logging in using client credential flow\n02:18:55.267  DEBUG  Your existing auth token is still valid, avoiding re-authentication.\n02:18:55.276  DEBUG  looking for existing byoc plugin  {"exists": false}\n02:18:55.276  DEBUG  requesting GET https://cloud-api.ppd.cloud.redpanda.com/api/v1/clusters/covdb27ek221n77eaavg\n02:18:55.541  DEBUG  got response for GET https://cloud-api.ppd.cloud.redpanda.com/api/v1/clusters/covdb27ek221n77eaavg: 200 OK\n02:18:55.541  DEBUG  requesting GET https://cloud-api.ppd.cloud.redpanda.com/api/v1/clusters-resources/install-pack-versions/24.1.20240510145204\n02:18:55.667  DEBUG  got response for GET https://cloud-api.ppd.cloud.redpanda.com/api/v1/clusters-resources/install-pack-versions/24.1.20240510145204: 200 OK\n02:18:55.730  DEBUG  downloading byoc plugin  {"version": "24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde"}\n02:18:55.730  DEBUG  requesting GET https://dl.redpanda.com/public/rpk-plugins-preprod/raw/names/byoc-linux-amd64/versions/24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde/byoc-linux-amd64-24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde.tar.gz\n02:18:56.568  DEBUG  got response for GET https://dl.redpanda.com/public/rpk-plugins-preprod/raw/names/byoc-linux-amd64/versions/24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde/byoc-linux-amd64-24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde.tar.gz: 200 OK\n02:18:59.431  DEBUG  writing byoc plugin to /home/ubuntu/.local/bin\n2024-05-11T02:19:00.691Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tvalidation/validate.go:62\tChecking project...\n2024-05-11T02:19:00.691Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tvalidation/validate.go:68\tChecking project... PASSED\n2024-05-11T02:19:01.620Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tgcp/apply.go:73\tReconciling agent infrastructure...\n2024-05-11T02:19:01.912Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:190\tRunning apply\t{"provisioner": "bootstrap"}\n2024-05-11T02:19:08.295Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:200\tFinished apply\t{"provisioner": "bootstrap"}\n2024-05-11T02:19:08.295Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:190\tRunning apply\t{"provisioner": "network"}\n2024-05-11T02:20:00.747Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:200\tFinished apply\t{"provisioner": "network"}\n2024-05-11T02:20:00.747Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:190\tRunning apply\t{"provisioner": "cluster-network"}\n2024-05-11T02:20:38.183Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:200\tFinished apply\t{"provisioner": "cluster-network"}\n2024-05-11T02:20:38.183Z\t\x1b[34mINFO\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:190\tRunning apply\t{"provisioner": "agent"}\n2024-05-11T02:20:45.778Z\t\x1b[31mERROR\x1b[0m\t.rpk.managed-byoc\tcli/cli.go:193\tFailed to apply provisioner\t{"provisioner": "agent", "error": "failed running terraform apply: exit status 1\\n\\nError: Provider produced inconsistent result after apply\\n\\nWhen applying changes to google_service_account.redpanda_agent[0], provider\\n\\"provider[\\\\\\"registry.terraform.io/hashicorp/google\\\\\\"]\\" produced an unexpected\\nnew value: Root object was present, but now absent.\\n\\nThis is a bug in the provider, which should be reported in the provider\'s own\\nissue tracker.\\n"}\nFailed to apply provisioners: failed to apply provisioner agent: failed running terraform apply: exit status 1\n\nError: Provider produced inconsistent result after apply\n\nWhen applying changes to google_service_account.redpanda_agent[0], provider\n"provider[\\"registry.terraform.io/hashicorp/google\\"]" produced an unexpected\nnew value: Root object was present, but now absent.\n\nThis is a bug in the provider, which should be reported in the provider\'s own\nissue tracker.\n\n', 1, '')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 177, in _do_run
    self.test = self.test_context.cls(self.test_context)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/config_profile_verify_test.py", line 31, in __init__
    super().__init__(test_context=test_context)
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_cloud_test.py", line 25, in __init__
    self.redpanda = make_redpanda_cloud_service(test_context)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 4928, in make_redpanda_cloud_service
    return RedpandaServiceCloud(context,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1597, in __init__
    cluster_id = self._cloud_cluster.create(superuser=self._superuser)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 922, in create
    self._create_new_cluster()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_cloud.py", line 699, in _create_new_cluster
    self.utils.rpk_cloud_apply(_cluster_id)
  File "/home/ubuntu/redpanda/tests/rptest/services/cloud_cluster_utils.py", line 136, in rpk_cloud_apply
    out = self._exec(cmd, timeout=1800)
  File "/home/ubuntu/redpanda/tests/rptest/services/cloud_cluster_utils.py", line 107, in _exec
    return self.rpk._execute(cmd, env=self.env, timeout=timeout)
  File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 1147, in _execute
    raise RpkException(
rptest.clients.rpk.RpkException: RpkException<command /opt/redpanda/bin/rpk cloud byoc gcp apply --redpanda-id=covdb27ek221n77eaavg --project-id=devprod-cicd-infra -v returned 1, output: ; stderr: 02:18:55.266  DEBUG  logging in using client credential flow
02:18:55.267  DEBUG  Your existing auth token is still valid, avoiding re-authentication.
02:18:55.276  DEBUG  looking for existing byoc plugin  {"exists": false}
02:18:55.276  DEBUG  requesting GET https://cloud-api.ppd.cloud.redpanda.com/api/v1/clusters/covdb27ek221n77eaavg
02:18:55.541  DEBUG  got response for GET https://cloud-api.ppd.cloud.redpanda.com/api/v1/clusters/covdb27ek221n77eaavg: 200 OK
02:18:55.541  DEBUG  requesting GET https://cloud-api.ppd.cloud.redpanda.com/api/v1/clusters-resources/install-pack-versions/24.1.20240510145204
02:18:55.667  DEBUG  got response for GET https://cloud-api.ppd.cloud.redpanda.com/api/v1/clusters-resources/install-pack-versions/24.1.20240510145204: 200 OK
02:18:55.730  DEBUG  downloading byoc plugin  {"version": "24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde"}
02:18:55.730  DEBUG  requesting GET https://dl.redpanda.com/public/rpk-plugins-preprod/raw/names/byoc-linux-amd64/versions/24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde/byoc-linux-amd64-24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde.tar.gz
02:18:56.568  DEBUG  got response for GET https://dl.redpanda.com/public/rpk-plugins-preprod/raw/names/byoc-linux-amd64/versions/24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde/byoc-linux-amd64-24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde.tar.gz: 200 OK
02:18:59.431  DEBUG  writing byoc plugin to /home/ubuntu/.local/bin
2024-05-11T02:19:00.691Z	�[34mINFO�[0m	.rpk.managed-byoc	validation/validate.go:62	Checking project...
2024-05-11T02:19:00.691Z	�[34mINFO�[0m	.rpk.managed-byoc	validation/validate.go:68	Checking project... PASSED
2024-05-11T02:19:01.620Z	�[34mINFO�[0m	.rpk.managed-byoc	gcp/apply.go:73	Reconciling agent infrastructure...
2024-05-11T02:19:01.912Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "bootstrap"}
2024-05-11T02:19:08.295Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:200	Finished apply	{"provisioner": "bootstrap"}
2024-05-11T02:19:08.295Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "network"}
2024-05-11T02:20:00.747Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:200	Finished apply	{"provisioner": "network"}
2024-05-11T02:20:00.747Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "cluster-network"}
2024-05-11T02:20:38.183Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:200	Finished apply	{"provisioner": "cluster-network"}
2024-05-11T02:20:38.183Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "agent"}
2024-05-11T02:20:45.778Z	�[31mERROR�[0m	.rpk.managed-byoc	cli/cli.go:193	Failed to apply provisioner	{"provisioner": "agent", "error": "failed running terraform apply: exit status 1\n\nError: Provider produced inconsistent result after apply\n\nWhen applying changes to google_service_account.redpanda_agent[0], provider\n\"provider[\\\"registry.terraform.io/hashicorp/google\\\"]\" produced an unexpected\nnew value: Root object was present, but now absent.\n\nThis is a bug in the provider, which should be reported in the provider's own\nissue tracker.\n"}
Failed to apply provisioners: failed to apply provisioner agent: failed running terraform apply: exit status 1

Error: Provider produced inconsistent result after apply

When applying changes to google_service_account.redpanda_agent[0], provider
"provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected
new value: Root object was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

; returncode: 1>

JIRA Link: CORE-2924