CI Failure (key symptom) in OMBValidationTest.test_max_connections

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/13644

Module: rptest.redpanda_cloud_tests.omb_validation_test
Class: OMBValidationTest
Method: test_max_connections

test_id:    OMBValidationTest.test_max_connections
status:     FAIL
run time:   152.920 seconds

SSHException('Error reading SSH protocol banner')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/paramiko/transport.py", line 2327, in _check_banner
    buf = self.packetizer.readline(timeout)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/paramiko/packet.py", line 381, in readline
    buf += self._read_timeout(timeout)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/paramiko/packet.py", line 618, in _read_timeout
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 341, in test_max_connections
    benchmark.start()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/service.py", line 265, in start
    self.start_node(node, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 265, in start_node
    self._create_workers()
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 250, in _create_workers
    self.workers.start()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/service.py", line 265, in start
    self.start_node(node, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 67, in start_node
    self.clean_node(node)
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 136, in clean_node
    self.stop_node(node, allow_fail=True)
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 129, in stop_node
    node.account.kill_process("openmessaging-benchmark",
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 422, in kill_process
    pids = [pid for pid in self.ssh_capture(cmd, allow_fail=True)]
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 41, in wrapper
    return method(self, *args, **kwargs)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 331, in ssh_capture
    client = self.ssh_client
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 217, in ssh_client
    self._set_ssh_client()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 41, in wrapper
    return method(self, *args, **kwargs)
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner

JIRA Link: CORE-2926

[rpdevmp]

Cloud Issue:

02:18:56.568  DEBUG  got response for GET https://dl.redpanda.com/public/rpk-plugins-preprod/raw/names/byoc-linux-amd64/versions/24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde/byoc-linux-amd64-24.1.20240510145204-sha256.95e8ab37ef37e401620eadf8bd7aafde.tar.gz: 200 OK
02:18:59.431  DEBUG  writing byoc plugin to /home/ubuntu/.local/bin
2024-05-11T02:19:00.691Z	�[34mINFO�[0m	.rpk.managed-byoc	validation/validate.go:62	Checking project...
2024-05-11T02:19:00.691Z	�[34mINFO�[0m	.rpk.managed-byoc	validation/validate.go:68	Checking project... PASSED
2024-05-11T02:19:01.620Z	�[34mINFO�[0m	.rpk.managed-byoc	gcp/apply.go:73	Reconciling agent infrastructure...
2024-05-11T02:19:01.912Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "bootstrap"}
2024-05-11T02:19:08.295Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:200	Finished apply	{"provisioner": "bootstrap"}
2024-05-11T02:19:08.295Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "network"}
2024-05-11T02:20:00.747Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:200	Finished apply	{"provisioner": "network"}
2024-05-11T02:20:00.747Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "cluster-network"}
2024-05-11T02:20:38.183Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:200	Finished apply	{"provisioner": "cluster-network"}
2024-05-11T02:20:38.183Z	�[34mINFO�[0m	.rpk.managed-byoc	cli/cli.go:190	Running apply	{"provisioner": "agent"}
2024-05-11T02:20:45.778Z	�[31mERROR�[0m	.rpk.managed-byoc	cli/cli.go:193	Failed to apply provisioner	{"provisioner": "agent", "error": "failed running terraform apply: exit status 1\n\nError: Provider produced inconsistent result after apply\n\nWhen applying changes to google_service_account.redpanda_agent[0], provider\n\"provider[\\\"registry.terraform.io/hashicorp/google\\\"]\" produced an unexpected\nnew value: Root object was present, but now absent.\n\nThis is a bug in the provider, which should be reported in the provider's own\nissue tracker.\n"}
Failed to apply provisioners: failed to apply provisioner agent: failed running terraform apply: exit status 1

Error: Provider produced inconsistent result after apply

When applying changes to google_service_account.redpanda_agent[0], provider
"provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected
new value: Root object was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

; returncode: 1>