[Core] Possible memory leak in gcs_server

[ScottShingler]

What happened + What you expected to happen

Running Ray Core using a driver that repeatedly invokes a method on an actor results in a continuous overall increase in the peak RSS memory usage of the ray/core/src/ray/gcs/gcs_server process:

Over the course of ~112 hours, the values of the peaks increased from an initial value of ~595 MB to ~662 MB. That works out to roughly 598 KB per hour.

This plot and the data used to create it can be downloaded here.

Versions / Dependencies

ray 2.20.0

Reproduction script

I have created a repo that contains the reproduction script, as well as scripts to collect and plot the data: https://github.com/Prolucid/ray-memleak-repro.

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[lmsh7]

In my own scenario of using uvicorn + fastapi + single-node ray, similar situation occurs. After high QPS testing, the gcs server did not fall back like uvicorn did

[lmsh7]

In my own scenario of using uvicorn + fastapi + single-node ray, similar situation occurs. After high QPS testing, the gcs server did not fall back like uvicorn did

However, I found some solutions in this issue: export RAY_task_events_max_num_task_in_gcs=100 can significantly reduce the memory usage of GCS."

[lmsh7]

It is also similar to this issue #43253

[rynewang]

Per #43253 can you retry with export RAY_task_events_max_num_task_in_gcs=100?

[ScottShingler]

I'll set up a test to run over the weekend and report back.