Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mem graph doesn't update after crash of the program when used all memory #520

Open
martincerven opened this issue May 9, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@martincerven
Copy link

Describe the bug

When using models that consume all available memory and eventually crash, the jtop doesn't update, only after ctrl+c, jtop it shows available memory correctly.

To Reproduce

Steps to reproduce the behavior:

  1. use audiocraft on Orin Nano with monitor
  2. When running jupyter cells, it will eventually run out of memory and crash container
  3. Mem won't update, only after closing and reopening

Expected behavior

I would expect I won't have to restart jtop

Additional context

When running any (pytorch) models that consume more memory than available, the jtop doesn't update after they eventually crash.

Board

Happened to both Orin Nano 8GB devkit, when trying audiocraft, and AGX 32GB devkit when running larger models (in containers)

@martincerven martincerven added the bug Something isn't working label May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant