-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unused vector name cost storage and (maybe)memory #4112
Comments
BTW, I want to know if there has any safe and efficient way to remove "gemini:models/embedding-001" in vector schema and those files in storage? |
Hey @hugh2slowmo, you are right, the current implementation of multiple vectors per point reserves space for all vectors even if some of them are not created. It is possible that we would make some optimizations for corner cases like yours, but overall that's the idea. I recommend to use on_disk option, so it would be only the disk which is used |
Thanks for your reply @generall, so in current state, the better solution to reduce mem&disk usage with a large collection, is not declaring vector config that are not being used currently when creating collection, am i get it right? |
from the disk perspective, it doesn't matter if you have one collection or many, but not creating empty vectors will definitely help |
Hello, i have a big collection which info shows below:
I have 2 vectors setting, "openai:text-embedding-ada-002" which is in current using, and "gemini:models/embedding-001" is design for future usage and i did not put any vectors in it.
what confusing me is that, even in current config which optimize for memory usage(vector&payload on_disk), this collection still consume over 180GB memory, dose the payload index costs? or empty vectors has memory costs? I'll appreciate it if you guys can give some guides cuz i cannot find docs about it by myself.
And another thing i found out is that unused vector name costs disk storage:
Dose it the normal case? or some thing wrong already happened on the collection causes it?
PS: currently i'm using qdrant version 1.8.3
Thx!
The text was updated successfully, but these errors were encountered: