Monitoring actual GPU memory usage

[ctuluhu]

Describe the problem the feature is intended to solve

I have several models loaded and not sure how can I know if Tensorflow still has some memory left. I can check using nvidia-smi how much memory is allocated by Tensorflow but I couldn't find a way to check loaded models usage.

Describe the solution

Tensorflow could provide some metrics for Prometheus about actual GPU memory usage by each loaded model.

Describe alternatives you've considered

None.

Additional context

I am not sure if this is actually a feature request or it can be done somehow at the moment.

[rmothukuru]

@ctuluhu ,
Can you please check this link and let us know if it helps.

[ctuluhu]

@rmothukuru
Thank you for your response but I didn't find what I am looking for in provided link.
What I need is a way to get the amount of free memory allocated by Tensorflow.

Issue eg: Tensorflow allocated 6GB memory, later I have loaded two models into Tensorflow memory, how can I know how much of this 6GB is used by loaded models and how much of this memory is free?

[peddybeats]

Hi there, we can easily export metrics that tell you host memory consumption on a per model basis but I think you're specifically looking for GPU's memory consumption/availability correct?
This is not straight forward but we will discuss it internally to understand exactly how difficult it would be an provide an update. Let us know if you have any additional info that you think might be useful for us to know!

[ctuluhu]

Hi, yes I am looking for a way to check GPU's memory availability. Such a feature would be great.

[troycheng]

We also met this problem. TF-serving occupied all GPU memory when it started and there is no way to know how much memory really needed for a specific model. If we deploy too much models in a server instance, sometimes it will hang up and do not response , all connections to it will timeout. Thus, for multiple models, we need to do lot of load test to decide which can be deployed together in one instance and which need to be deployed in another.