You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
prod-titiler-xarray started returning 500s with the error "command not allowed when used memory > 'maxmemory'.". I found this while trying to make changes to the VEDA UI codebase to accommodate a new titiler-cmr layer type.
looking at cloudwatch logs I could see more, that this was a redis error "redis.exceptions.OutOfMemoryError: command not allowed when used memory > 'maxmemory'."
decided to try rebooting, since I believed this would clear the cache
rebooting worked to resolve the errors but would still be good to know if it ran out of memory from all existing cached requests or if it was just one request that was too large
from the graph of database memory usage, you can see a steady increase to capacity, so I believe this was just a result of reaching the total capacity of the in-memory database.
Solutions:
increase capacity via node size. right now it's using a t3.small which has 1.37GB memory. I think we should do this if we're planning to use titiler-xarray in production, which we don't have plans to do now but will likely do so in the future.
Problem:
Solutions:
@vincentsarago @sharkinsspatial do you think the ttl should be 5 minutes, a day, a week?
The text was updated successfully, but these errors were encountered: