Open
Description
Problem:
- prod-titiler-xarray started returning 500s with the error "command not allowed when used memory > 'maxmemory'.". I found this while trying to make changes to the VEDA UI codebase to accommodate a new titiler-cmr layer type.
- looking at cloudwatch logs I could see more, that this was a redis error "redis.exceptions.OutOfMemoryError: command not allowed when used memory > 'maxmemory'."
- decided to try rebooting, since I believed this would clear the cache
- rebooting worked to resolve the errors but would still be good to know if it ran out of memory from all existing cached requests or if it was just one request that was too large
- from the graph of database memory usage, you can see a steady increase to capacity, so I believe this was just a result of reaching the total capacity of the in-memory database.

Solutions:
- increase capacity via node size. right now it's using a t3.small which has 1.37GB memory. I think we should do this if we're planning to use titiler-xarray in production, which we don't have plans to do now but will likely do so in the future.
- Add a ttl to cache.set (https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Strategies.html#Strategies.WithTTL). Should it be 5 minutes? A week?
- Add an alarm to cloudwatch to notify us when database memory usage is near capacity
@vincentsarago @sharkinsspatial do you think the ttl should be 5 minutes, a day, a week?
Metadata
Metadata
Assignees
Labels
No labels