Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

发现Poalris Server运行状态正常情况下,仍存在客户端服务实例注册异常,同时检查客户端服务实例运行正常 #1395

Open
flashbai opened this issue Nov 15, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@flashbai
Copy link

Describe the bug

  1. 问题现象:在polaris server组件内存未溢出、CPU未打满、polaris-server服务pod运行正常未发生重启情况下,发现客户端服务实例注册polaris异常,同时在客户端服务的相关日志中发现如下报错:
    image
  • 另外Polaris Server发现内存一直上涨,但一直没有超过mem limit的8G限制
  • 同时将Polaris server服务Pod的replica副本数调整为1后,再重启polaris server,发现客户端服务注册正常,同时观察后续Polaris server内存也没有上涨;
  • 另外发现3副本时Polaris server中存在如下报错信息:
    polaris-health-check-error

To Reproduce
Steps to reproduce the behavior.

  1. Polaris Server部署版本:1.18.1
  2. 部署方式:Polaris Server组件通过K8S Helm部署,3副本,cpu limit为4c, mem limit为8g;
  3. Polaris监控:检查Polaris Server组件cpu、mem、网卡流量、负载等相关监控指标正常,唯一异常点是内存监控指标持续上涨,但内存使用率未到limit上限,同时没有发生过OOM;
  4. 客户端服务与polaris部署在同一个k8s集群,同时客户端服务配置1s超时
  5. 检查客户端服务pod与polaris server pod之间的网络延迟不高,同时没有网络抖动;
    Expected behavior
    A clear and concise description of what you expected to happen.

Environment

  • Version: polaris-server版本1.18.1
  • OS: 宿主机系统Ubuntu18
    Additional context
    Add any other context about the problem here.
@flashbai flashbai added the bug Something isn't working label Nov 15, 2024
@flashbai
Copy link
Author

目前的话Polaris-Server在单副本情况下运行正常,调整为多副本后,运行一段时间,就会出现客户端服务注册异常情况

@flashbai
Copy link
Author

polaris-server pod在内存limit上限为8G,内存持续上涨的问题在调整为pod为单副本并重启pod后恢复正常,但调整为多副本就会出现内存一直上涨问题
Uploading polaris-server pod内存使用率持续上涨在调整为单副本并重启后恢复正常.png…

@flashbai
Copy link
Author

客户端注册polaris时的1s超时导致注册异常截图
Uploading image.png…

@chuntaojun
Copy link
Member

截屏2024-11-20 09 46 16

看着有数据库死锁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants