[BUG] The incremental update interface seems to have performance problems.

[18620385582]

OpenIM Server Version

3.8.2

Operating System and CPU Architecture

Linux (AMD)

Deployment Method

Source Code Deployment

Bug Description and Steps to Reproduce

服务器的批量存储接口 BatchGetIncrementalGroupMember的实现中， 最后的resp一次只返回200条， 可以确在第一次把所有的群组成员信息都查询出来， 并且openim支持10万人的大群， 这样会有严重的性能浪费，并且增量查询的时候logs还无索引， 这些一起会导致mongodb占用奇高的cpu

	Resp: func(versions map[string]*model.VersionLog, deleteIdsMap map[string][]string, insertListMap, updateListMap map[string][]*sdkws.GroupMemberFullInfo, fullMap map[string]bool) *pbgroup.BatchGetIncrementalGroupMemberResp {
		resList := make(map[string]*pbgroup.GetIncrementalGroupMemberResp)

		for groupID, versionLog := range versions {
			resList[groupID] = &pbgroup.GetIncrementalGroupMemberResp{
				VersionID:   versionLog.ID.Hex(),
				Version:     uint64(versionLog.Version),
				Full:        fullMap[groupID],
				Delete:      deleteIdsMap[groupID],
				Insert:      insertListMap[groupID],
				Update:      updateListMap[groupID],
				SortVersion: sortVersionMap[groupID],
			}

			requestBodyLen += len(insertListMap[groupID]) + len(updateListMap[groupID]) + len(deleteIdsMap[groupID])

			// TODO: 实现可能有问题, 为啥只要200个， 第一次的时候versions还都查一遍， 并且logs无索引
			if requestBodyLen > 200 {
				break
			}
		}

		return &pbgroup.BatchGetIncrementalGroupMemberResp{
			RespList: resList,
		}
	},

Screenshots Link

No response

[OpenIM-Robot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

OpenIM Server Version

3.8.2

Operating System and CPU Architecture

Linux (AMD)

Deployment Method

Source Code Deployment

Bug Description and Steps to Reproduce

In the implementation of the server's batch storage interface BatchGetIncrementalGroupMember, the last resp only returns 200 items at a time. It can be found that all group member information is queried for the first time, and openim supports a large group of 100,000 people, which will be serious. Performance is wasted, and logs have no index during incremental query, which will cause mongodb to occupy a very high CPU.

Resp: func(versions map[string]*model.VersionLog, deleteIdsMap map[string][]string, insertListMap, updateListMap map[string][]*sdkws.GroupMemberFullInfo, fullMap map[string]bool) *pbgroup.BatchGetIncrementalGroupMemberResp {
resList := make(map[string]*pbgroup.GetIncrementalGroupMemberResp)

for groupID, versionLog := range versions {
resList[groupID] = &pbgroup.GetIncrementalGroupMemberResp{
VersionID: versionLog.ID.Hex(),
Version: uint64(versionLog.Version),
Full: fullMap[groupID],
Delete: deleteIdsMap[groupID],
Insert: insertListMap[groupID],
Update: updateListMap[groupID],
SortVersion: sortVersionMap[groupID],
}

requestBodyLen += len(insertListMap[groupID]) + len(updateListMap[groupID]) + len(deleteIdsMap[groupID])

// TODO: There may be problems with the implementation, why only 200 versions are checked for the first time, and logs have no index
if requestBodyLen > 200 {
break
}
}

return &pbgroup.BatchGetIncrementalGroupMemberResp{
RespList: resList,
}
},

Screenshots Link

No response

[OpenIM-Robot]

Hello! Thank you for filing an issue.

If this is a bug report, please include relevant logs to help us debug the problem.

Join slack 🤖 to connect and communicate with our developers.