Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improve pinyin fuzzy segement algorithm
Previously, we blindly choose the segment to always prefer the longer next match, this is prove wrong in the case of "sangeren". Which should produce, "san ge ren", "sang er en", "sang e ren". Instead, we change the check to be: if (current + next match) is valid, and complete pinyin, make it an acceptable option, unless (current, next match) is actually an inner fuzzy, which is handled separately below. For example: 1. For sangeren, will produce sang & san, since next match of "san", which is "ge", is a complete pinyin. 2. For hua, will only produce hua, since hu a is a inner fuzzy. Even if it will produce "extra" segement, for example, in the case of "sanger" will produce a partial pinyin "san" "ge" "r". We may still consider it as make sense. Since partial pinyin match is considered fuzzy and will have a penalty score. People may even benefit from such segement, since "san ge r" seems to be the most possible option. Fix #87
- Loading branch information