Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

partial ripgrep support #1448

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

HoxhaEndri
Copy link
Member

  • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
    It uses ripgrep to search for regex.

  • What is the current behavior? (You can also link to an open issue here)

  • What is the new behavior (if this is a feature change)? If possible add a screenshot.
    It is considerably faster.

  • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

  • Other information:

@m-1-k-3
Copy link
Member

m-1-k-3 commented Jan 27, 2025

Is ripgrep already available in our docker image?

@HoxhaEndri
Copy link
Member Author

Is ripgrep already available in our docker image?

It is available in the standard debian/kali repo.

@m-1-k-3
Copy link
Member

m-1-k-3 commented Jan 27, 2025

ok, I will build a new docker-base image for further testing. The current results are very cool!

@m-1-k-3
Copy link
Member

m-1-k-3 commented Jan 28, 2025

Please update the docker-compose file to use the base image 1.5.1c:

image: embeddedanalyzer/emba:1.5.1b

image: embeddedanalyzer/emba:1.5.1b

Then rq should be available and you can do some performance tests. Probably we can also check other greps for using rg

@m-1-k-3 m-1-k-3 added enhancement New feature or request docker docker related things cve-search Some cve-search question/issue EMBA labels Jan 28, 2025
@m-1-k-3
Copy link
Member

m-1-k-3 commented Jan 29, 2025

I did a quick benchmark yesterday with the original EMBA master branch vs the ripgrep PR:

Default grep:
[*] Tue Jan 28 21:43:06 CET 2025 - F20_vul_aggregator starting
[*] Tue Jan 28 22:01:50 CET 2025 - F20_vul_aggregator finished
[!] Tue Jan 28 22:02:31 CET 2025 - Test ended on Tue Jan 28 22:02:31 CET 2025 and took about 0 days and 00:24:55 

Ripgrep
[*] Tue Jan 28 19:11:30 CET 2025 - F20_vul_aggregator starting
[*] Tue Jan 28 19:30:08 CET 2025 - F20_vul_aggregator finished
[!] Tue Jan 28 19:30:50 CET 2025 - Test ended on Tue Jan 28 19:30:50 CET 2025 and took about 0 days and 00:24:52 

Looks as we do not get any speedup of ripgrep. Can you also reproduce these results?

@@ -58,7 +58,7 @@ IF20_nvd_feed() {
git clone --depth 1 -b main https://github.com/EMBA-support-repos/EPSS-data.git external/EPSS-data || ( echo "Could not install EMBA component NIST EPSS data feed" && exit 1 )
fi

if [[ $(grep -l -E "cpe.*busybox:" external/nvd-json-data-feeds/* -r 2>/dev/null | wc -l) -gt 18 ]]; then
if [[ $(rg -l "cpe.*busybox:" external/nvd-json-data-feeds/* 2>/dev/null | wc -l) -gt 18 ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not recommend using rg in the installer as we need to install it on the host first

@@ -38,7 +38,7 @@ for STRING_ENTRY in "${STRING_ENTRY_ARR[@]}"; do

echo "[*] Testing SBOM entry ${COMPONENT//::}"

if [[ "$(grep -h "cpe.*${COMPONENT//::}:" external/nvd-json-data-feeds/* -r | wc -l 2>/dev/null)" -gt 0 ]]; then
if [[ "$(rg -I -N "cpe.*${COMPONENT//::}:" external/nvd-json-data-feeds/* | wc -l 2>/dev/null)" -gt 0 ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a helper script for the host and I would not recommend using rg in such a script as we need to install it on the host first

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cve-search Some cve-search question/issue docker docker related things EMBA enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants