Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: integrate meilisearch for searching #56

Open
wants to merge 58 commits into
base: main
Choose a base branch
from

Conversation

Codycody31
Copy link
Contributor

@Codycody31 Codycody31 commented Apr 9, 2024

Adds support for the use of meilisearch for quick indexing and searching of discuit communities.

Required

  • Move code for updating a communities index used before the API returns info to separate function
  • Rather than placing search code in the core move to internal/search so we can have a package for it and not have to worry about names being the same
  • Integration into the UI for searching (only if it is enabled)
  • When indexing, break up the community name on each capital letter (to fix searching for Dev not matching DiscuitDev, etc)
  • Allow searching on the communities page but default it when blank to the standard API

If Possible

  • Index posts and users
  • Sorting based on members, post creation time, user's account age, etc

General

After this is merged, someone who is better at UI stuff than me should probably make the searching be nice lol

Known Issues

  • When searching for a community while creating a new post, the picture is not used due to it not being included in the data set to meilisearch

Related Issues:

@Codycody31 Codycody31 changed the title Featureintegrate meilisearch for search endpoint Feature: integrate meilisearch for searching Apr 9, 2024
@Codycody31 Codycody31 marked this pull request as ready for review April 10, 2024 16:58
@Codycody31 Codycody31 marked this pull request as ready for review April 26, 2024 13:54
@Codycody31
Copy link
Contributor Author

Codycody31 commented Apr 26, 2024

@previnder, the indexing logic along with everything else is finished. Now when pulling data to index it takes it in batches of 5000 records (I'll change it if you want it to be higher or lower, probably higher around 10,000 or 15,000). When pushing the data over to meilisearch we chunk it into batches of 50MiB (or less if we don't have that much data). By default, the searching ability is disabled and still makes use of the Google search to allow for the hoster to fully configure everything without having to worry about meilisearch till they are ready.

@Codycody31
Copy link
Contributor Author

Merged latest, and now configured to also work with environment variables

@Codycody31
Copy link
Contributor Author

Along with the conflicting files, I've found some other bugs when implementing this PR in a fork, I'll take a look at fixing the issues either today, or this weekend.

@Codycody31
Copy link
Contributor Author

Note, going to refactor a bit of code to be a bit more search engine agnositic, and just refer to the package as search, rather than meilisearch as that can be decided when we store it in context.

…rch interactions, standardizing interactions and setting a framework to allow for future alternative search engines
…lisearch-for-search-endpoint

cut out the searching in the UI, not sure how we want this to be done
@Codycody31
Copy link
Contributor Author

@previnder I cut out the searching in the UI for communities, not sure how we want this to be done. If you want I can also remove the searching from the main search bar in the UI as it's not complete yet and only supports searching communities in the UI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add in-site search When searched moving to google Request for Additional Search Engines
1 participant