About Books App

The Books API app is designed to aggregate and manage book data from a multitude of providers, seamlessly importing this data into a MySQL database. Our Books API features a robust search endpoint that allows users to perform quick searches across millions of stored books.

To ensure optimal search performance, we have integrated Elasticsearch, which significantly speeds up full-text search capabilities. Additionally, we have implemented a Redis caching layer to further enhance the overall performance and efficiency of the API. With these technologies, the Books API delivers a fast, reliable, and scalable solution for accessing and managing extensive book data.

• Book Crawler in step 1 has not been implemented yet in this source. For now, we are temporarily using PHP Faker to create some dummy data.

Performance comparison between ES & Mysql

• The performance of Elasticsearch and MySQL is nearly identical when the total number of book records is below 100,000.
• However, when the number of book records is increased to approximately 800,000:
  • Increasing ES nodes from 1 to 2 resulted in ES being three times faster than MySQL.
  • Currently, the search API applies sorting and pagination, which impacts search performance.
  • Considering scenarios where the user is typing in the search input and we implement an API for auto-suggesting books relevant to the typed keywords without sorting and pagination, I believe the search performance in ES can be significantly faster.

MySQL CURL request

curl --request GET \
  --url 'http://localhost:8080/api/books?query=and&per_page=100&order_by=title&order_type=desc&page=401'

ES CURL request

curl --request GET \
  --url 'http://localhost:8080/api/books/search?query=and&per_page=100&order_by=title&order_type=desc&page=401'

Prerequisites

• Docker >=25
• PHP 8.2
• MySQL 8
• Elasticsearch 8.13.4
• Kibana 8.13.4
• Redis 7.2.5
• Nginx 1.25.5

How to Run on a Local Machine

1. Clone the repository:

   git clone [email protected]:moitran/books-app.git
   cd books-app
   cp .env.example .env

2. Start the Docker containers:

   docker compose up -d --build

3. Composer install

   docker compose exec -it app composer install
   # Generate Laravel application keys
   docker compose exec -it app php artisan key:generate
   # clear cache & reload config
   docker compose exec -it app php artisan optimize

4. Run database migrations

   docker compose exec -it app php artisan migrate

   

5. Run database seeding

   docker compose exec -it app php artisan db:seed

   

6. Perform a full sync of book data into Elasticsearch:

   # Publish book data into ES
   docker compose exec -it app php artisan scout:import "App\Models\Book"
   
   # Setting ES index.max_result_window upto a million
   docker compose exec -it app curl -X PUT "localhost:9200/my_index/_settings" -H 'Content-Type: application/json' -d'
   {
     "index" : {
       "max_result_window" : 100000
     }
   }
   '

   

7. Access the homepage at http://localhost:8080.

8. Perform API by CURL or you can perform on Swagger

   • Search by ES:

       curl --request GET \
       --url 'http://localhost:8080/api/books/search?query=and&per_page=100&order_by=title&order_type=desc&page=100' \
       --header 'User-Agent: insomnia/9.2.0'
   

   • Search by Mysql:

       curl --request GET \
       --url 'http://localhost:8080/api/books?query=and&per_page=100&order_by=title&order_type=desc&page=1' \
       --header 'User-Agent: insomnia/9.2.0'
   

9. Laravel Swagger is integrated for API specifications. You can check it out at http://localhost:8080/api/documentation.

10. Application monitoring is integrated with Laravel Telescope. You can access it at http://localhost:8080/telescope to monitor requests, queues, Redis, etc.

11. To visualize Elasticsearch data, Kibana is integrated. You can access it at http://localhost:5601.