Alok protobuf make clientlib module (#75)

* make clientlib module

* gradle fixes for compilation

* gradle fixes for compilation

* gradle changes

* rename org.apache.platypus to com.yelp.platypus

* fix groupId in clientlib build.gradle

* Revert "rename org.apache.platypus to com.yelp.platypus"

This reverts commit a1640e8.

* rename org.apache.platypus to com.yelp.nrtsearch

* fix publication groupID

* Change all traces of "platypus" (except for repo names) to "nrtSearch"

* fix unfound java src in clientlib error

* fixes for building grpc-gateway

* fixes for building grpc-gateway

Author: alok-sm

.gitignore

@@ -44,3 +44,5 @@ replica_state/
 serverPrimary/
 serverSecondary/
 server1RootDirName1/
+primary_index_base/
+replica_index_base/

NOTICE

@@ -21,5 +21,5 @@ at:
       * https://github.com/mikemccand/luceneserver/blob/master/src/java/org/apache/lucene/server/Connection.java#L3 (Apache License 2.0)
     * HOMEPAGE:
       * https://github.com/mikemccand/luceneserver
-    * LOCATION_IN_PLATYPUS:
-      * org.apache.platypus.server.luceneserver
+    * LOCATION_IN_NRTSEARCH:
+      * com.yelp.nrtsearch.server.luceneserver

README.md

@@ -1,4 +1,4 @@
-# Platypus
+# nrtSearch
 A high performance gRPC server on top of [Apache Lucene](http://lucene.apache.org/) version 8.x source, exposing lucene's
 core functionality over a simple gRPC based API.
 
@@ -17,8 +17,8 @@ Fields must first be registered with the *registerFields* command, where you exp
 There is no transaction log, so you must call *commit* yourself periodically to make recent changes durable on disk. This means that if a node crashes, all indexed documents since the last commit are lost.
 
 # Indexing a stream of documents
-platypus supports client side gRPC streaming for its *addDocuments* endpoint. This means that the server API accepts a stream of documents . The client can choose to stream the documents however it wishes.
-The example platypus client implemented here reads a CSV file and streams documents from it over to the server. The server can index chunks of documents the size of which is configurable as the client
+nrtSearch supports client side gRPC streaming for its *addDocuments* endpoint. This means that the server API accepts a stream of documents . The client can choose to stream the documents however it wishes.
+The example nrtSearch client implemented here reads a CSV file and streams documents from it over to the server. The server can index chunks of documents the size of which is configurable as the client
 continues to send more documents over its stream. gRPC enables this with minimal application code and yields higher performance compared to JSON. TODO[citation needed]: Add performance numbers of stream based indexing for some datasets.
 
 # Near-real-time-replication
@@ -48,20 +48,20 @@ Note: This code has been tested on *Java13*
 # Run Server
 
 ```
-./build/install/platypus/bin/lucene-server
+./build/install/nrtsearch/bin/lucene-server
 ```
 
 # Example to run some basic client commands
 ## Create Index
 
 ```
-./build/install/platypus/bin/lucene-client createIndex --indexName  testIdx --rootDir testIdx
+./build/install/nrtsearch/bin/lucene-client createIndex --indexName  testIdx --rootDir testIdx
 ```
 
 ## Update Settings
 
 ```
-./build/install/platypus/bin/lucene-client settings -f settings.json
+./build/install/nrtsearch/bin/lucene-client settings -f settings.json
 cat settings.json
 {             "indexName": "testIdx",
               "indexVerbose": false,
@@ -76,7 +76,7 @@ cat settings.json
 ## Start Index
 
 ```
-./build/install/platypus/bin/lucene-client startIndex -f startIndex.json
+./build/install/nrtsearch/bin/lucene-client startIndex -f startIndex.json
 cat startIndex.json
 {
   "indexName" : "testIdx"
@@ -86,7 +86,7 @@ cat startIndex.json
 ## RegisterFields
 
 ```
-./build/install/platypus/bin/lucene-client registerFields -f registerFields.json
+./build/install/nrtsearch/bin/lucene-client registerFields -f registerFields.json
 cat registerFields.json
 {             "indexName": "testIdx",
               "field":
@@ -100,7 +100,7 @@ cat registerFields.json
 ## Add Documents
 
 ```
-./build/install/platypus/bin/lucene-client addDocuments -i testIdx -f docs.csv
+./build/install/nrtsearch/bin/lucene-client addDocuments -i testIdx -f docs.csv
 cat docs.csv
 doc_id,vendor_name,license_no
 0,first vendor,100;200
@@ -110,7 +110,7 @@ doc_id,vendor_name,license_no
 ## Search
 
 ```
-./build/install/platypus/bin/lucene-client search -f search.json
+./build/install/nrtsearch/bin/lucene-client search -f search.json
 cat search.json
 {
         "indexName": "testIdx",
@@ -138,11 +138,11 @@ This should create a src/main/docs/index.html file that can be seen in your loca
 This tool indexes yelp reviews available at [Yelp dataset challenge](https://www.yelp.com/dataset/challenge). It runs a default version with only 1k reviews of the `reviews.json` or you could download the yelp dataset and place the review.json in the user.home dir and the tool will use that instead. The complete review.json should have close to 7Million reviews. The tool runs multi-threaded indexing and a search thread in parallel reporting the `totalHits`.  Command to run this specific test:
 
 ```
-./gradlew clean && ./gradlew installDist && ./gradlew test -PincludePerfTests=* --tests "org.apache.platypus.server.YelpReviewsTest.runYelpReviews" --info
+./gradlew clean && ./gradlew installDist && ./gradlew test -PincludePerfTests=* --tests "com.yelp.nrtsearch.server.YelpReviewsTest.runYelpReviews" --info
 ```
 
 # Suggestions
 
-This test indexes businesses, creates an Infix Suggester and fetches suggestions. It requires a host, a port and a writeable directory in a standalone Platypus server.
+This test indexes businesses, creates an Infix Suggester and fetches suggestions. It requires a host, a port and a writeable directory in a standalone nrtSearch server.
 
-```./gradlew test -DsuggestTmp=remoteServerDir -DsuggestHost=yourStandaloneServerHost -DsuggestPort=yourStandaloneServerHost --tests "org.apache.platypus.server.YelpSuggestTest"```
+```./gradlew test -DsuggestTmp=remoteServerDir -DsuggestHost=yourStandaloneServerHost -DsuggestPort=yourStandaloneServerHost --tests "com.yelp.nrtsearch.server.YelpSuggestTest"```

build.gradle

@@ -1,8 +1,6 @@
 plugins {
     // Provide convenience executables for trying out the examples.
     id 'application'
-    // ASSUMES GRADLE 2.12 OR HIGHER. Use plugin version 0.7.5 with earlier gradle versions
-    id 'com.google.protobuf' version '0.8.8'
     // Generate IntelliJ IDEA's .idea & .iml project files
     id 'idea'
 }
@@ -22,9 +20,6 @@ targetCompatibility = 1.12
 
 // Feel free to delete the comment at the next line. It is just for safely
 // updating the version in our release process.
-def grpcVersion = '1.26.0' // CURRENT_GRPC_VERSION
-def protobufVersion = '3.9.0'
-def protocVersion = protobufVersion
 def luceneVersion = '8.2.0'
 def slf4jVersion = '2.0.0-alpha0'
 def gsonVersion = '2.8.5'
@@ -39,17 +34,12 @@ def prometheusVersion = '0.3.0'
 def prometheusClientVersion = '0.8.0'
 
 dependencies {
-    //grpc deps
-    implementation "io.grpc:grpc-protobuf:${grpcVersion}"
-    implementation "io.grpc:grpc-stub:${grpcVersion}"
 
     //prometheus (metrics) deps
     implementation "me.dinowernli:java-grpc-prometheus:${prometheusVersion}"
-    implementation "io.grpc:grpc-okhttp:${grpcVersion}"
     implementation "io.prometheus:simpleclient_servlet:${prometheusClientVersion}"
 
     //
-    implementation "javax.annotation:javax.annotation-api:1.2"
     implementation "org.slf4j:slf4j-api:${slf4jVersion}"
     implementation "org.slf4j:slf4j-jdk14:2.0.0-alpha1"
     implementation "com.google.code.gson:gson:${gsonVersion}"
@@ -75,57 +65,42 @@ dependencies {
     implementation "org.apache.lucene:lucene-grouping:${luceneVersion}"
     implementation "org.apache.lucene:lucene-queries:${luceneVersion}"
 
-    // examples/advanced need this for JsonFormat
-    implementation "com.google.protobuf:protobuf-java-util:${protobufVersion}"
-
     //cli deps
     implementation 'info.picocli:picocli:4.0.4'
     implementation 'org.apache.commons:commons-csv:1.7'
 
-    runtimeOnly "io.grpc:grpc-netty-shaded:${grpcVersion}"
-
-    //test deps
-    testImplementation "io.grpc:grpc-testing:${grpcVersion}"
     testImplementation "junit:junit:4.12"
     testImplementation "org.mockito:mockito-core:2.25.1"
     testImplementation "org.apache.lucene:lucene-test-framework:${luceneVersion}"
     testImplementation "org.locationtech.spatial4j:spatial4j:${spatial4jVersion}"
     testImplementation "io.findify:s3mock_2.12:${s3mockVersion}"
 
-
+    compile project(':clientlib')
 }
 
-protobuf {
-    protoc { artifact = "com.google.protobuf:protoc:${protocVersion}" }
-    plugins {
-        grpc { artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}" }
-    }
-    generateProtoTasks {
-        all()*.plugins { grpc {} }
-    }
-}
+
 
 // Inform IDEs like IntelliJ IDEA, Eclipse or NetBeans about the generated code.
 sourceSets {
     main {
         java {
-            srcDirs 'build/generated/source/proto/main/grpc'
-            srcDirs 'build/generated/source/proto/main/java'
+            srcDirs 'clientlib/build/generated/source/proto/main/grpc'
+            srcDirs 'clientlib/build/generated/source/proto/main/java'
         }
     }
 }
 
 startScripts.enabled = false
 
 task luceneServer(type: CreateStartScripts) {
-    mainClassName = 'org.apache.platypus.server.grpc.LuceneServer'
+    mainClassName = 'com.yelp.nrtsearch.server.grpc.LuceneServer'
     applicationName = 'lucene-server'
     outputDir = new File(project.buildDir, 'tmp')
     classpath = startScripts.classpath
 }
 
 task luceneServerClient(type: CreateStartScripts) {
-    mainClassName = 'org.apache.platypus.server.grpc.LuceneServerClient'
+    mainClassName = 'com.yelp.nrtsearch.server.grpc.LuceneServerClient'
     applicationName = 'lucene-client'
     outputDir = new File(project.buildDir, 'tmp')
     classpath = startScripts.classpath

build_grpc_gateway.sh

@@ -1,4 +1,4 @@
 docker build -t grpc-gateway -f grpc-gateway/Dockerfile .
 docker run grpc-gateway
-docker cp $(docker ps -alq):/code/output/. ./grpc-gateway/
-docker cp $(docker ps -alq):/code/bin/. ./build/install/platypus/bin
+docker cp $(docker ps -alq):/code/output/yelp/nrtsearch/. ./grpc-gateway/
+docker cp $(docker ps -alq):/code/bin/. ./build/install/nrtsearch/bin

clientlib/build.gradle

@@ -0,0 +1,76 @@
+plugins {
+    // Provide convenience executables for trying out the examples.
+    id 'application'
+    // ASSUMES GRADLE 2.12 OR HIGHER. Use plugin version 0.7.5 with earlier gradle versions
+    id 'com.google.protobuf' version '0.8.8'
+    // Generate IntelliJ IDEA's .idea & .iml project files
+    id 'idea'
+    // Publish clientlib to maven central
+    id 'maven-publish'
+}
+
+repositories {
+    maven { // The google mirror is less flaky than mavenCentral()
+        url "https://maven-central.storage-download.googleapis.com/repos/central/data/"
+    }
+    mavenLocal()
+}
+
+sourceCompatibility = 1.12
+targetCompatibility = 1.12
+startScripts.enabled = false
+
+def grpcVersion = '1.26.0' // CURRENT_GRPC_VERSION
+def protobufVersion = '3.11.1'
+def protocVersion = protobufVersion
+
+dependencies {
+    //grpc deps
+    compile "io.grpc:grpc-protobuf:${grpcVersion}"
+    implementation "io.grpc:grpc-stub:${grpcVersion}"
+    implementation "io.grpc:grpc-okhttp:${grpcVersion}"
+    implementation "javax.annotation:javax.annotation-api:1.2"
+
+    // examples/advanced need this for JsonFormat
+    compile "com.google.protobuf:protobuf-java-util:${protobufVersion}"
+
+    runtimeOnly "io.grpc:grpc-netty-shaded:${grpcVersion}"
+
+    //test deps
+    compile "io.grpc:grpc-testing:${grpcVersion}"
+}
+
+sourceSets {
+    all {
+        proto {
+            srcDirs 'proto'
+        }
+    }
+}
+
+protobuf {
+    protoc { artifact = "com.google.protobuf:protoc:${protocVersion}" }
+    plugins {
+        grpc { artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}" }
+    }
+    generateProtoTasks {
+        all()*.plugins { grpc {} }
+    }
+}
+
+publishing {
+    publications {
+        maven(MavenPublication) {
+            groupId = 'com.yelp.nrtsearch'
+            artifactId = 'clientlib'
+            version = '0.1'
+        }
+    }
+}
+
+test {
+    // Add tests when there's java code in this module
+    // Until then, exclude everything to allow
+    // build to run
+    exclude '**/*'
+}

src/main/proto/analysis.proto clientlib/proto/yelp/nrtsearch/analysis.proto

@@ -2,7 +2,7 @@
 syntax = "proto3";
 
 option java_multiple_files = true;
-option java_package = "org.apache.platypus.server.grpc";
+option java_package = "com.yelp.nrtsearch.server.grpc";
 option java_outer_classname = "AnalysisProto";
 option objc_class_prefix = "HLW";
 

src/main/proto/luceneserver.proto clientlib/proto/yelp/nrtsearch/luceneserver.proto

@@ -1,13 +1,13 @@
-/* Description of Platypus Service APIs and message types */
+/* Description of NRTSearch Service APIs and message types */
 syntax = "proto3";
 
-import "search.proto";
-import "analysis.proto";
+import "yelp/nrtsearch/search.proto";
+import "yelp/nrtsearch/analysis.proto";
 import "google/api/annotations.proto";
 
 
 option java_multiple_files = true;
-option java_package = "org.apache.platypus.server.grpc";
+option java_package = "com.yelp.nrtsearch.server.grpc";
 option java_outer_classname = "LuceneServerProto";
 option objc_class_prefix = "HLW";
 

src/main/proto/search.proto clientlib/proto/yelp/nrtsearch/search.proto

@@ -2,10 +2,10 @@
 syntax = "proto3";
 
 import "google/type/latlng.proto";
-import "analysis.proto";
+import "yelp/nrtsearch/analysis.proto";
 
 option java_multiple_files = true;
-option java_package = "org.apache.platypus.server.grpc";
+option java_package = "com.yelp.nrtsearch.server.grpc";
 option java_outer_classname = "SearchResponseProto";
 option objc_class_prefix = "HLW";
 

grpc-gateway/Dockerfile

@@ -37,10 +37,50 @@ RUN go get -u \
     github.com/golang/protobuf/protoc-gen-go \
     google.golang.org/grpc
 
+ENV PROTO_PATH=/code/clientlib/proto
+ENV PROTO_BUILD_PATH=/code/clientlib/build
+
 #run protoc plugin for protoc-gen-go
-RUN protoc -I/code/src/main/proto -I/code/build/extracted-protos/main -I/code/build/extracted-include-protos/main --plugin=protoc-gen-go=$GOPATH/bin/protoc-gen-go --go_out=plugins=grpc:$OUTPATH /code/src/main/proto/analysis.proto /code/src/main/proto/luceneserver.proto /code/src/main/proto/search.proto
-RUN protoc -I/code/src/main/proto -I/code/build/extracted-protos/main -I/code/build/extracted-include-protos/main --plugin=protoc-gen-grpc-gateway=$GOPATH/bin/protoc-gen-grpc-gateway --grpc-gateway_out=logtostderr=true:$OUTPATH /code/src/main/proto/analysis.proto /code/src/main/proto/luceneserver.proto /code/src/main/proto/search.proto
-RUN protoc -I/code/src/main/proto -I/code/build/extracted-protos/main -I/code/build/extracted-include-protos/main --plugin=protoc-gen-swagger=$GOPATH/bin/protoc-gen-swagger --swagger_out=logtostderr=true:$OUTPATH /code/src/main/proto/analysis.proto /code/src/main/proto/luceneserver.proto /code/src/main/proto/search.proto
+RUN protoc \
+-I $PROTO_PATH \
+-I  $PROTO_BUILD_PATH/extracted-protos/main \
+-I $PROTO_BUILD_PATH/extracted-include-protos/main \
+--plugin=protoc-gen-go=$GOPATH/bin/protoc-gen-go \
+--go_out=plugins=grpc:$OUTPATH \
+$PROTO_PATH/yelp/nrtsearch/analysis.proto \
+$PROTO_PATH/yelp/nrtsearch/luceneserver.proto \
+$PROTO_PATH/yelp/nrtsearch/search.proto
+
+RUN protoc \
+-I $PROTO_PATH \
+-I $PROTO_BUILD_PATH/extracted-protos/main \
+-I $PROTO_BUILD_PATH/extracted-include-protos/main \
+--plugin=protoc-gen-grpc-gateway=$GOPATH/bin/protoc-gen-grpc-gateway \
+--grpc-gateway_out=logtostderr=true:$OUTPATH \
+$PROTO_PATH/yelp/nrtsearch/analysis.proto \
+$PROTO_PATH/yelp/nrtsearch/luceneserver.proto \
+$PROTO_PATH/yelp/nrtsearch/search.proto
+
+RUN protoc \
+-I $PROTO_PATH \
+-I $PROTO_BUILD_PATH/extracted-protos/main \
+-I $PROTO_BUILD_PATH/extracted-include-protos/main \
+--plugin=protoc-gen-swagger=$GOPATH/bin/protoc-gen-swagger \
+--swagger_out=logtostderr=true:$OUTPATH \
+$PROTO_PATH/yelp/nrtsearch/analysis.proto \
+$PROTO_PATH/yelp/nrtsearch/luceneserver.proto \
+$PROTO_PATH/yelp/nrtsearch/search.proto
+
+RUN cp $OUTPATH/yelp/nrtsearch/* grpc-gateway/
+
 # build go executables for various platforms
-RUN for GOOS in darwin linux windows; do for GOARCH in 386 amd64; do echo "Building $GOOS-$GOARCH"; export GOOS=$GOOS; export GOARCH=$GOARCH; go build -o bin/http_wrapper-$GOOS-$GOARCH http_wrapper.go; done; done
+RUN for GOOS in darwin linux windows; do \
+    for GOARCH in 386 amd64; do \
+        echo "Building $GOOS-$GOARCH"; \
+        export GOOS=$GOOS; \
+        export GOARCH=$GOARCH; \
+        go build -o bin/http_wrapper-$GOOS-$GOARCH http_wrapper.go; \
+    done; \
+done
+
 CMD /bin/sh