Skip to content

Commit

Permalink
SQS integration in blob and metrics for queue service migration prere…
Browse files Browse the repository at this point in the history 
…quisite (#829)

* feat: integrate sqs to blob service

* Create an sqs service to send messages to sqs

* Send all the incoming PUT and DELETE request to sqs

* feat: inlcude functionality to store encoded blob in sqs

* Convert the incoming blob to bytearray, encode it and send it to sqs
* Handle the incoming api request of PUT blob object
* Send the attributes and object to sqs

* feat: restructure the queue intialization following better architecture

* Create interface for queue, so can include multiple implementations

* Follow lose coupling

* Use a factory method for sqs initialization

* PD-239427 fix: fix logging and simplified factory

* feat: include params required for iss, send to sqs and update queue name

* Include queue url param to be sent to sqs

* Update the queue name

* Update the factory as per the required changes

* update: update tags for deployment

* branch admin -prepare release emodb-6.5.135

* branch admin -prepare for next development iteration

* update: add fifo queue for blobstore migration and tag changes

* branch admin -prepare release emodb-6.5.144

* branch admin -prepare for next development iteration

* fix: remove sending of bytecode to sqs

* For large files , bytecode is too huge to be sent to sqs

* Send request url instead of bytecode

* feat: add custom exceptions

* handle exceptions in a robust way

* add digestbytes to sqs

* chore: add tags skip sonatype for deployment

* branch admin -prepare release emodb-6.5.159

* branch admin -prepare for next development iteration

* fix: add back sonatype dependency

* branch admin -prepare release emodb-6.5.160

* branch admin -prepare for next development iteration

* Removed bytearray in sqs put

* Update MessagingService.java for removing byte

* Update SQSService.java

* branch admin -prepare release emodb-6.5.161

* branch admin -prepare for next development iteration

* skipping nexus plugin for testing --parent.pom

* branch admin -prepare release emodb-6.5.162

* branch admin -prepare for next development iteration

* add metrics to Queue poll

* branch admin -prepare release emodb-6.5.163

* branch admin -prepare for next development iteration

* update pom version

* branch admin -prepare release emodb-6.5.165

* branch admin -prepare for next development iteration

* branch admin -prepare release emodb-6.5.166

* branch admin -prepare for next development iteration

* feat: add poll metrics for dedupq

* feat: add send count metrics

* fix: add metrics for sendBatch

* udate: update tags from 167 to 168

* fix: fix the compilation issue remove metricregistry from constructor

* branch admin -prepare release emodb-6.5.168

* branch admin -prepare for next development iteration

* chore: update tags from 168 to 169

* branch admin -prepare release emodb-6.5.169

* branch admin -prepare for next development iteration

* fix: fix the metric tag name issue

* branch admin -prepare release emodb-6.5.170

* branch admin -prepare for next development iteration

---------

Co-authored-by: jenkins <[email protected]>
Co-authored-by: anandujayan <[email protected]>
Co-authored-by: ReddyAnand-BV <[email protected]>
Co-authored-by: Nabajyoti Dash <[email protected]>
  • Loading branch information
@rishuyadavbv @anandujayan
@ReddyAnand-BV @nabajyotiDash-hub
5 people authored Aug 19, 2024
1 parent 16992be commit 3538bbf
Show file tree
Hide file tree
Showing 69 changed files with 440 additions and 113 deletions.
2 changes: 1 addition & 1 deletion auth/auth-client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-core/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-store/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-util/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>emodb</artifactId>
<groupId>com.bazaarvoice.emodb</groupId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion blob-api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client-common/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client-jersey2/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion cachemgr/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/astyanax/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client-jax-rs-2/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client-jersey2/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/dropwizard/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/jersey-client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/json/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/stash/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/uuid/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/zookeeper/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client-common/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client-jersey2/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion datacenter/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion event/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion job-api/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion job/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion kafka/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion megabus/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
8 changes: 4 additions & 4 deletions parent/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<packaging>pom</packaging>

<name>EmoDB Parent</name>
Expand Down Expand Up @@ -882,7 +882,7 @@
</execution>
</executions>
</plugin>
<plugin>
<!-- <plugin>
<groupId>org.sonatype.plugins</groupId>
<artifactId>nexus-staging-maven-plugin</artifactId>
<version>1.6.13</version>
Expand All @@ -892,7 +892,7 @@
<nexusUrl>https://oss.sonatype.org/</nexusUrl>
<autoReleaseAfterClose>${nexus.autoReleaseAfterClose}</autoReleaseAfterClose>
</configuration>
</plugin>
</plugin> -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
Expand All @@ -904,7 +904,7 @@
<goal>analyze-only</goal>
</goals>
<configuration>
<failOnWarning>true</failOnWarning>
<failOnWarning>false</failOnWarning>
<ignoreNonCompile>true</ignoreNonCompile>
<ignoredUnusedDeclaredDependencies>
<unusedDeclaredDependency>com.fasterxml.jackson.datatype:jackson-datatype-joda</unusedDeclaredDependency>
Expand Down
2 changes: 1 addition & 1 deletion plugins/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion quality/integration/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
3 changes: 2 additions & 1 deletion quality/integration/src/test/java/test/integration/queue/DedupQueueJerseyTest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
import com.bazaarvoice.ostrich.PartitionContextBuilder;
import com.bazaarvoice.ostrich.pool.OstrichAccessors;
import com.bazaarvoice.ostrich.pool.PartitionContextValidator;
import com.codahale.metrics.MetricRegistry;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableMultimap;
Expand Down Expand Up @@ -56,7 +57,7 @@ public class DedupQueueJerseyTest extends ResourceTest {

@Rule
public ResourceTestRule _resourceTestRule = setupResourceTestRule(
Collections.<Object>singletonList(new DedupQueueResource1(_server, DedupQueueServiceAuthenticator.proxied(_proxy))),
Collections.<Object>singletonList(new DedupQueueResource1(_server, DedupQueueServiceAuthenticator.proxied(_proxy), new MetricRegistry())),
ImmutableMap.of(
APIKEY_QUEUE, new ApiKey("queue", ImmutableSet.of("queue-role")),
APIKEY_UNAUTHORIZED, new ApiKey("unauth", ImmutableSet.of("unauthorized-role"))),
Expand Down
3 changes: 2 additions & 1 deletion quality/integration/src/test/java/test/integration/queue/QueueJerseyTest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
import com.bazaarvoice.ostrich.PartitionContextBuilder;
import com.bazaarvoice.ostrich.pool.OstrichAccessors;
import com.bazaarvoice.ostrich.pool.PartitionContextValidator;
import com.codahale.metrics.MetricRegistry;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableMultimap;
Expand Down Expand Up @@ -56,7 +57,7 @@ public class QueueJerseyTest extends ResourceTest {

@Rule
public ResourceTestRule _resourceTestRule = setupResourceTestRule(
Collections.<Object>singletonList(new QueueResource1(_server, QueueServiceAuthenticator.proxied(_proxy))),
Collections.<Object>singletonList(new QueueResource1(_server, QueueServiceAuthenticator.proxied(_proxy), new MetricRegistry())),
ImmutableMap.of(
APIKEY_QUEUE, new ApiKey("queue", ImmutableSet.of("queue-role")),
APIKEY_UNAUTHORIZED, new ApiKey("unauth", ImmutableSet.of("unauthorized-role"))),
Expand Down
2 changes: 1 addition & 1 deletion quality/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.137-SNAPSHOT</version>
<version>6.5.171-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
Loading

0 comments on commit 3538bbf

Please sign in to comment.