#580 - Fix in product sync while fixing unresolved references. (#581)

* We need all cache of key to ids for finding missing references during product sync, test case was missing because it was not working with multiple batches.

* add release notes entry.
* bump version docs to 2.2.1

README.md

@@ -2,9 +2,9 @@
 # commercetools sync
 [![Build Status](https://travis-ci.org/commercetools/commercetools-sync-java.svg?branch=master)](https://travis-ci.org/commercetools/commercetools-sync-java)
 [![codecov](https://codecov.io/gh/commercetools/commercetools-sync-java/branch/master/graph/badge.svg)](https://codecov.io/gh/commercetools/commercetools-sync-java)
-[![Benchmarks 2.2.0](https://img.shields.io/badge/Benchmarks-2.2.0-orange.svg)](https://commercetools.github.io/commercetools-sync-java/benchmarks/)
+[![Benchmarks 2.2.1](https://img.shields.io/badge/Benchmarks-2.2.1-orange.svg)](https://commercetools.github.io/commercetools-sync-java/benchmarks/)
 [![Download](https://api.bintray.com/packages/commercetools/maven/commercetools-sync-java/images/download.svg) ](https://bintray.com/commercetools/maven/commercetools-sync-java/_latestVersion)
-[![Javadoc](http://javadoc-badge.appspot.com/com.commercetools/commercetools-sync-java.svg?label=Javadoc)](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/)
+[![Javadoc](http://javadoc-badge.appspot.com/com.commercetools/commercetools-sync-java.svg?label=Javadoc)](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/)
 [![Known Vulnerabilities](https://snyk.io/test/github/commercetools/commercetools-sync-java/4b2e26113d591bda158217c5dc1cf80a88665646/badge.svg)](https://snyk.io/test/github/commercetools/commercetools-sync-java/4b2e26113d591bda158217c5dc1cf80a88665646)
 
 More at https://commercetools.github.io/commercetools-sync-java
@@ -37,7 +37,7 @@ The library supports synchronising the following entities in commercetools
     - [Ivy](#ivy)
 - [Roadmap](#roadmap)
 - [Release Notes](/docs/RELEASE_NOTES.md)
-- [Javadoc](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/)
+- [Javadoc](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/)
 - [Benchmarks](https://commercetools.github.io/commercetools-sync-java/benchmarks/)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -79,26 +79,26 @@ Here are the most popular ones:
 <dependency>
   <groupId>com.commercetools</groupId>
   <artifactId>commercetools-sync-java</artifactId>
-  <version>2.2.0</version>
+  <version>2.2.1</version>
 </dependency>
 ````
 
 #### Gradle
 
 ````groovy
-implementation 'com.commercetools:commercetools-sync-java:2.2.0'
+implementation 'com.commercetools:commercetools-sync-java:2.2.1'
 ````
 
 #### SBT 
 
 ````
-libraryDependencies += "com.commercetools" % "commercetools-sync-java" % "2.2.0"
+libraryDependencies += "com.commercetools" % "commercetools-sync-java" % "2.2.1"
 ````
 
 #### Ivy 
 
 ````xml
-<dependency org="com.commercetools" name="commercetools-sync-java" rev="2.2.0"/>
+<dependency org="com.commercetools" name="commercetools-sync-java" rev="2.2.1"/>
 ````
 
 

docs/README.md

@@ -2,9 +2,9 @@
 # commercetools sync
 [![Build Status](https://travis-ci.org/commercetools/commercetools-sync-java.svg?branch=master)](https://travis-ci.org/commercetools/commercetools-sync-java)
 [![codecov](https://codecov.io/gh/commercetools/commercetools-sync-java/branch/master/graph/badge.svg)](https://codecov.io/gh/commercetools/commercetools-sync-java)
-[![Benchmarks 2.2.0](https://img.shields.io/badge/Benchmarks-2.2.0-orange.svg)](https://commercetools.github.io/commercetools-sync-java/benchmarks/)
+[![Benchmarks 2.2.1](https://img.shields.io/badge/Benchmarks-2.2.1-orange.svg)](https://commercetools.github.io/commercetools-sync-java/benchmarks/)
 [![Download](https://api.bintray.com/packages/commercetools/maven/commercetools-sync-java/images/download.svg) ](https://bintray.com/commercetools/maven/commercetools-sync-java/_latestVersion)
-[![Javadoc](http://javadoc-badge.appspot.com/com.commercetools/commercetools-sync-java.svg?label=Javadoc)](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/)
+[![Javadoc](http://javadoc-badge.appspot.com/com.commercetools/commercetools-sync-java.svg?label=Javadoc)](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/)
 [![Known Vulnerabilities](https://snyk.io/test/github/commercetools/commercetools-sync-java/4b2e26113d591bda158217c5dc1cf80a88665646/badge.svg)](https://snyk.io/test/github/commercetools/commercetools-sync-java/4b2e26113d591bda158217c5dc1cf80a88665646)
 
 
@@ -56,18 +56,18 @@ Here are the most popular ones:
 <dependency>
   <groupId>com.commercetools</groupId>
   <artifactId>commercetools-sync-java</artifactId>
-  <version>2.2.0</version>
+  <version>2.2.1</version>
 </dependency>
 ````
 #### Gradle
 ````groovy
-implementation 'com.commercetools:commercetools-sync-java:2.2.0'
+implementation 'com.commercetools:commercetools-sync-java:2.2.1'
 ````
 #### SBT 
 ````
-libraryDependencies += "com.commercetools" % "commercetools-sync-java" % "2.2.0"
+libraryDependencies += "com.commercetools" % "commercetools-sync-java" % "2.2.1"
 ````
 #### Ivy 
 ````xml
-<dependency org="com.commercetools" name="commercetools-sync-java" rev="2.2.0"/>
+<dependency org="com.commercetools" name="commercetools-sync-java" rev="2.2.1"/>
 ````

docs/RELEASE_NOTES.md

@@ -28,6 +28,16 @@
 
 -->
 
+### 2.2.1 - Sep 29, 2020
+[Commits](https://github.com/commercetools/commercetools-sync-java/compare/2.2.0...2.2.1) |
+[Javadoc](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/) | 
+[Jar](https://bintray.com/commercetools/maven/commercetools-sync-java/2.2.1)
+
+ 🐞 **Bug Fixes** (1)
+    - **Product Sync** - Fixed a bug in the `ProductSync` related handling of unresolved product references provided in 
+    different batches. [#580](https://github.com/commercetools/commercetools-sync-java/issues/580)
+
+
 ### 2.2.0 - Sep 25, 2020
 [Commits](https://github.com/commercetools/commercetools-sync-java/compare/2.1.0...2.2.0) |
 [Javadoc](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/) | 

docs/usage/CART_DISCOUNT_SYNC.md

@@ -32,7 +32,7 @@ fields set, otherwise they won't be matched.
 Types are matched by their `key`s. Therefore, in order for the sync to resolve the 
 actual ids of the type reference, the `key` of the `Type` has to be supplied.
 
-   - When syncing from a source commercetools project, you can use [`mapToCartDiscountDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/com/commercetools/sync/cartdiscounts/utils/CartDiscountReferenceResolutionUtils.html#mapToCartDiscountDrafts-java.util.List-)
+   - When syncing from a source commercetools project, you can use [`mapToCartDiscountDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/com/commercetools/sync/cartdiscounts/utils/CartDiscountReferenceResolutionUtils.html#mapToCartDiscountDrafts-java.util.List-)
     method that maps from a `CartDiscount` to `CartDiscountDraft` in order to make them ready for reference resolution by the sync:
 
     ````java

docs/usage/CATEGORY_SYNC.md

@@ -34,7 +34,7 @@ otherwise they won't be matched.
 These references are matched by their `key`s. Therefore, in order for the sync to resolve the 
 actual ids of the references, their `key`s has to be supplied.
 
-   - When syncing from a source commercetools project, you can use [`mapToCategoryDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/com/commercetools/sync/categories/utils/CategoryReferenceResolutionUtils.html#mapToCategoryDrafts-java.util.List-)
+   - When syncing from a source commercetools project, you can use [`mapToCategoryDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/com/commercetools/sync/categories/utils/CategoryReferenceResolutionUtils.html#mapToCategoryDrafts-java.util.List-)
      method that maps from a `Category` to `CategoryDraft` in order to make them ready for reference resolution by the sync:
      ````java     
      final List<CategoryDraft> categoryDrafts = CategoryReferenceResolutionUtils.mapToCategoryDrafts(categories);

docs/usage/IMPORTANT_USAGE_TIPS.md

@@ -31,7 +31,7 @@ productSync.sync(batch1)
 By design, scaling the sync process should **not** be done by executing the batches themselves in parallel. However, it can be done either by:
 
  - Changing the number of [max parallel requests](https://github.com/commercetools/commercetools-sync-java/tree/master/src/main/java/com/commercetools/sync/commons/utils/ClientConfigurationUtils.java#L116) within the `sphereClient` configuration. It defines how many requests the client can execute in parallel.
- - or changing the draft [batch size](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/com/commercetools/sync/commons/BaseSyncOptionsBuilder.html#batchSize-int-). It defines how many drafts can one batch contain.
+ - or changing the draft [batch size](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/com/commercetools/sync/commons/BaseSyncOptionsBuilder.html#batchSize-int-). It defines how many drafts can one batch contain.
 
 The current overridable default [configuration](https://github.com/commercetools/commercetools-sync-java/tree/master/src/main/java/com/commercetools/sync/commons/utils/ClientConfigurationUtils.java#L45) of the `sphereClient` 
 is the recommended good balance for stability and performance for the sync process.

docs/usage/INVENTORY_SYNC.md

@@ -32,7 +32,7 @@ against a [InventoryEntryDraft](https://docs.commercetools.com/http-api-projects
    references are matched by their `key`s. Therefore, in order for the sync to resolve the actual ids of those references,
    their `key`s has to be supplied.
 
-   - When syncing from a source commercetools project, you can use [`mapToInventoryEntryDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/com/commercetools/sync/inventories/utils/InventoryReferenceResolutionUtils.html#mapToInventoryEntryDrafts-java.util.List-)
+   - When syncing from a source commercetools project, you can use [`mapToInventoryEntryDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/com/commercetools/sync/inventories/utils/InventoryReferenceResolutionUtils.html#mapToInventoryEntryDrafts-java.util.List-)
      method that that maps from a `InventoryEntry` to `InventoryEntryDraft` in order to make them ready for reference resolution by the sync:
      ````java
      final List<InventoryEntryDraft> inventoryEntryDrafts = InventoryReferenceResolutionUtils.mapToInventoryEntryDrafts(inventoryEntries);

docs/usage/PRODUCT_SYNC.md

@@ -38,7 +38,7 @@ all the variants in the target project are expected to have the `sku` fields set
 of the product also have prices, where each price also has some references including a reference to the `Type` of its 
 custom fields and a reference to a `channel`. All these referenced resources are matched by their `key`s. Therefore, in 
 order for the sync to resolve the actual ids of those references, those `key`s have to be supplied in the following way:
-    - When syncing from a source commercetools project, you can use [`mapToProductDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/com/commercetools/sync/products/utils/ProductReferenceResolutionUtils.html#mapToProductDrafts-java.util.List-)
+    - When syncing from a source commercetools project, you can use [`mapToProductDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/com/commercetools/sync/products/utils/ProductReferenceResolutionUtils.html#mapToProductDrafts-java.util.List-)
      method that maps from a `Product` to `ProductDraft` in order to make them ready for reference resolution by the sync:
      ````java
      final List<ProductDraft> productDrafts = ProductReferenceResolutionUtils.mapToProductDrafts(products);

docs/usage/PRODUCT_TYPE_SYNC.md

@@ -38,7 +38,7 @@ references, those `key`s have to be supplied in the following way:
     - Provide the `key` value on the `id` field of the reference. This means that calling `getId()` on the
     reference would return its `key`. 
 
-        **Note**: When syncing from a source commercetools project, you can use [`mapToProductTypeDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.0/com/commercetools/sync/producttypes/utils/ProductTypeReferenceResolutionUtils.html#mapToProductTypeDrafts-java.util.List-)
+        **Note**: When syncing from a source commercetools project, you can use [`mapToProductTypeDrafts`](https://commercetools.github.io/commercetools-sync-java/v/2.2.1/com/commercetools/sync/producttypes/utils/ProductTypeReferenceResolutionUtils.html#mapToProductTypeDrafts-java.util.List-)
 
          that replaces the references id fields with keys, in order to make them ready for reference resolution by the sync:
          ````java

docs/usage/QUICK_START.md

@@ -37,7 +37,7 @@
 <dependency>
   <groupId>com.commercetools</groupId>
   <artifactId>commercetools-sync-java</artifactId>
-  <version>2.2.0</version>
+  <version>2.2.1</version>
 </dependency>
 ````
 - For Gradle users:
@@ -48,7 +48,7 @@ implementation 'com.commercetools.sdk.jvm.core:commercetools-java-client:1.53.0'
 implementation 'com.commercetools.sdk.jvm.core:commercetools-convenience:1.53.0'
 
 // Add commercetools-sync-java dependency.
-implementation 'com.commercetools:commercetools-sync-java:2.2.0'
+implementation 'com.commercetools:commercetools-sync-java:2.2.1'
 ````
 
 ### 2. Setup Syncing Options

mkdocs.yml

@@ -59,7 +59,7 @@ nav:
       - Advanced:
         - Sync Options: usage/SYNC_OPTIONS.md
         - Usage Tips: usage/IMPORTANT_USAGE_TIPS.md
-  - Javadoc: https://commercetools.github.io/commercetools-sync-java/v/2.2.0/
+  - Javadoc: https://commercetools.github.io/commercetools-sync-java/v/2.2.1/
   - Release notes: RELEASE_NOTES.md
   - Roadmap: https://github.com/commercetools/commercetools-sync-java/milestones
   - Issues: https://github.com/commercetools/commercetools-sync-java/issues

src/integration-test/java/com/commercetools/sync/integration/externalsource/products/ProductSyncWithReferencedProductsInAnyOrderIT.java

@@ -112,6 +112,7 @@ private ProductSyncOptions buildSyncOptions() {
                                         .errorCallback((syncException, draft, product, updateActions)
                                             -> collectErrors(syncException.getMessage(), syncException))
                                         .beforeUpdateCallback(this::collectActions)
+                                        .batchSize(1)
                                         .warningCallback(warningCallback)
                                         .build();
     }

src/main/java/com/commercetools/sync/categories/helpers/CategoryBatchValidator.java

@@ -39,7 +39,6 @@ public CategoryBatchValidator(
      * <ol>
      * <li>It is not null</li>
      * <li>It has a key which is not blank (null/empty)</li>
-     * </li>
      * </ol>
      *
      * @param categoryDrafts the category drafts to validate and collect referenced keys.

src/main/java/com/commercetools/sync/inventories/helpers/InventoryBatchValidator.java

@@ -38,7 +38,6 @@ public InventoryBatchValidator(
      * <ol>
      * <li>It is not null</li>
      * <li>It has a sku which is not blank (null/empty)</li>
-     * </li>
      * </ol>
      *
      * @param inventoryDrafts the inventory drafts to validate and collect referenced keys.

src/main/java/com/commercetools/sync/products/helpers/ProductReferenceResolver.java

@@ -432,6 +432,6 @@ public CompletableFuture<Map<String, String>> populateKeyToIdCachesForReferenced
         }
 
         return collectionOfFuturesToFutureOfCollection(futures, toList())
-            .thenApply(maps -> productKeys.isEmpty() ? Collections.emptyMap() : maps.get(0));
+            .thenCompose(ignored -> productService.cacheKeysToIds(Collections.emptySet())); // return all cache.
     }
 }