Merge pull request #73 from spryker/develop

Upgrade main ES service from 6.x to 7.x

README.md

@@ -1,9 +1,8 @@
 # Spryker OS - Reference repository for DevVM
 
-This repository contains Vagrantfile which is responsible for setting up
-initial state of the Dev VM. Provisioning of the machine is done using SaltStack.
+This repository contains the Vagrantfile for setting up initial state of the DevVM. Provisioning of the machine is done using SaltStack.
 
-Please refer to the [Installation guide - B2C](https://documentation.spryker.com/installation/installation-guide-b2c.htm) or [Installation guide - B2B](https://documentation.spryker.com/installation/installation-guide-b2b.htm) to install Spryker.
+To install Spryker, refer to [Installation guide - B2C](https://documentation.spryker.com/docs/en/b2c-demo-shop-installation-mac-os-or-linux-with-devvm) or [Installation guide - B2B](https://documentation.spryker.com/docs/en/installation-guide-b2b).
 
 This repository contains:
  - [saltstack](saltstack) - SaltStack implementation for provisioning reference infrastructure for development
@@ -16,16 +15,14 @@ This repository contains:
  - `vagrant-hostmanager` plugin
 
 ## VM Settings
-The VM will start with the default configuration for project `demoshop` and IP `10.10.0.33`.
-If you would like to change project name, you need to edit `Vagrantfile` and change value of
-variable `VM_PROJECT` (ie. to demoshop, project) and `VM_IP` - last digit only. The IP address must
-be unique, so each VM on your workstation must have unique IP address.
-You should also adjust `VM_DOMAIN` to the value that corresponds to your config_default-development
-hostnames. If you do not specify value of `VM_DOMAIN`, it will take the value
-from `VM_PROJECT`.
+The VM starts with the default configuration for project `demoshop` and IP `10.10.0.33`.
+To change project name, edit `Vagrantfile` and change the value of
+the `VM_PROJECT` (ie. to demoshop, project) and `VM_IP`(digits only) variables. The IP address must
+be unique, so each VM on your workstation has a unique IP address.
 
-As an example, for default `VM_PROJECT=demoshop` - following hostnames will
-be generated:
+Adjust the `VM_DOMAIN` variable value according to your config_default-development hostnames. 
+The default value is taken from the `VM_PROJECT` variable. For example, the following hostnames 
+are generated for the default value `VM_PROJECT=demoshop`:
 
 ```
 www.de.demoshop.local
@@ -36,18 +33,18 @@ zed-test.de.demoshop.local
 static-test.demoshop.local
 ```
 
-## Note on PHP opcache
-In order to use opcache for CLI calls as well, the VM ships with PHP opcache file cache enabled. Cache contents are stored in `/var/tmp/opcache` directory. After enabling/disabling PHP modules (and as a possible fix if you are getting unexpected PHP error, like Segmentation fault), you must clean the cache directory with:
+## Note on PHP OPcache
+To use OPcache for CLI calls, the VM ships with PHP opcache file cache enabled. Cache contents are stored in `/var/tmp/opcache`. After enabling or disabling PHP modules, for example to fix an unexpected PHP error like Segmentation fault, make sure to clear cache by running the command:
 ```
 sudo rm -rf /var/tmp/opcache/*; sudo systemctl restart php7.4-fpm
 ```
 
 ## Customizing the VM
 
 ### PHP development modules
-The PHP module `xdebug` are pre-installed on the DevVM, but not enabled by default.
-Using `xdebug` and `opcache` at the same time might be dangerous and is not recommended.
-To enable xdebug, use the following commands:
+The PHP module `xdebug` is pre-installed on the DevVM, but not enabled by default.
+We do not recommend using `xdebug` and `opcache` simultaneously as it is dangerous.
+To enable Xdebug, run the commands:
 ```
 # Enable XDebug, disable OpCache, clear disk cache, restart FPM
 sudo -i bash -c " \
@@ -60,8 +57,8 @@ sudo -i bash -c " \
   "
 ```
 
-Running with xdebug enabled and opcache disabled will cause the application to be slower, so consider
-enabling it only when you need. To disable xdebug and re-enable opcache, use the following commands:
+Running with xdebug enabled and opcache disabled cause the application to be slower, so consider
+keeping it disabled when not needed. To disable Xdebug and re-enable OPcache, run the commands:
 ```
 # Disable XDebug, enable OpCache, clear disk cache, restart FPM
 sudo -i bash -c " \
@@ -76,15 +73,17 @@ sudo -i bash -c " \
 
 
 ## Version tree and lifecycle
-After release `ci-119` we decided to stop using auto-incremented release numbers and switch to semantic versioning. Next version becomes `1.0.0`.
-We follow git-flow - branch `master` is used to release tested features, where branch `develop` is used for release candidates (tags like `v2.0.0-RC1`)
-and is merged into master whenever we officialy release new version, after internal QA process.
+After release `ci-119`, we decided to stop using auto-incremented release numbers and switch to semantic versioning. Next version becomes `1.0.0`.
+We follow git-flow:
+* Branch `master` is used to release tested features.
+* Branch `develop` is used for release candidates (tags like `v2.0.0-RC1`) and is merged into master whenever we officialy release a new version, after internal QA process.
 
 
 ### 1.x.x
-1.x.x is old stable version, with PHP 7.1 and Elasticsearch 2.x
+PHP 7.1 and Elasticsearch 2.x [EOL]
 
 ### 2.x.x
-2.x.x is non-backward-compatible version, which includes PHP 7.2 and Elasticsearch 5.x
-
+Includes PHP 7.2 and Elasticsearch 5.x
 
+### 3.x.x
+Upgraded Elasticsearch to 7.8.1, Kibana to 7.8.1, since 3.1.0 also PHP 7.4

Vagrantfile

@@ -40,7 +40,7 @@ else
   # Settings for the Virtualbox VM
   VM_IP_PREFIX = ENV['VM_IP_PREFIX'] || '10.10.0.'                         # Prefix for IP address of DEV VM
   VM_IP        = ENV['VM_IP']        || VM_IP_PREFIX + unique_byte         # IP Address of the DEV VM, must be unique
-  VM_MEMORY    = ENV['VM_MEMORY']    || '3200'                             # Number of memory for DEV VM, in MB
+  VM_MEMORY    = ENV['VM_MEMORY']    || '4000'                             # Number of memory for DEV VM, in MB
   VM_CPUS      = ENV['VM_CPUS']      || '4'                                # Number of CPU cores for DEV VM
   VM_NAME      = ENV['VM_NAME']      || "Spryker Dev VM (#{VM_PROJECT})"   # Display name for VirtualBox
   VM_SKIP_SF   = ENV['VM_SKIP_SF']   || '0'                                # Don't mount shared folders

Vagrantfile-quick

@@ -48,7 +48,7 @@ else
   # Settings for the Virtualbox VM
   VM_IP_PREFIX  = ENV['VM_IP_PREFIX'] || '10.10.0.'                         # Prefix for IP address of DEV VM
   VM_IP         = ENV['VM_IP']        || VM_IP_PREFIX + unique_byte         # IP Address of the DEV VM, must be unique
-  VM_MEMORY     = ENV['VM_MEMORY']    || '3200'                             # Number of memory for DEV VM, in MB
+  VM_MEMORY     = ENV['VM_MEMORY']    || '4000'                             # Number of memory for DEV VM, in MB
   VM_CPUS       = ENV['VM_CPUS']      || '4'                                # Number of CPU cores for DEV VM
   VM_NAME       = ENV['VM_NAME']      || "Spryker Dev VM (#{VM_PROJECT})"   # Display name for VirtualBox
   VM_SKIP_SF    = ENV['VM_SKIP_SF']   || '0'                                # Don't mount shared folders

pillar/dev/elasticsearch/init.sls

@@ -1,7 +1,7 @@
 # Note: Elasticsearch plugins should have version, which fits installed elasticsearch version
 
 elasticsearch:
-  version: 6.8.6
+  version: 7.8.1
   # plugins:
   #   head:
   #     name: mobz/elasticsearch-head

pillar/dev/elk/init.sls

@@ -9,4 +9,4 @@ elk:
     port: 10005
 
   kibana:
-    version: 6.8.6
+    version: 7.8.1

pillar/dev/top.sls

@@ -8,4 +8,4 @@ dev:
     - postgresql
     - rabbitmq
     - elasticsearch
-    - elk
+    - elk

saltstack/base/elasticsearch/environments.sls

@@ -8,6 +8,7 @@
 {% from 'settings/init.sls' import settings with context %}
 {% from 'elasticsearch/macros/elasticsearch_instance.sls' import elasticsearch_instance with context %}
 
+
 /etc/logrotate.d/elasticsearch-instances:
   file.managed:
     - source: salt://elasticsearch/files/etc/logrotate.d/elasticsearch-instances

saltstack/base/elasticsearch/es6.sls

@@ -0,0 +1,44 @@
+#
+# Install Elasticsearch 6.8.6 as the second service
+#
+
+install-elasticsearch6:
+  cmd.run:
+    - name: cd /opt && wget -q https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.6.tar.gz && tar zxf elasticsearch-6.8.6.tar.gz && rm -f elasticsearch-6.8.6.tar.gz && chown -R elasticsearch. /opt/elasticsearch-6.8.6 && rm -rf /opt/elasticsearch-6.8.6/config/*
+    - unless: test -d /opt/elasticsearch-6.8.6
+
+/opt/elasticsearch-6.8.6/config:
+  file.recurse:
+    - source: salt://elasticsearch/files/es6_instance/config
+    - user: elasticsearch
+    - group: elasticsearch
+    - file_mode: 644
+    - dir_mode: 755
+
+/etc/systemd/system/elasticsearch6-development.service:
+  file.managed:
+    - source: salt://elasticsearch/files/es6_instance/etc/systemd/system/elasticsearch6.service
+    - mode: 644
+    - user: root
+    - group: root
+
+/etc/default/elasticsearch6-development:
+  file.managed:
+    - source: salt://elasticsearch/files/es6_instance/etc/default/elasticsearch6
+    - mode: 644
+    - user: root
+    - group: root
+
+/home/vagrant/es6.sh:
+  file.managed:
+    - source: salt://elasticsearch/files/es6_instance/es6.sh
+    - mode: 744
+    - user: root
+    - group: root
+
+/home/vagrant/es7.sh:
+  file.managed:
+    - source: salt://elasticsearch/files/es6_instance/es7.sh
+    - mode: 744
+    - user: root
+    - group: root

saltstack/base/elasticsearch/files/elasticsearch_instance/etc/default/elasticsearch

@@ -4,3 +4,4 @@ ES_PATH_CONF=/etc/elasticsearch-{{ environment }}
 RESTART_ON_UPGRADE=true
 LOG_PATH=/data/logs/{{ environment }}/elasticsearch
 ES_JAVA_OPTS="-Des.path.logs/${LOG_PATH}"
+JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/jre"

saltstack/base/elasticsearch/files/elasticsearch_instance/etc/elasticsearch/elasticsearch.yml

@@ -29,6 +29,7 @@ bootstrap.memory_lock: {{ salt['pillar.get']('elasticsearch.environments:mlockal
 
 ############################## Network And HTTP ##############################
 network.bind_host: 0.0.0.0
+network.host: 0.0.0.0
 network.publish_host: {{ settings.publish_ip }}
 http.port: {{ settings.environments[environment].elasticsearch.http_port }}
 transport.tcp.port: {{ settings.environments[environment].elasticsearch.transport_port }}
@@ -46,11 +47,12 @@ cluster.routing.allocation.node_initial_primaries_recoveries: 4
 cluster.routing.allocation.node_concurrent_recoveries: 2
 cluster.routing.allocation.disk.watermark.low: 90%
 cluster.routing.allocation.disk.watermark.high: 95%
+cluster.initial_master_nodes : ["{{ grains.fqdn }}"]
+discovery.seed_hosts : ["{{ grains.fqdn }}"]
 
 
 ################################### Indices ##################################
 indices.recovery.max_bytes_per_sec: 50mb
-##indices.store.throttle.max_bytes_per_sec: 100mb
 indices.fielddata.cache.size: 100M
 
 
@@ -64,3 +66,9 @@ action.destructive_requires_name: true
 
 ################################# Scripting ##################################
 ##script.inline: true
+
+################################# Disable X-Pack extension ###################
+xpack.security.enabled: false
+xpack.watcher.enabled: false
+xpack.ml.enabled: false
+xpack.graph.enabled: false

saltstack/base/elasticsearch/files/es6_instance/config/elasticsearch.yml

@@ -0,0 +1,57 @@
+################################### Cluster ##################################
+cluster.name: spryker-development
+
+
+#################################### Node ####################################
+node.name: vm-suite
+node.master: true
+node.data: true
+node.ingest: true
+node.max_local_storage_nodes: 1
+
+
+#################################### Paths ###################################
+path.data: /data/shop/development/shared/elasticsearch
+path.logs: /data/logs/development/elasticsearch
+path.repo: ["/data/shop/development/shared/elasticsearch/snapshots"]
+
+################################### Memory ###################################
+bootstrap.memory_lock: true
+
+
+############################## Network And HTTP ##############################
+network.bind_host: 0.0.0.0
+network.publish_host: localhost
+http.port: 10005
+transport.tcp.port: 20005
+transport.tcp.compress: false
+
+
+################################### Gateway ##################################
+gateway.recover_after_nodes: 1
+gateway.recover_after_time: 3m
+gateway.expected_nodes: 1
+
+
+############################# Cluster management #############################
+cluster.routing.allocation.node_initial_primaries_recoveries: 4
+cluster.routing.allocation.node_concurrent_recoveries: 2
+cluster.routing.allocation.disk.watermark.low: 90%
+cluster.routing.allocation.disk.watermark.high: 95%
+
+
+################################### Indices ##################################
+indices.recovery.max_bytes_per_sec: 50mb
+indices.fielddata.cache.size: 100M
+
+
+################################## Discovery #################################
+discovery.zen.minimum_master_nodes: 1
+discovery.zen.ping_timeout: 5s
+
+
+################################### Safety ###################################
+action.destructive_requires_name: true
+
+################################# Scripting ##################################
+#script.inline: true