Add opensearch to the toolchain #8
Conversation
…lopment environment Script install OpenSearch from a tarball. File tarbasll.md contains instructions on how to use the script correctly
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @@ -0,0 +1,77 @@ | |||
| wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.18.0/opensearch-2.18.0-linux-x64.tar.gz | |||
Check failure
Code scanning / shellcheck
SC2148 Error
| @@ -0,0 +1,77 @@ | |||
| wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.18.0/opensearch-2.18.0-linux-x64.tar.gz | |||
| tar -xvf opensearch-2.18.0-linux-x64.tar.gz | |||
| cd opensearch-2.18.0/ | |||
Check warning
Code scanning / shellcheck
SC2164 Warning
| curl -X GET https://localhost:9200 -u 'admin:admin' --insecure | ||
| curl -X GET https://localhost:9200/_cat/plugins?v -u 'admin:admin' --insecure | ||
|
|
||
| cd config/ |
Check warning
Code scanning / shellcheck
SC2164 Warning
|
|
||
| cd .. | ||
| export OPENSEARCH_JAVA_HOME=/jdk | ||
| cd config/ |
Check warning
Code scanning / shellcheck
SC2164 Warning
| EOF | ||
|
|
||
| cd .. | ||
| cd plugins/opensearch-security/tools |
Check warning
Code scanning / shellcheck
SC2164 Warning
| OPENSEARCH_JAVA_HOME=~/scylladb/scripts/opensearch-2.18.0/jdk ./hash.sh | ||
|
|
||
| cd ../../../ | ||
| cd config/opensearch-security |
Check warning
Code scanning / shellcheck
SC2164 Warning
QuerthDP
changed the title
| @@ -0,0 +1,27 @@ | |||
| run script: bash tarball.sh | |||
There was a problem hiding this comment.
Installing OpenSearch should not require a new, manual process that is separate from our default flow for installing things. More specifically, this should be taken care of in ./install-dependencies.sh. Many things depend on this, including our frozen toolchain based on docker images (see tools/toolchain/dbuild).
test/pylib/opensearch_cluster.py
Outdated
| await loop.run_in_executor(io_executor, partial(shutil.rmtree, directory, *args, **kwargs)) | ||
|
|
||
|
|
||
| class OpenSearchServer: |
There was a problem hiding this comment.
Instead of mirroring the code in test/pylib/scylla_cluster.py, I would recommend looking at our current support code for running minio instead - AFAICS we use it in tests which require access to the S3 API, minio has a compatible API.
I would take a look at test/pylib/minio_server.py. The MinioServer is used from the test.py initialization code, we have only one instance of it for the whole duration of a test.py execution - I think it will be completely fine for our purposes, we can share a single instance in all tests and make sure that names of the objects that we will use will be unique and/or namespaced somehow.
Minio is downloaded in ./install-dependencies.sh, and I think that it's where you should download OpenSearch as well. Minio is a single binary though, so it is installed right into /usr/bin, whereas we need something more complicated for OpenSearch. There is the /opt/scylladb/dependencies directory to which the install-dependencies.sh process downloads some binaries already, so I think we can use it for this purpose as well. Another example - we used to download some (build-time) dependencies to that repository and extract them - see https://github.com/scylladb/scylladb/pull/9151/files.
test/topology/conftest.py
Outdated
| @pytest.fixture(scope="function") | ||
| async def opensearch(): | ||
| cluster = OpenSearchCluster(logger=logging.getLogger()) | ||
| await cluster.install_and_start() | ||
| yield cluster |
There was a problem hiding this comment.
As a proof that this PR works and sets up an OpenSearch instance properly, the dummy test should attempt to send some request to the instance - something non-destructive, like e.g. list the current indexes or something like that.
Add download and unpack of OpenSearch library to install-dependencies.sh
| OPEN_SEARCH_VERSION=2.18.0 | ||
| declare -A OPEN_SEARCH_CHECKSUM=( | ||
| ["x86_64"]=ae3cb4107b2e0cdbb9b98bb4e5f2f019b736b68e995442c718e459d39ff01df1 | ||
| ["aarch64"]= # TODO: Add the checksum for the "aarch64" architecture. |
Check warning
Code scanning / shellcheck
SC2192 Warning
Remove @smoczy123 changes in test_raft_service_levels.py and conftest.py, feels unnecessary here. Add minimal implementation of new OpenSearch test suite (the most basic suite.yaml, surely to look into in the future), and test.py OpensearchTestSuite and OpensearchTest classes (still need to be adjusted properly, them being just copy of Topology* for now). Add simple OpenSearch test creating index and listing indices. The test to work needs a manual installation of OpenSearch instance, and will pass only once, as the instance does not allow to recreate the index. This is still hardly WIP, will be adjusted and rebased properly soon.
Bump OpenSearch version from 2.18.0 to 2.19.0. Adjust installation script to allow the caller to run OpenSearch instance. Fix simple test to be reusable. Add very simple opensearch cluster management (start/stop).
Fixes #4