Skip to content

Commit

Permalink
Merge branch 'master' into jeanbez-patch-8
Browse files Browse the repository at this point in the history
  • Loading branch information
jeanbez authored Dec 11, 2023
2 parents 6c7bcf8 + 7b2a7d8 commit 75c96af
Show file tree
Hide file tree
Showing 15 changed files with 667 additions and 23 deletions.
2 changes: 1 addition & 1 deletion .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ metrics-perlmutter:
- export RESULTS="/global/homes/j/jeanbez/h5bench-ci-metrics/perlmutter/$(date +%s)"
- mkdir -p $RESULTS
- echo "Fetching Darshan log file"
- cp /pscratch/darshanlogs/$(date +%Y)/$(date +%-m)/$(date +%d | sed 's/^0*//')/jeanbez_h5bench* ${RESULTS}
- cp /pscratch/darshanlogs/$(date +%Y)/$(date +%-m)/$(date +%d | sed 's/^0*//')/jeanbez_h5bench* ${RESULTS} 2> /dev/null || true
- echo "Fetching runtime files"
- cp h5bench-suite-sync.err ${RESULTS}
- cp h5bench-suite-sync.out ${RESULTS}
Expand Down
61 changes: 61 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: h5bench
message: >-
If you use this software, please cite it using the
metadata from this file.
type: software
authors:
- given-names: Suren
family-names: Byna
email: [email protected]
affiliation: Lawrence Berkeley National Laboratory
orcid: 'https://orcid.org/0000-0003-3048-3448'
- given-names: Jean Luca
family-names: Bez
email: [email protected]
affiliation: Lawrence Berkeley National Laboratory
orcid: 'https://orcid.org/0000-0002-3915-1135'
- given-names: Tonglin
family-names: Li
affiliation: Lawrence Berkeley National Laboratory
- given-names: Quincey
family-names: Koziol
affiliation: National Energy Research Scientific Computing Center
- given-names: Houjun
family-names: Tang
email: [email protected]
affiliation: Lawrence Berkeley National Laboratory
orcid: 'https://orcid.org/0000-0001-7038-8360'
- given-names: Qiao
family-names: Kang
affiliation: Lawrence Berkeley National Laboratory
identifiers:
- type: url
value: >-
https://sdm.lbl.gov/~sbyna/research/papers/2021/202106-CUG_2021_h5bench.pdf
description: CUG Paper
- type: url
value: 'https://github.com/hpc-io/h5bench'
description: Github Repository
- type: url
value: 'https://h5bench.readthedocs.io'
description: h5bench Documentation
repository-code: 'https://github.com/hpc-io/h5bench'
abstract: >-
h5bench is a suite of parallel I/O benchmarks or kernels
representing I/O patterns that are commonly used in HDF5
applications on high performance computing systems.
h5bench measures I/O performance from various aspects,
including the I/O overhead, observed I/O rate, etc.
keywords:
- HDF5
- I/O
- HPC
- I/O benchmark
- I/O kernel
commit: ' 67a3e6d91508e5ab77db79c6187b2eb3c96119e0'
version: v.1.4
date-released: '2023-05-31'
159 changes: 139 additions & 20 deletions commons/h5bench_util.c
Original file line number Diff line number Diff line change
Expand Up @@ -587,23 +587,77 @@ _set_io_pattern(bench_params *params_in_out)
(params_in_out->io_op == IO_APPEND)) { // file --> mem
if (params_in_out->mem_pattern == PATTERN_CONTIG) {
if (params_in_out->file_pattern == PATTERN_CONTIG) {
switch (params_in_out->num_dims) {
case 1:
(*params_in_out).access_pattern.pattern_read = CONTIG_1D;
ret = 0;
break;
case 2:
(*params_in_out).access_pattern.pattern_read = CONTIG_2D;
ret = 0;
break;
case 3:
(*params_in_out).access_pattern.pattern_read = CONTIG_3D;
ret = 0;
break;
default:
ret = -1;
printf("%s() failed on line %d\n", __func__, __LINE__);
break;
if (params_in_out->read_option == LDC) {
switch (params_in_out->num_dims) {
case 2:
(*params_in_out).access_pattern.pattern_read = LDC_2D;
ret = 0;
break;
default:
ret = -1;
printf("%s(). Unexpected Dimensions for LDC. failed on line %d\n", __func__,
__LINE__);
break;
}
}
else if (params_in_out->read_option == RDC) {
switch (params_in_out->num_dims) {
case 2:
(*params_in_out).access_pattern.pattern_read = RDC_2D;
ret = 0;
break;
default:
ret = -1;
printf("%s(). Unexpected Dimensions for RDC. failed on line %d\n", __func__,
__LINE__);
break;
}
}
else if (params_in_out->read_option == PRL) {
switch (params_in_out->num_dims) {
case 2:
(*params_in_out).access_pattern.pattern_read = PRL_2D;
ret = 0;
break;
default:
ret = -1;
printf("%s(). Unexpected Dimensions for PRL. failed on line %d\n", __func__,
__LINE__);
break;
}
}
else if (params_in_out->read_option == CS) {
switch (params_in_out->num_dims) {
case 2:
(*params_in_out).access_pattern.pattern_read = CS_2D;
ret = 0;
break;
default:
ret = -1;
printf("%s(). Unexpected Dimensions for CS. failed on line %d\n", __func__,
__LINE__);
break;
}
}
else {
switch (params_in_out->num_dims) {
case 1:
(*params_in_out).access_pattern.pattern_read = CONTIG_1D;
ret = 0;
break;
case 2:
(*params_in_out).access_pattern.pattern_read = CONTIG_2D;
ret = 0;
break;
case 3:
(*params_in_out).access_pattern.pattern_read = CONTIG_3D;
ret = 0;
break;
default:
ret = -1;
printf("%s() failed on line %d\n", __func__, __LINE__);
break;
}
}
}
else if (params_in_out->file_pattern == PATTERN_STRIDED) {
Expand Down Expand Up @@ -772,15 +826,28 @@ _set_params(char *key, char *val_in, bench_params *params_in_out, int do_write)
}
}
else if (strcmp(key, "READ_OPTION") == 0) {
if (val_in[0] == 'F') { // FULL
if (strcmp(val_in, "FULL") == 0) { // FULL
(*params_in_out).read_option = READ_FULL;
}
else if (val_in[0] == 'P') { // PARTIAL
else if (strcmp(val_in, "PARTIAL") == 0) { // PARTIAL
(*params_in_out).read_option = READ_PARTIAL;
}
else if (val_in[0] == 'S') { // STRIDED
else if (strcmp(val_in, "STRIDED") == 0) { // STRIDED
(*params_in_out).read_option = READ_STRIDED;
}
else if (strcmp(val_in, "LDC") == 0) {
(*params_in_out).read_option = LDC;
}
else if (strcmp(val_in, "RDC") == 0) {
(*params_in_out).read_option = RDC;
}
else if (strcmp(val_in, "PRL") == 0) {
(*params_in_out).read_option = PRL;
}
else if (strcmp(val_in, "CS") == 0) {
(*params_in_out).read_option = CS;
}

else
(*params_in_out).read_option = READ_OPTION_INVALID;
}
Expand Down Expand Up @@ -873,12 +940,36 @@ _set_params(char *key, char *val_in, bench_params *params_in_out, int do_write)
return -1;
(*params_in_out).stride = num;
}
else if (strcmp(key, "STRIDE_SIZE_2") == 0) {
unsigned long long num = 0;
if (str_to_ull(val, &num) < 0)
return -1;
(*params_in_out).stride_2 = num;
}
else if (strcmp(key, "STRIDE_SIZE_3") == 0) {
unsigned long long num = 0;
if (str_to_ull(val, &num) < 0)
return -1;
(*params_in_out).stride_3 = num;
}
else if (strcmp(key, "BLOCK_SIZE") == 0) {
unsigned long long num = 0;
if (str_to_ull(val, &num) < 0)
return -1;
(*params_in_out).block_size = num;
}
else if (strcmp(key, "BLOCK_SIZE_2") == 0) {
unsigned long long num = 0;
if (str_to_ull(val, &num) < 0)
return -1;
(*params_in_out).block_size_2 = num;
}
else if (strcmp(key, "BLOCK_SIZE_3") == 0) {
unsigned long long num = 0;
if (str_to_ull(val, &num) < 0)
return -1;
(*params_in_out).block_size_3 = num;
}
else if (strcmp(key, "BLOCK_CNT") == 0) {
unsigned long long num = 0;
if (str_to_ull(val, &num) < 0)
Expand Down Expand Up @@ -985,7 +1076,11 @@ bench_params_init(bench_params *params_out)
(*params_out).num_dims = 1;

(*params_out).stride = 0;
(*params_out).stride_2 = 0;
(*params_out).stride_3 = 0;
(*params_out).block_size = 0;
(*params_out).block_size_2 = 0;
(*params_out).block_size_3 = 0;
(*params_out).block_cnt = 0;
(*params_out).dim_1 = 1;
(*params_out).dim_2 = 1;
Expand Down Expand Up @@ -1109,6 +1204,30 @@ read_config(const char *file_path, bench_params *params_out, int do_write)
return -1;
}
}
if (params_out->access_pattern.pattern_read == LDC_2D) {
if (params_out->block_size < 1 || params_out->block_size_2 < 1) {
printf("LDC read requires BLOCK_SIZE/BLOCK_SIZE_2 no less than 1.\n");
return -1;
}
}
if (params_out->access_pattern.pattern_read == RDC_2D) {
if (params_out->block_size < 1 || params_out->block_size_2 < 1) {
printf("RDC read requires BLOCK_SIZE/BLOCK_SIZE_2 no less than 1.\n");
return -1;
}
}
if (params_out->access_pattern.pattern_read == PRL_2D) {
if (params_out->block_size < 1 || params_out->block_size_2 < 1) {
printf("PRL read requires BLOCK_SIZE/BLOCK_SIZE_2 no less than 1.\n");
return -1;
}
}
if (params_out->access_pattern.pattern_read == CS_2D) {
if (params_out->stride < 1 || params_out->stride_2 < 1) {
printf("CS read requires STRIDE_SIZE/STRIDE_SIZE_2 no less than 1.\n");
return -1;
}
}
}
if (params_out->subfiling > 0 && params_out->data_coll == 1) {
printf("Subfiling does not support collective data buffering for data.\n");
Expand Down
19 changes: 18 additions & 1 deletion commons/h5bench_util.h
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@ typedef enum read_pattern {
STRIDED_1D,
CONTIG_2D,
CONTIG_3D,
LDC_2D,
RDC_2D,
CS_2D,
PRL_2D,
} read_pattern;

typedef enum pattern {
Expand All @@ -85,7 +89,16 @@ typedef enum io_operation {
IO_APPEND,
} io_operation;

typedef enum read_option { READ_OPTION_INVALID, READ_FULL, READ_PARTIAL, READ_STRIDED } read_option;
typedef enum read_option {
READ_OPTION_INVALID,
READ_FULL,
READ_PARTIAL,
READ_STRIDED,
LDC,
RDC,
PRL,
CS
} read_option;

typedef struct bench_params {
io_operation io_op;
Expand Down Expand Up @@ -115,7 +128,11 @@ typedef struct bench_params {
duration compute_time;
int num_dims;
unsigned long stride;
unsigned long stride_2;
unsigned long stride_3;
unsigned long block_size;
unsigned long block_size_2;
unsigned long block_size_3;
unsigned long block_cnt;
unsigned long dim_1;
unsigned long dim_2;
Expand Down
2 changes: 2 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
sphinx==6.2.1
sphinx-rtd-theme==1.2.2
sphinxemoji
urllib3==1.26.15
37 changes: 37 additions & 0 deletions h5bench_patterns/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
## Additions to H5Bench_read
4 new patterns are added to the h5bench read benchmark:
1. CS: Refers to the Cross Stencil data access pattern. A block of fixed sides is used to read data from HDF5. This block is given a fixed stride in each dimension and data till end of file is read.
2. LDC: Refers to the Left Diagonal Corner data access pattern. Data is read from two identical blocks of fixed sides, one in the top left corner and the other in the bottom right corner in the 2D HDF5 file
3. RDC: Refers to the Right Diagonal Corner data access pattern. Data is read from two identical blocks of fixed sides, one in the top right corner and the other in the bottom left corner in the 2D HDF5 file
4. PRL: Refers to the Peripheral data access pattern. Data is read from the periphery of the file, which is a frame of fixed width and height around the file.

These patterns work with a single MPI process. In case multiple processes are used, only the root performs the read operations and all other processes skip the reads. Illustrations of the patterns are given in the table below:

| Pattern -> | CS | LDC | RDC | PRL |
| ---------- | ---------- | ---------- | ---------- | ---------- |
|Illustration | ![CS Pattern](./images/CS.png) | ![LDC Pattern](./images/LDC.png) | ![RDC Pattern](./images/RDC.png) | ![PRL Pattern](./images/PRL.png) |



Steps for running these benchmarks are the same as the other benchmarks. All parameter and requirements and running instructions remain same, except for the following parameters which are additionally required to be provided in the configuration:
1. CS
| Parameter | Description |
| --------- | ----------- |
| `BLOCK_SIZE` | Size of the block of data along `dim_1` |
| `BLOCK_SIZE_2` | Size of the block of data along `dim_2` |
| `STRIDE_SIZE` | Size of the block of data along `dim_1` |
| `STRIDE_SIZE_2` | Size of the block of data along `dim_2` |

2. LDC/RDC
| Parameter | Description |
| --------- | ----------- |
| `BLOCK_SIZE` | Size of the block of data along `dim_1` |
| `BLOCK_SIZE_2` | Size of the block of data along `dim_2` |

3. PRL
| Parameter | Description |
| --------- | ----------- |
| `BLOCK_SIZE` | Size of the frame along `dim_1` |
| `BLOCK_SIZE_2` | Size of the frame along `dim_2` |

Exmaples of some configurations are provided in `h5bench/samples/sync-write-1d-contig-contig-read-<pattern>.json`
Loading

0 comments on commit 75c96af

Please sign in to comment.