Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client-2699 C Metrics #147

Closed
wants to merge 67 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
cf31bad
Initial pass at metrics
shannonklaus Jan 19, 2024
81f9f4d
Add event loop info to file printing
shannonklaus Jan 19, 2024
a823188
Merge branch 'stage' into CLIENT-2699_metrics
BrianNichols Jan 22, 2024
dc50d0f
Update copyright year, fix timeout and error count in node_stats
shannonklaus Jan 22, 2024
65a6f7e
Change enum to macros for latency_type, add comments to cluster, and …
shannonklaus Jan 22, 2024
8091d95
Restore common submodule change.
BrianNichols Jan 22, 2024
a53adb2
Update comments in metrics files, make timeout_count unsigned
shannonklaus Jan 22, 2024
b739b91
Add metrics_writer type
shannonklaus Jan 23, 2024
afd42f4
Add errors to enable and disable metrics
shannonklaus Jan 23, 2024
6775bc6
Comments and other small fixes
shannonklaus Jan 23, 2024
a0e3b62
Fix warnings and errors
shannonklaus Jan 23, 2024
8fe251a
Add as_string_builder_append_int64() and as_string_builder_append_uin…
BrianNichols Jan 23, 2024
233a236
Work on warnings
shannonklaus Jan 23, 2024
fc7f322
Fix warnings
shannonklaus Jan 24, 2024
a8f8bc0
Fixing bugs
shannonklaus Jan 25, 2024
aa92db7
Fixing more bugs
shannonklaus Jan 25, 2024
126270f
Add logging to help debug
shannonklaus Jan 31, 2024
a9c192b
Fix mac compiler errors/warnings.
BrianNichols Jan 31, 2024
2339eab
Add code to check for trailing slash and add if missing based on oper…
shannonklaus Jan 31, 2024
3887ac2
Free memory
shannonklaus Feb 1, 2024
7b9302f
Fix memory leak
shannonklaus Feb 2, 2024
f57b4d2
Fix formatting
shannonklaus Feb 2, 2024
343eb10
PUshing latest
shannonklaus Feb 5, 2024
3d804c0
Update cpu and mem
shannonklaus Feb 9, 2024
5b1ee26
Call new as_event_connection_complete() when connection is complete i…
BrianNichols Feb 9, 2024
2ef80a4
Merge branch 'stage' into CLIENT-2699_metrics
BrianNichols Feb 9, 2024
b7bedf3
Add connection metrics
shannonklaus Feb 12, 2024
5339598
Work on async and connection metrics
shannonklaus Feb 13, 2024
455a1e1
Update copyright year
shannonklaus Feb 13, 2024
ee3c585
Merge branch 'stage' into CLIENT-2699_metrics
BrianNichols Feb 13, 2024
6632734
Rename as_sum_init() to as_conn_stats_init() and make inline.
BrianNichols Feb 13, 2024
8dd9bee
Fix batch, scan, query transaction counts
shannonklaus Feb 14, 2024
26f8af5
Move assignment of cluster pointer
shannonklaus Feb 14, 2024
c9df71e
Change where begin is calculated, remove connection from transaction …
shannonklaus Feb 14, 2024
df40dd0
Add code for pipe connection metrics
shannonklaus Feb 14, 2024
abf2a60
Use _MSC_VER instead of _WIN32
shannonklaus Feb 14, 2024
bdb9bcc
remove unnecessary includes.
BrianNichols Feb 15, 2024
37ba6d3
Document metrics policy fields.
BrianNichols Feb 15, 2024
b920f37
Pass in report directory to metrics init, change from pointer to array
shannonklaus Feb 15, 2024
2306b3f
Add as_metrics_policy_set_report_dir().
BrianNichols Feb 15, 2024
b274d28
Move metrics_writer implementation to it's own separate files.
BrianNichols Feb 16, 2024
c85ba8f
Use cf_calloc to initialize as_metrics_writer struct. Ignore certain …
BrianNichols Feb 16, 2024
5d126c0
Working on error cases
shannonklaus Feb 20, 2024
a20aa21
Fix for unit tests stopping unexpectedly
shannonklaus Feb 20, 2024
c9ac449
Do not exit(-1) in aerospike_test.c
BrianNichols Feb 20, 2024
287dc63
Only reset error if a previous error was set.
BrianNichols Feb 20, 2024
3d280ee
To be consistent revert back to older tend warning message format whe…
BrianNichols Feb 20, 2024
e8d11cb
Changes after error testing, disable metrics in unit tests
shannonklaus Feb 21, 2024
5deb361
Move latency bucket structs to as_latency.{c,h}
BrianNichols Feb 21, 2024
b9bc5d0
Port previous commit to windows.
BrianNichols Feb 21, 2024
63d5315
Add metrics lock and run all metrics calls under this lock.
BrianNichols Feb 22, 2024
981701a
Comment out metrics enable in tests.
BrianNichols Feb 22, 2024
a12161d
Add timeout_count and retry_count to aerospike_stats.
BrianNichols Feb 22, 2024
1f0e5ef
Inline short cluster/node metrics functions.
BrianNichols Feb 23, 2024
5563928
Call as_node_add_error() on errors other than timeout.
BrianNichols Feb 23, 2024
574c55c
Call "as_cluster_add_retries(cluster, batch_nodes.size)" after batch_…
BrianNichols Feb 23, 2024
7192e98
Pack as_cluster_stats and as_node for optimal size.
BrianNichols Feb 23, 2024
37a374a
Optimize setting async cmd latency_type.
BrianNichols Feb 26, 2024
3f14ae6
Do not increment node error count on record not found.
BrianNichols Feb 26, 2024
39a3795
Print socket address and port separated by a comma to be consistent w…
BrianNichols Feb 26, 2024
f68b483
Initialize error_count, timeout_count before initializing node metrics.
BrianNichols Feb 26, 2024
082a088
Calculate latency on AEROSPIKE_ERR_RECORD_NOT_FOUND too.
BrianNichols Feb 26, 2024
7e1a304
Increment retries when applicable in sync scans.
BrianNichols Feb 27, 2024
b3c2289
Increment retries when applicable in sync queries.
BrianNichols Feb 27, 2024
ced01be
Do not pass in as_error to as_cluster_remove_nodes() because it never…
BrianNichols Feb 27, 2024
901ded2
Remove <p> as doxygen does not use it.
BrianNichols Feb 27, 2024
8a647af
Remove upgrade log
BrianNichols Feb 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,12 @@ AEROSPIKE += as_host.o
AEROSPIKE += as_info.o
AEROSPIKE += as_job.o
AEROSPIKE += as_key.o
AEROSPIKE += as_latency.o
AEROSPIKE += as_list_operations.o
AEROSPIKE += as_lookup.o
AEROSPIKE += as_map_operations.o
AEROSPIKE += as_metrics.o
AEROSPIKE += as_metrics_writer.o
AEROSPIKE += as_node.o
AEROSPIKE += as_operations.o
AEROSPIKE += as_partition.o
Expand Down
30 changes: 27 additions & 3 deletions src/include/aerospike/aerospike_stats.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright 2008-2021 Aerospike, Inc.
* Copyright 2008-2024 Aerospike, Inc.
*
* Portions may be licensed to Aerospike, Inc. under one or more contributor
* license agreements.
Expand Down Expand Up @@ -88,9 +88,16 @@ typedef struct as_node_stats_s {
as_conn_stats pipeline;

/**
* Node error count within current window.
* Transaction error count since node was initialized. If the error is retryable, multiple errors per
* transaction may occur.
*/
uint32_t error_count;
uint64_t error_count;

/**
* Transaction timeout count since node was initialized. If the timeout is retryable (ie socket timeout),
* multiple timeouts per transaction may occur.
*/
uint64_t timeout_count;

} as_node_stats;

Expand Down Expand Up @@ -128,6 +135,11 @@ typedef struct as_cluster_stats_s {
*/
as_event_loop_stats* event_loops;

/**
* Count of transaction retries since cluster was started.
*/
uint64_t retry_count;

/**
* Node count.
*/
Expand Down Expand Up @@ -242,6 +254,18 @@ aerospike_event_loop_stats(as_event_loop* event_loop, as_event_loop_stats* stats
AS_EXTERN char*
aerospike_stats_to_string(as_cluster_stats* stats);

static inline void
as_conn_stats_init(as_conn_stats* stats)
{
stats->in_pool = 0;
stats->in_use = 0;
stats->opened = 0;
stats->closed = 0;
}

void
as_conn_stats_sum(as_conn_stats* stats, as_async_conn_pool* pool);

#ifdef __cplusplus
} // end extern "C"
#endif
13 changes: 11 additions & 2 deletions src/include/aerospike/as_async.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright 2008-2023 Aerospike, Inc.
* Copyright 2008-2024 Aerospike, Inc.
*
* Portions may be licensed to Aerospike, Inc. under one or more contributor
* license agreements.
Expand Down Expand Up @@ -110,7 +110,9 @@ as_async_write_command_create(
cmd->flags = 0;
cmd->replica_size = pi->replica_size;
cmd->replica_index = 0;
cmd->latency_type = AS_LATENCY_TYPE_WRITE;
wcmd->listener = listener;
as_cluster_add_tran(cluster);
return cmd;
}

Expand All @@ -119,7 +121,8 @@ as_async_record_command_create(
as_cluster* cluster, const as_policy_base* policy, as_partition_info* pi,
as_policy_replica replica, uint8_t replica_index, bool deserialize, bool heap_rec,
uint8_t flags, as_async_record_listener listener, void* udata, as_event_loop* event_loop,
as_pipe_listener pipe_listener, size_t size, as_event_parse_results_fn parse_results
as_pipe_listener pipe_listener, size_t size, as_event_parse_results_fn parse_results,
as_latency_type latency_type
)
{
// Allocate enough memory to cover: struct size + write buffer size + auth max buffer size
Expand Down Expand Up @@ -158,7 +161,9 @@ as_async_record_command_create(

cmd->replica_size = pi->replica_size;
cmd->replica_index = replica_index;
cmd->latency_type = latency_type;
rcmd->listener = listener;
as_cluster_add_tran(cluster);
return cmd;
}

Expand Down Expand Up @@ -197,7 +202,9 @@ as_async_value_command_create(
cmd->flags = 0;
cmd->replica_size = pi->replica_size;
cmd->replica_index = 0;
cmd->latency_type = AS_LATENCY_TYPE_WRITE;
vcmd->listener = listener;
as_cluster_add_tran(cluster);
return cmd;
}

Expand Down Expand Up @@ -233,7 +240,9 @@ as_async_info_command_create(
cmd->flags = 0;
cmd->replica_size = 1;
cmd->replica_index = 0;
cmd->latency_type = AS_LATENCY_TYPE_NONE;
icmd->listener = listener;
as_cluster_add_tran(node->cluster);
return cmd;
}

Expand Down
2 changes: 1 addition & 1 deletion src/include/aerospike/as_cdt_ctx.h
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ as_cdt_ctx_add_list_value(as_cdt_ctx* ctx, as_val* val)

/**
* Lookup map by index offset.
* <p>
*
* If the index is negative, the resolved index starts backwards from end of list.
* If an index is out of bounds, a parameter error will be returned. Examples:
* <ul>
Expand Down
Loading
Loading