Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Periodically capture resource metrics for every sandbox #216

Merged
Merged
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
749fbf2
make metrics struct more readable and bump env version
0div Dec 5, 2024
882f847
update orchestrator checks to use ticker and check metrics at differe…
0div Dec 5, 2024
c1c4994
add logger methods to log cpu percent and memory MB
0div Dec 5, 2024
203e525
log metrics on start and on end; reduce httpClient timeout to one second
0div Dec 5, 2024
290bbd3
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Dec 6, 2024
419b581
check if envd version is correct on every metrics log attempt
0div Dec 6, 2024
ff936ff
make CPU and MEM logger send event
0div Dec 6, 2024
f49461a
address PR comments
0div Dec 6, 2024
5ccbb3d
Update packages/envd/internal/host/metrics.go
0div Dec 6, 2024
2baee66
Update packages/envd/internal/host/metrics.go
0div Dec 6, 2024
805b77d
fix old method name
0div Dec 6, 2024
3c0343f
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Dec 6, 2024
c705e1c
remove legacu stats logic
0div Dec 6, 2024
9c94a61
specify platform/arch in docker image tf resrource to avoid docker pu…
0div Dec 6, 2024
5f859c6
report more memory data for debugging purposes; attempt to calculte f…
0div Dec 7, 2024
5df1d59
Update packages/shared/pkg/logs/logger.go
0div Dec 9, 2024
d1df2ac
Update packages/shared/pkg/logs/logger.go
0div Dec 9, 2024
1eb4d77
Update packages/shared/pkg/logs/logger.go
0div Dec 9, 2024
d7589b2
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Dec 9, 2024
27e9859
address PR review comments
0div Dec 9, 2024
c1294a1
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Dec 16, 2024
bbd42e6
add metrics endpoint to api spec and generate
0div Dec 17, 2024
9f44dc9
unmarshall log line into metric
0div Dec 18, 2024
4736cdf
fix merge conflicts
0div Jan 17, 2025
562e22c
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Jan 17, 2025
11880e6
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Jan 18, 2025
a3034c1
* add cpuTotal to spec
0div Jan 18, 2025
fc49ae0
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Jan 20, 2025
8545560
update metrics model and fix logger
0div Jan 20, 2025
017eb99
update envd minor version
0div Jan 20, 2025
80be000
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Jan 20, 2025
698f010
fix spec typos and Sandbox properties in checks
0div Jan 21, 2025
f4a3bd8
bump version of envd for version checks of the metrics endpoint
0div Jan 21, 2025
0269598
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Jan 22, 2025
9bd0db4
use "category" label and update log-collector sink to index it
0div Jan 22, 2025
edb28c5
fix naming mismatch and log all cpu and mem in same row
0div Jan 23, 2025
84cf9d6
Fix spacing in envd.yaml
jakubno Jan 23, 2025
044ca36
Fix typo
jakubno Jan 24, 2025
4d544bd
Increase cpu for logs collector
jakubno Jan 24, 2025
66a07a9
lower metrics interval to 2s; update cpuPct field to cpuUsedPct
0div Jan 24, 2025
5468546
Merge branch 'periodically-capture-resource-metrics-for-every-sandbox…
0div Jan 24, 2025
63b33e8
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Jan 24, 2025
e83d1c3
Regenerate API
jakubno Jan 24, 2025
585efe6
Update spec
jakubno Jan 24, 2025
17ef0a5
fix typo cpuUsedPCt
0div Jan 24, 2025
41a5011
Merge branch 'main' of https://github.com/e2b-dev/infra into periodic…
0div Jan 24, 2025
8c1fbbd
remove debug log and escape loki query params
0div Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
lower metrics interval to 2s; update cpuPct field to cpuUsedPct
  • Loading branch information
0div committed Jan 24, 2025
commit 66a07a935ce9801db6dad88eafa5bde820c17349
4 changes: 2 additions & 2 deletions packages/api/internal/handlers/sanbox_metrics.go
Original file line number Diff line number Diff line change
@@ -65,7 +65,7 @@ func (a *APIStore) GetSandboxesSandboxIDMetrics(
for _, entry := range stream.Entries {

var metric struct {
CPUPct float32 `json:"cpuPct"`
CPUUsedPct float32 `json:"cpuUsedPct"`
CPUCount int32 `json:"cpuCount"`
MemTotalMiB int64 `json:"memTotalMiB"`
MemUsedMiB int64 `json:"memUsedMiB"`
@@ -78,7 +78,7 @@ func (a *APIStore) GetSandboxesSandboxIDMetrics(
}
metrics = append(metrics, api.SandboxMetric{
Timestamp: entry.Timestamp,
CpuPct: metric.CPUPct,
CpuUsedPct: metric.CPUUsedPct,
CpuCount: metric.CPUCount,
MemTotalMiB: metric.MemTotalMiB,
MemUsedMiB: metric.MemUsedMiB,
30 changes: 15 additions & 15 deletions packages/envd/internal/host/metrics.go
Original file line number Diff line number Diff line change
@@ -9,11 +9,11 @@ import (
)

type Metrics struct {
Timestamp int64 `json:"ts"` // Unix Timestamp in UTC
CPUCount uint32 `json:"cpu_count"` // Total CPU cores
CPUPercent float32 `json:"cpu_pct"` // Percent rounded to 2 decimal places
MemTotalMiB uint64 `json:"mem_total_mib"` // Total virtual memory in MiB
MemUsedMiB uint64 `json:"mem_used_mib"` // Used virtual memory in MiB
Timestamp int64 `json:"ts"` // Unix Timestamp in UTC
CPUCount uint32 `json:"cpu_count"` // Total CPU cores
CPUUsedPercent float32 `json:"cpu_used_pct"` // Percent rounded to 2 decimal places
MemTotalMiB uint64 `json:"mem_total_mib"` // Total virtual memory in MiB
MemUsedMiB uint64 `json:"mem_used_mib"` // Used virtual memory in MiB
}

func GetMetrics() (*Metrics, error) {
@@ -30,22 +30,22 @@ func GetMetrics() (*Metrics, error) {
return nil, err
}

cpuPcts, err := cpu.Percent(0, false)
cpuUsedPcts, err := cpu.Percent(0, false)
if err != nil {
return nil, err
}

cpuPct := cpuPcts[0]
cpuPctRounded := float32(cpuPct)
if cpuPct > 0 {
cpuPctRounded = float32(math.Round(cpuPct*100) / 100)
cpuUsedPct := cpuUsedPcts[0]
cpuUsedPctRounded := float32(cpuUsedPct)
if cpuUsedPct > 0 {
cpuUsedPctRounded = float32(math.Round(cpuUsedPct*100) / 100)
}

return &Metrics{
Timestamp: time.Now().UTC().Unix(),
CPUCount: uint32(cpuTotal),
CPUPercent: cpuPctRounded,
MemUsedMiB: memUsedMiB,
MemTotalMiB: memTotalMiB,
Timestamp: time.Now().UTC().Unix(),
CPUCount: uint32(cpuTotal),
CPUUsedPercent: cpuUsedPctRounded,
MemUsedMiB: memUsedMiB,
MemTotalMiB: memTotalMiB,
}, nil
}
2 changes: 1 addition & 1 deletion packages/envd/spec/envd.yaml
Original file line number Diff line number Diff line change
@@ -207,7 +207,7 @@ components:
type: object
description: Resource usage metrics
properties:
cpu_pct:
cpu_used_pct:
type: number
format: float
description: CPU usage percentage
2 changes: 1 addition & 1 deletion packages/orchestrator/internal/sandbox/checks.go
Original file line number Diff line number Diff line change
@@ -15,7 +15,7 @@ import (

const (
healthCheckInterval = 10 * time.Second
metricsCheckInterval = 5 * time.Second
metricsCheckInterval = 2 * time.Second
minEnvdVersionForMetrcis = "0.1.5"
)

10 changes: 5 additions & 5 deletions packages/orchestrator/internal/sandbox/metrics.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package sandbox

type SandboxMetrics struct {
Timestamp int64 `json:"ts"` // Unix Timestamp in UTC
CPUCount uint32 `json:"cpu_count"` // Total CPU cores
CPUPercent float32 `json:"cpu_pct"` // Percent rounded to 2 decimal places
MemTotalMiB uint64 `json:"mem_total_mib"` // Total virtual memory in MiB
MemUsedMiB uint64 `json:"mem_used_mib"` // Used virtual memory in MiB
Timestamp int64 `json:"ts"` // Unix Timestamp in UTC
CPUCount uint32 `json:"cpu_count"` // Total CPU cores
CPUUsedPercent float32 `json:"cpu_used_pct"` // Percent rounded to 2 decimal places
MemTotalMiB uint64 `json:"mem_total_mib"` // Total virtual memory in MiB
MemUsedMiB uint64 `json:"mem_used_mib"` // Used virtual memory in MiB
}
4 changes: 2 additions & 2 deletions packages/shared/pkg/logs/logger.go
Original file line number Diff line number Diff line change
@@ -182,13 +182,13 @@ func (l *SandboxLogger) MemoryUsage(memoryMiB float64) {
}
}

func (l *SandboxLogger) Metrics(memTotalMiB, memUsedMiB uint64, cpuCount uint32, cpuPct float32) {
func (l *SandboxLogger) Metrics(memTotalMiB, memUsedMiB uint64, cpuCount uint32, cpuUsedPct float32) {
l.exporter.logger.Info().
Str("category", "metrics").
Str("instanceID", l.instanceID).
Str("envID", l.envID).
Str("teamID", l.teamID).
Float32("cpuPct", cpuPct).
Float32("cpuUsedPct", cpuUsedPct).
Uint32("cpuCount", cpuCount).
Uint64("memTotalMiB", memTotalMiB).
Uint64("memUsedMiB", memUsedMiB).
4 changes: 2 additions & 2 deletions spec/openapi.yml
Original file line number Diff line number Diff line change
@@ -172,7 +172,7 @@ components:
required:
- timestamp
- cpuCount
- cpuPct
- cpuUsedPct
- memUsedMiB
- memTotalMiB
properties:
@@ -184,7 +184,7 @@ components:
type: integer
format: int32
description: Number of CPU cores
cpuPct:
cpuUsedPct:
type: number
format: float
description: CPU usage percentage