Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EDU-5581] [EDU-5582] [EDU-5590] feature: add documentation related to the httpBreakdownMetrics dataset #1348

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
dbd8a38
feature: add item in GraphQL features file EN
MarianaAguilera Oct 25, 2024
6bdc826
feature: add table to Metrics Fields file EN
MarianaAguilera Oct 25, 2024
ac918bf
feature: add query guide EN
MarianaAguilera Oct 25, 2024
167b18f
refactor: reorganize order
MarianaAguilera Oct 25, 2024
8fa6524
feature: add link to Guides page and menu
MarianaAguilera Oct 25, 2024
1ee90df
fix: broken url
MarianaAguilera Oct 25, 2024
7694d21
feature: add query guide PT
MarianaAguilera Oct 25, 2024
9ece666
Update httpBreakdownMetrics-dataset.mdx
MarianaAguilera Oct 25, 2024
7dcfc27
fix: adjust code block
MarianaAguilera Oct 25, 2024
b8dc0d9
feature: add link to Guides page and menu
MarianaAguilera Oct 25, 2024
ddc1bee
feature: add in GQL features PT
MarianaAguilera Oct 25, 2024
7ee6df5
feature: add table to Metrics Fields file PT
MarianaAguilera Oct 25, 2024
f48f3d5
fix: add note and adjust time
MarianaAguilera Oct 25, 2024
d68a247
Update src/content/docs/en/pages/guides/graphql/httpBreakdownMetrics-…
MarianaAguilera Oct 25, 2024
fafa419
Merge branch 'main' into edu-5581-httpBreakdownMetrics-dataset-docs
MarianaAguilera Oct 30, 2024
8aaaa27
feature: add item to Release Notes page - EN/PT
MarianaAguilera Oct 30, 2024
25e7f10
Merge branch 'main' into edu-5581-httpBreakdownMetrics-dataset-docs
MarianaAguilera Dec 17, 2024
629ceac
Merge branch 'main' into edu-5581-httpBreakdownMetrics-dataset-docs
MarianaAguilera Jan 22, 2025
cd6d859
Apply suggestions from code review
MarianaAguilera Jan 23, 2025
ee62505
feature: add items to datasets tables EN/PT
MarianaAguilera Jan 23, 2025
71620cd
refactor: delete tip EN/PT
MarianaAguilera Jan 24, 2025
f086df3
Merge branch 'main' into edu-5581-httpBreakdownMetrics-dataset-docs
MarianaAguilera Jan 28, 2025
57b89fe
Merge branch 'main' into edu-5581-httpBreakdownMetrics-dataset-docs
MarianaAguilera Feb 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ Four datasets are deprecated and were replaced by other datsets:
| dataStreamedMetrics | Sent events of data by [Data Stream](/en/documentation/products/observe/data-stream/) to the clients' endpoint |
| edgeFunctionsMetrics | Events executed by [Edge Functions](/en/documentation/products/build/edge-application/edge-functions/) |
| httpMetrics | Request events registered by [Edge Application](/en/documentation/products/build/edge-application/) and [Edge Firewall](/en/documentation/products/secure/edge-firewall/) |
| httpBreakdownMetrics | Query real-time, detailed, aggregated data on [HTTP request events blocked](/en/documentation/products/guides/query-httpBreakdownMetrics-data-with-graphql/) |
| edgeDnsQueriesMetrics | Query events performed on [Edge DNS](/en/documentation/products/secure/edge-dns/) |
| imagesProcessedMetrics | Image processing events by [Image Processor](/en/documentation/products/build/edge-application/image-processor/) |
| tieredCacheMetrics | Request events registered by [Tiered Cache](/en/documentation/products/build/edge-application/tiered-cache/) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,14 @@ The Real-Time Metrics interface on Azion Console also has the `domains` field fo
| sentHttpXOriginalImageSize | “X-Original-Image-Size” header sent in the origin’s response. Used by IMS to inform original image size. This field is the result of a sum. Example: `987390` |
| serverProtocol | Version of the request protocol. Example: `HTTP/1.1`, `HTTP/2.0`, `HTTP/3.0` |
| sourceLocPop | Location and PoP of the edge that received the request. Example: `lax-bso` |
| sslProtocol | Protocol for an established TLS connection. Example: `TLS v1.2` |
| status | HTTP status code of the request. Example: `200` |
| ts | Timestamp of when the event was created. Example: `2022-10-20T10:10:10` |
| upstreamBytesReceived | Number of bytes received by the origin’s edge if the content isn’t cached. This field is the result of a sum. Example: `8304` |
| upstreamCacheStatus | Status of the local edge cache. Can be: `MISS`, `BYPASS`, `EXPIRED`, `STALE`, `UPDATING`, `REVALIDATED`, `HIT`, or `-` |
| upstreamResponseTime | Time it takes for the edge to receive a default response from the origin, in seconds, including headers and body. This field is the result of a sum. Example: `0.876`. **In case of cache, the response is `-`** |
| upstreamStatus | HTTP status code of the origin. If a server can't be selected, the variable keeps the 502 (Bad Gateway) status code. Example: `200`. **In case of cache, the response is -**. |
| wafAttackFamily | Category or type of attack detected by the Web Application Firewall (WAF), based on their characteristics. Example: `$SQL`, `$RFI`, `$SQL`, `$XSS`, `$OTHERS` |
| wafBlock | Informs whether the WAF blocked the action or not. `0` when action wasn’t blocked and `1` when action was blocked. When in *Learning Mode*, it won’t be blocked regardless of the return. |
| wafLearning | Informs if WAF is in Learning mode. Returns `0` if it isn’t and `1` if it’s. |

Expand Down Expand Up @@ -107,6 +109,33 @@ When a field is a result of any kind of calculation, such as a sum, they're cons

---

## httpBreakdownMetrics

| Field | Description |
|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `ts` | Timestamp of when the event was created. Example: `2022-10-20T10:10:10` |
| `host` | Host information sent on the request line. Stores: hostname from the request line, or hostname from the Host request header field, or the server name matching a request. Example: `hello.myhost.net` |
| `remoteAddress` | IP address of the origin that generated the request. Example: `127.0.0.1` |
| `requestPath` | Path of the request made by the end user, without the host, protocol, and arguments. Example: `/v1/application` |
| `geolocAsn` | Autonomous System Number (ASN) Allocation queried from the MaxMind table. Example: `AS52580 Azion Technologies Ltda.` |
| `geolocCountryName`| Remote client's country detected via IP address geolocation. Example: `United States`, `Russian Federation` |
| `httpUserAgent` | End user's application, operating system, vendor, and/or version. Value of the User-Agent header. Example: `Mozilla/5.0 (Windows NT 10.0; Win64; x64)` |
| `httpReferer` | Address of the page the user made the request from. Example: `https://example.com` |
| `sentHttpContentType` | Content-Type header sent in the origin's response. Example: `text/html; charset=UTF-8` |
| `upstreamAddr`| Client's IP address and port. It can also store multiple servers or server groups. Example: `192.168.1.1:80`. When the response is `127.0.0.1:1666`, the upstream is Azion Cells Runtime |

### Calculated fields

When a field results from any kind of calculation, such as a sum, they're considered as *calculated fields*.

| Calculated field | Description |
|----------|----------|
| `blockedRequests` | Total amount of blocked requests in the aggregation being used. This field is the result of a sum. Example: `17` |
| `requests` | Total amount of requests in the aggregation being used. This field is the result of a sum. Example: `11` |
| `wafThreatRequests` | Total amount of requests detected and blocked by the Web Application Firewall (WAF) due to identified security threats. Example: `1523` |

---

## tieredCacheMetrics (Tiered Cache)

| Field | Description |
Expand Down Expand Up @@ -311,7 +340,6 @@ When a field results from any kind of calculation, such as a sum, they're consid
| `uniqRequestUrl` | Number of unique URLs accessed by bots. This field represents the count of distinct URLs requested by bots. Example: `5` |

---

## Deprecated datasets

The following datasets were discontinued. It's recommended to use the [new datasets which replaced them](/en/documentation/devtools/graphql-api/features/#datasets).
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
---
title: How to query data from the httpBreakdownMetrics dataset
description: This guide will explain how to query data from the httpBreakdownMetrics dataset using the GraphiQL playground.
meta_tags: graphql, graphql playground, security metrics, edge applications, requests
namespace: docs_guides_query_httpBreakdownMetrics_graphql
permalink: /documentation/products/guides/query-httpbreakdownmetrics-data-with-graphql/
menu_namespace: graphqlMenu
---

The **httpBreakdownMetrics** dataset provides real-time, detailed, aggregated data on HTTP request events blocked. This dataset is part of the Real-Time Metrics GraphQL.

This data is retained and available for up to *90* days.

This guide will explain how to query data from the httpBreakdownMetrics dataset using the GraphiQL playground.

---

## Querying data

This example queries the top 20 blocked `remoteAddress` entries. To know more about the available fields, check the [Real-Time Metrics GraphQL API Fields](/en/documentation/devtools/graphql-api/features/gql-real-time-metrics-fields/) documentation.

1. Access the GraphiQL playground in this link: `https://manager.azion.com/metrics/graphql`.
- You must be logged in to your Azion account. Otherwise, you'll receive an error message.
2. Send a query following this format:

```graphql
query {
httpBreakdownMetrics(
aggregate: { sum: blockedRequests }
groupBy: [remoteAddress]
orderBy: [sum_DESC],
limit: 20,
filter: {
tsGte: "2024-10-21T11:00:00"
tsLt: "2024-10-21T12:00:00"
}
) {
remoteAddress
totalBlocked: sum
}
}
```

Where:

| Field | Description |
|----------|----------|
| `sum: blockedRequests` | Returns the total number of requests blocked within the specified time range, after applying any filters |
| `groupBy` | Specifies the fields by which the query results should be grouped. Example: `[remoteAddress]` |
| `orderBy` | Specifies the order in which the results should be returned. Examples: `[sum_DESC]`, for descending order, and `[sum_ASC]`, for ascending order |
| `limit` | Specifies the maximum number of results to return. Example: `20` for retrieving the top 20. System maximum: `10,000` |
| `filter` | Defines the criteria used to filter the data returned by the query |
| `tsGte` | A subfield of `filter`. Specifies the start time (greater than or equal to) for the data query, ensuring results include records from this timestamp onward. Format: "YYYY-MM-DDTHH:mm:ss"; example: `"2024-10-21T11:00:00"` |
| `tsLt` | A subfield of `filter`. Specifies the end time (less than) for the data query, filtering out any records with timestamps equal to or after this timestamp. Format: "YYYY-MM-DDTHH:mm:ss"; example: `"2024-10-21T12:00:00"` |

3. You'll receive a response similar to this:

```graphql
{
"data": {
"httpBreakdownMetrics": [
{
"remoteAddress": "192.168.0.1",
"totalBlocked": 6732
},
{
"remoteAddress": "10.0.0.2",
"totalBlocked": 5872
},
{
"remoteAddress": "172.16.0.3",
"totalBlocked": 3958
},
{
"remoteAddress": "192.168.1.4",
"totalBlocked": 3952
},
{
"remoteAddress": "10.0.1.5",
"totalBlocked": 3806
},
{
"remoteAddress": "172.16.1.6",
"totalBlocked": 3730
},
{
"remoteAddress": "192.168.2.7",
"totalBlocked": 3378
},
{
"remoteAddress": "10.0.2.8",
"totalBlocked": 3318
},
{
"remoteAddress": "172.16.2.9",
"totalBlocked": 3284
},
{
"remoteAddress": "192.168.3.10",
"totalBlocked": 3282
},
{
"remoteAddress": "10.0.3.11",
"totalBlocked": 2958
},
{
"remoteAddress": "172.16.3.12",
"totalBlocked": 2884
},
{
"remoteAddress": "192.168.4.13",
"totalBlocked": 2530
},
{
"remoteAddress": "10.0.4.14",
"totalBlocked": 2348
},
{
"remoteAddress": "172.16.4.15",
"totalBlocked": 2004
},
{
"remoteAddress": "192.168.5.16",
"totalBlocked": 1902
},
{
"remoteAddress": "10.0.5.17",
"totalBlocked": 1538
},
{
"remoteAddress": "172.16.5.18",
"totalBlocked": 1440
},
{
"remoteAddress": "192.168.6.19",
"totalBlocked": 1390
},
{
"remoteAddress": "10.0.6.20",
"totalBlocked": 1314
}
]
}
}
```

Where:

| Field | Description |
|----------|----------|
| `remoteAddress` | Specifies the IP address of the source making the request. Example: `10.0.6.20` |
| `totalBlocked` | Refers to the total number of times requests from this IP address have been blocked. This field is the result of a sum. Example: `1314` |
1 change: 1 addition & 0 deletions src/content/docs/en/pages/guides/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ permalink: /documentation/products/guides/
- [How to query the top URLs impacted by bots with GraphQL API](/en/documentation/products/guides/query-bot-manager-breakdown-data-with-graphql/)
- [How to identify the Top IPs generating attack traffic using GraphQL API](/en/documentation/products/guides/query-top-ips-attack-traffic-with-graphql/)
- [How to identify the top attacks using GraphQL API](/en/documentation/products/guides/query-top-attacks-with-graphql/)
- [How to query data from the httpBreakdownMetrics dataset](/en/documentation/products/guides/query-httpbreakdownmetrics-data-with-graphql/)

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,16 @@ For **creating applications with Sanity**, you can choose from the following tem

---

## December 20, 2024

### GraphQL API

The new `httpBreakdownMetrics` dataset is available in the **Real-Time Metrics GraphQL API**. Through this dataset, you can get real-time, detailed, aggregated data on HTTP request events blocked.

Find out more on [Real⁠-⁠Time Metrics GraphQL API Fields](/en/documentation/devtools/graphql-api/features/gql-real-time-metrics-fields/) and learn [How to query data from the httpBreakdownMetrics dataset](/en/documentation/products/guides/query-httpbreakdownmetrics-data-with-graphql/).

---

## December 16, 2024

### Data Stream
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,14 @@ A interface do Real-Time Metrics no Azion Console também possui o campo `domain
| serverProtocol | Versão do protocolo da requisição. Exemplo: `HTTP/1.1`, `HTTP/2.0`, `HTTP/3.0` |
| sentHttpXOriginalImageSize | Cabeçalho `X-Original-Image-Size` enviado na resposta da origem. Informa o tamanho da imagem original em bytes. Este campo é resultado de uma soma. Exemplo: `987390` |
| sourceLocPop | Localização e PoP do edge que recebeu a requisição. Exemplo: `lax-bso` |
| sslProtocol | Protocolo para uma conexão TLS estabelecida. Exemplo: `TLS v1.2` |
| status | Código de status HTTP da requisição. Exemplo: `200` |
| ts | Data e hora de quando o evento foi criado. Exemplo: `2022-10-20T10:10:10` |
| upstreamBytesReceived | Número de bytes recebidos pelo edge da origem, se o conteúdo não estiver em cache. Este campo é resultado de uma soma. Exemplo: `8304` |
| upstreamCacheStatus | Status do cache local do edge. Pode ser: `MISS`, `BYPASS`, `EXPIRED`, `STALE`, `UPDATING`, `REVALIDATED`, `HIT` ou `-` |
| upstreamResponseTime | Tempo para o edge receber uma resposta padrão da origem, em segundos, incluindo cabeçalhos e corpo. Este campo é resultado de uma soma. Exemplo: `0.876`. **No caso de cache, a resposta é -** |
| upstreamStatus | Código de status HTTP da origem. Se um servidor não pode ser selecionado, a variável mantém o código de status 502 (Bad Gateway). Exemplo: `200`. **No caso de cache, a resposta é -**. |
| wafAttackFamily | Categoria ou tipo de ataque detectado pelo Web Application Firewall (WAF), com base em suas características. Exemplo: `$SQL`, `$RFI`, `$SQL`, `$XSS`, `$OTHERS` |
| wafBlock | Informa se o WAF bloqueou ou não a ação. `0` quando não bloqueado e `1` quando bloqueado. Quando em *Learning Mode*, ele não será bloqueado, independentemente do retorno. |
| wafLearning | Informa se o WAF está em Learning Mode. Retorna `0` se está e `1` se não está. |

Expand Down Expand Up @@ -106,6 +108,33 @@ Quando um campo é o resultado de algum tipo de cálculo, como uma soma, ele é

---

## httpBreakdownMetrics

| Campo | Descrição |
|------------------------|---------------------|
| `ts` | Timestamp de quando o evento foi criado. Exemplo: `2022-10-20T10:10:10` |
| `host` | Informações do host enviadas na linha da requisição. Armazena: nome do host da linha da requisição, ou nome do host do campo do cabeçalho da requisição Host, ou o nome do servidor correspondente à requisição. Exemplo: `hello.myhost.net` |
| `remoteAddress` | Endereço IP da origem que gerou a requisição. Exemplo: `127.0.0.1` |
| `requestPath` | Caminho da requisição feita pelo usuário final, sem o host, protocolo e argumentos. Exemplo: `/v1/application` |
| `geolocAsn` | Autonomous System Number (ASN) consultado na tabela MaxMind. Exemplo: `AS52580 Azion Technologies Ltda.` |
| `geolocCountryName`| País do cliente remoto detectado via geolocalização do endereço IP. Exemplo: `United States`, `Russian Federation` |
| `httpUserAgent` | Aplicativo do usuário final, sistema operacional, fornecedor e/ou versão. Valor do cabeçalho User-Agent. Exemplo: `Mozilla/5.0 (Windows NT 10.0; Win64; x64)` |
| `httpReferer` | Endereço da página de onde o usuário fez a requisição. Exemplo: `https://example.com` |
| `sentHttpContentType` | Cabeçalho Content-Type enviado na resposta da origem. Exemplo: `text/html; charset=UTF-8` |
| `upstreamAddr`| Endereço IP e porta do cliente. Também pode armazenar múltiplos servidores ou grupos de servidores. Exemplo: `192.168.1.1:80`. Quando a resposta é `127.0.0.1:1666`, o upstream é Azion Cells Runtime |

### Campos calculados

Quando um campo resulta de qualquer tipo de cálculo, como uma soma, eles são considerados como *campos calculados*.

| Campo calculado | Descrição |
|----------|----------|
| `blockedRequests` | Quantidade total de requisições bloqueadas na agregação utilizada. Este campo é o resultado de uma soma. Exemplo: `17` |
| `requests` | Quantidade total de requisições na agregação utilizada. Este campo é o resultado de uma soma. Exemplo: `11` |
| `wafThreatRequests` | Quantidade total de requisições detectadas e bloqueadas pelo Web Application Firewall (WAF) devido a ameaças de segurança identificadas. Exemplo: `1523` |

---

## tieredCacheMetrics (Tiered Cache)

| Campo | Descrição |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ Quatro conjuntos de dados foram descontinuados e substituiídos por outros:
| dataStreamedMetrics | Registros de envio de dados do [Data Stream](/pt-br/documentacao/produtos/observe/data-stream/) para o endpoint do cliente |
| edgeFunctionsMetrics | Eventos de execução do [Edge Functions](/pt-br/documentacao/produtos/build/edge-application/edge-functions/) |
| httpMetrics | Eventos de requisições registradas pelo [Edge Application](/pt-br/documentacao/produtos/build/edge-application/) e [Edge Firewall](/pt-br/documentacao/produtos/secure/edge-firewall/) |
| httpBreakdownMetrics | Consulte dados agregados, detalhados e em tempo real sobre [eventos de requisições HTTP bloqueadas](/pt-br/documentacao/produtos/guias/consultar-dados-httpbreakdownmetrics-com-graphql/) |
| edgeDnsQueriesMetrics | Eventos de consultas realizadas no [Edge DNS](/pt-br/documentacao/produtos/secure/edge-dns/) |
| imagesProcessedMetrics | Eventos de processamento de imagens do [Image Processor](/pt-br/documentacao/produtos/build/edge-application/image-processor/) |
| tieredCacheMetrics | Eventos de requisições registradas pelo [Tiered Cache](/pt-br/documentacao/produtos/build/edge-application/tiered-cache/) |
Expand Down
Loading