Skip to content

[celery] Logs parsing rules and tests #20164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

[celery] Logs parsing rules and tests #20164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
dd7ada1
[mongo][sqlserver] Release (#20152)
Kyle-Neale Apr 25, 2025
81be8ff
Celery logs
Kyle-Neale Apr 28, 2025
9a3f1d6
Merge branch 'master' into kyle.neale/celery-logs
Kyle-Neale Apr 28, 2025
ddfcfaf
Update CHANGELOG.md
Kyle-Neale Apr 28, 2025
4a8e520
Fix tests
Kyle-Neale Apr 28, 2025
d40b283
fixes
Kyle-Neale Apr 28, 2025
906d9f9
Merge branch 'master' into kyle.neale/celery-logs
Kyle-Neale Apr 28, 2025
a831e68
fixes
Kyle-Neale Apr 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions celery/assets/logs/celery.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
id: celery
metric_id: celery
backend_only: false
facets:
- groups:
- Celery
name: Task Name
path: celery.task_name
source: log
- groups:
- Celery
name: Task ID
path: celery.task_id
source: log
- groups:
- Celery
name: Worker ID
path: celery.worker_id
source: log
- facetType: range
groups:
- Measure
name: Duration
path: celery.duration
source: log
type: double
unit:
family: time
name: second
- groups:
- Source Code
name: Process Name
path: logger.process_name
source: log
- groups:
- Database
name: Operation Type
path: db.operation
source: log

pipeline:
type: pipeline
name: Celery
enabled: true
filter:
query: source:celery
processors:
- type: grok-parser
name: Parsing Celery Default formats
enabled: true
source: message
samples:
- "[2025-04-23 20:02:25,658: INFO/MainProcess] Task tasks.multiply[c05345e1-cfa5-43cb-9671-30a98115c5db] received"
- "[2025-04-23 20:02:25,660: INFO/ForkPoolWorker-4] Task tasks.add[c25d9c94-4ef6-4828-a1a7-37f6f5a328da] succeeded in 0.00030375000278581865s: 9"
- "celery@17133dbfe7a6 [2025-04-01 13:33:33.400755] heartbeat: clock=1452, freq=5, local_received=1743514413.4011424, pid=1, sw_ident=py-celery, sw_sys=Linux, sw_ver=5.4.0, utcoffset=0"
- "celery@17133dbfe7a6 [2025-10-01 13:33:34.986589] task succeeded: tasks.add(eaae3cff-f198-4fe3-9f69-ad59b0412bed) args=(4, 5) kwargs={} clock=1458, local_received=1743702814.987442, pid=1, result=9, runtime=0.00012291595339775085, utcoffset=0"
grok:
supportRules: |
_date_standard %{date("yyyy-MM-dd HH:mm:ss,SSS"):timestamp}
_date_worker %{date("yyyy-MM-dd HH:mm:ss.SSSSSS"):timestamp}
_level %{word:level}
_task_name %{word:task_name}
_task_id %{notSpace:task_id}
matchRules: |
celery_heartbeat celery@%{notSpace:worker_id} \[%{_date_worker}\] heartbeat: clock=%{number:clock}, freq=%{number:freq}, local_received=%{number:local_received}, pid=%{number:pid}, sw_ident=%{notSpace:sw_ident}, sw_sys=%{notSpace:sw_sys}, sw_ver=%{data:sw_ver}, utcoffset=%{number:utcoffset}
celery_task_status celery@%{notSpace:worker_id} \[%{_date_worker}\] task %{word:task_status}: %{data:task_name}\(%{notSpace:task_id}\) args=%{data:args} kwargs=%{data:kwargs} %{data::keyvalue("=", "", "", " ,")}
celery_standard \[%{_date_standard}:\s+%{_level}/%{data}\]\s*%{data:message}

- type: date-remapper
name: Define `timestamp` as the official date of the log
enabled: true
sources:
- timestamp
- type: status-remapper
name: Define `level` as the official status of the log
enabled: true
sources:
- level
- type: message-remapper
name: Define `message` as the official message of the log
enabled: true
sources:
- message
87 changes: 87 additions & 0 deletions celery/assets/logs/celery_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
id: "celery"
tests:
-
sample: "[2025-10-23 20:02:25,658: INFO/MainProcess] Task tasks.multiply[c05345e1-cfa5-43cb-9671-30a98115c5db] received"
result:
custom:
level: "INFO"
timestamp: 1761249745658
message: "Task tasks.multiply[c05345e1-cfa5-43cb-9671-30a98115c5db] received"
status: "info"
tags:
- "source:LOGS_SOURCE"
timestamp: 1761249745658
-
sample: "[2025-10-23 20:02:25,660: INFO/ForkPoolWorker-4] Task tasks.add[c25d9c94-4ef6-4828-a1a7-37f6f5a328da] succeeded in 0.00030375000278581865s: 9"
result:
custom:
level: "INFO"
timestamp: 1761249745660
message: "Task tasks.add[c25d9c94-4ef6-4828-a1a7-37f6f5a328da] succeeded in 0.00030375000278581865s: 9"
status: "info"
tags:
- "source:LOGS_SOURCE"
timestamp: 1761249745660
-
sample: "celery@17133dbfe7a6 [2025-10-01 13:33:33.400755] heartbeat: clock=1452, freq=5, local_received=1743702813.4011424, pid=1, sw_ident=py-celery, sw_sys=Linux, sw_ver=5.4.0, utcoffset=0"
result:
custom:
clock: 1452.0
freq: 5.0
local_received: 1.7437028134011424E9
pid: 1.0
sw_ident: "py-celery"
sw_sys: "Linux"
sw_ver: "5.4.0"
timestamp: 1759325613400
utcoffset: 0.0
worker_id: "17133dbfe7a6"
message: "celery@17133dbfe7a6 [2025-10-01 13:33:33.400755] heartbeat: clock=1452, freq=5, local_received=1743702813.4011424, pid=1, sw_ident=py-celery, sw_sys=Linux, sw_ver=5.4.0, utcoffset=0"
tags:
- "source:LOGS_SOURCE"
timestamp: 1759325613400
-
sample: "celery@17133dbfe7a6 [2025-10-01 13:33:34.982177] task received: tasks.add(eaae3cff-f198-4fe3-9f69-ad59b0412bed) args=(4, 5) kwargs={} clock=1455, eta=None, expires=None, local_received=1743702814.983349, parent_id=None, pid=1, retries=0, root_id=eaae3cff-f198-4fe3-9f69-ad59b0412bed, utcoffset=0"
result:
custom:
args: "(4, 5)"
clock: 1455
eta: "None"
expires: "None"
kwargs: "{}"
local_received: 1.743702814983349E9
parent_id: "None"
pid: 1
retries: 0
root_id: "eaae3cff-f198-4fe3-9f69-ad59b0412bed"
task_id: "eaae3cff-f198-4fe3-9f69-ad59b0412bed"
task_name: "tasks.add"
task_status: "received"
timestamp: 1759325614982
utcoffset: 0
worker_id: "17133dbfe7a6"
message: "celery@17133dbfe7a6 [2025-10-01 13:33:34.982177] task received: tasks.add(eaae3cff-f198-4fe3-9f69-ad59b0412bed) args=(4, 5) kwargs={} clock=1455, eta=None, expires=None, local_received=1743702814.983349, parent_id=None, pid=1, retries=0, root_id=eaae3cff-f198-4fe3-9f69-ad59b0412bed, utcoffset=0"
tags:
- "source:LOGS_SOURCE"
timestamp: 1759325614982
-
sample: "celery@17133dbfe7a6 [2025-10-01 13:33:34.986589] task succeeded: tasks.add(eaae3cff-f198-4fe3-9f69-ad59b0412bed) args=(4, 5) kwargs={} clock=1458, local_received=1743702814.987442, pid=1, result=9, runtime=0.00012291595339775085, utcoffset=0"
result:
custom:
args: "(4, 5)"
clock: 1458
kwargs: "{}"
local_received: 1.743702814987442E9
pid: 1
result: 9
runtime: 1.2291595339775085E-4
task_id: "eaae3cff-f198-4fe3-9f69-ad59b0412bed"
task_name: "tasks.add"
task_status: "succeeded"
timestamp: 1759325614986
utcoffset: 0
worker_id: "17133dbfe7a6"
message: "celery@17133dbfe7a6 [2025-10-01 13:33:34.986589] task succeeded: tasks.add(eaae3cff-f198-4fe3-9f69-ad59b0412bed) args=(4, 5) kwargs={} clock=1458, local_received=1743702814.987442, pid=1, result=9, runtime=0.00012291595339775085, utcoffset=0"
tags:
- "source:LOGS_SOURCE"
timestamp: 1759325614986
Loading