New-processors #74
New-processors #74
Conversation
WalkthroughThis pull request involves comprehensive documentation updates for various pipeline processors across the documentation. The changes primarily focus on improving clarity, organization, and readability of the documentation. Key modifications include removing the "Functionality" label, adding "Details" sections, restructuring examples, and standardizing section titles. A new processor (LEEF) was also added, and some configuration files were updated to reflect these changes. Changes
Possibly related PRs
Poem
Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
Docs Feature Deployment https://a38e4a7e.virtualmetric-docs.pages.dev |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (52)
docs/administration/pipelines/processors/circle.mdx (1)
62-64: Consider enhancing the warning message.While the warning is clear, it would be more helpful to briefly explain why circles containing poles cannot be converted. This would help users understand the limitation and potentially avoid issues.
Consider expanding the warning like this:
:::warning[attention] -Circles containing poles cannot be converted. +Circles containing poles (the Earth's north or south poles) cannot be converted due to the geometric limitations of representing polar regions in standard coordinate systems. :::docs/administration/pipelines/processors/pipeline.mdx (1)
38-48: Excellent addition of implementation details and warnings!The new Details section provides crucial information about pipeline references and potential pitfalls. The warnings about runtime resolution and circular dependencies are particularly valuable for preventing issues.
Consider adding a code example demonstrating a circular dependency scenario to help users better understand what patterns to avoid.
🧰 Tools
🪛 LanguageTool
[grammar] ~46-~46: Did you mean “is” or “can be”?
Context: ...` if they are optional. ::: :::warning Be careful with recursive pipeline referen...(BE_IS)
docs/administration/pipelines/processors/fail.mdx (3)
36-61: Consider adding inline comments to YAML examples.The Details section is well-structured and provides valuable context. To further enhance clarity, consider adding inline comments to explain the specific conditions in each YAML example.
Example enhancement:
fail: - message: "Invalid amount {{amount}} for currency {{currency}}" - if: "ctx.amount <= 0" + # Fails if amount is zero or negative
71-73: Add comma for better readability.Add a comma before 'but' to improve sentence structure.
-When `ignore_failure` is set to `true`, errors are still logged but they don't stop the processing. +When `ignore_failure` is set to `true`, errors are still logged, but they don't stop the processing.🧰 Tools
🪛 LanguageTool
[uncategorized] ~72-~72: Use a comma before ‘but’ if it connects two independent clauses (unless they are closely connected and short).
Context: ...s set totrue, errors are still logged but they don't stop the processing. ::: ##...(COMMA_COMPOUND_SENTENCE)
96-96: Enhance example comments with more context.The example comments could be more descriptive to help users better understand:
- The purpose of each example
- When to use each pattern
- Expected outcomes and implications
Example enhancement for the Templates section:
-Use document fields... +Demonstrate how to create dynamic error messages using document fields. This pattern is useful for providing context-rich error messages that help with debugging...Also applies to: 114-114, 134-134, 152-152, 168-168, 185-185, 201-201
docs/administration/pipelines/processors/gsub.mdx (1)
46-54: Great addition of performance considerations!The performance notes about regex caching and the warning about complex patterns are valuable for users.
Consider adding a transition word between the note and warning blocks for better flow:
:::note The processor caches compiled regular expressions for performance. Complex patterns are only compiled once and reused for subsequent matches. ::: + :::warning Be careful with complex regular expressions on large text fields, as they may impact performance. Test patterns thoroughly and consider using simpler patterns when possible. :::🧰 Tools
🪛 LanguageTool
[grammar] ~52-~52: Did you mean “is” or “can be”?
Context: ...for subsequent matches. ::: :::warning Be careful with complex regular expression...(BE_IS)
docs/administration/pipelines/processors/kv.mdx (2)
34-34: Enhance the description with use cases.Consider expanding the description to include common use cases or scenarios where this processor is particularly useful, such as parsing log messages, configuration strings, or query parameters.
-Parses strings containing key-value pairs into structured fields. +Parses strings containing key-value pairs into structured fields. Commonly used for processing log messages, configuration strings, query parameters, and other structured text data.
60-71: Expand the Details section with specific information.The Details section would be more helpful with:
- Examples of supported data types for automatic casting
- Specific examples of malformed inputs and their handling
## Details The processor supports various delimiter patterns, key filtering, value formatting, and automatic type casting. :::note Whitespace around field and value delimiters is automatically trimmed. + +Automatic type casting supports: +- Numbers (integers and floats) +- Booleans ("true"/"false") +- Null values ("null") ::: :::warning Malformed inputs may result in missing entries or incomplete parsing. + +Examples of malformed inputs: +- Missing value delimiter: "key1=value1;key2" +- Empty key: "=value1;key2=value2" +- Multiple value delimiters: "key1==value1" :::docs/administration/pipelines/processors/join.mdx (2)
50-50: Improve readability of common use cases.The common uses would be more readable as a bulleted list.
Consider this format:
-Common uses are converting arrays into human-readable lists, combining path or URL segments with appropriate separators, concatenating multiple tags, creating comma-separated values from arrays. +Common uses include: +- Converting arrays into human-readable lists +- Combining path or URL segments with appropriate separators +- Concatenating multiple tags +- Creating comma-separated values from arrays
152-185: Add context for the error handling behavior.While the example demonstrates error handling well, it would be helpful to explain why the status is set to "field_missing" and when this might be useful in practice.
Consider adding a brief explanation before the example:
+The following example shows how to gracefully handle missing fields and track their occurrence for monitoring or debugging purposes.docs/administration/pipelines/processors/enrich.mdx (2)
33-45: Consider adding example values to the parameters table.The parameters table is well-structured. Consider enhancing it by adding an "Example" column to provide quick reference values, especially for the
typeandlookup_tableparameters.
71-71: Fix grammar and improve word choice.Two minor improvements needed:
- Add a comma: "CSV files must have headers in the first row, as the column names become field names..."
- Replace "very large" with a more specific descriptor like "multi-gigabyte" or provide size thresholds.
Also applies to: 74-74
🧰 Tools
🪛 LanguageTool
[uncategorized] ~71-~71: Possible missing comma found.
Context: ...SV files must have headers in the first row as the column names become field names ...(AI_HYDRA_LEO_MISSING_COMMA)
docs/administration/pipelines/processors/network_direction.mdx (1)
70-70: Fix typo and improve sentence structure.The sentence has a typo ("reporti") and would be more readable if split into separate points.
Consider this revision:
-The processor is useful for analyzing network traffic patterns and flows to identify potential security threats, to monitor and log access to internal resources, to track and reporti on network traffic for compliance requirements, and to validate network segmentation policies. +The processor is useful for: +- Analyzing network traffic patterns and flows to identify potential security threats +- Monitoring and logging access to internal resources +- Tracking and reporting on network traffic for compliance requirements +- Validating network segmentation policiesdocs/administration/pipelines/processors/dot_expander.mdx (2)
40-40: Consider using inline code formatting for better readability.The explanation is clear, but the code-like elements could be more distinguishable.
-This processor is particularly useful when you need to convert flat field names with dots into hierarchical data structures that can be more easily processed by other processors. For example, `foo.bar.baz: 123` becomes `foo: { bar: { baz: 123 } }`. +This processor is particularly useful when you need to convert flat field names with dots into hierarchical data structures that can be more easily processed by other processors. For example: + +`foo.bar.baz: 123` → `foo: { bar: { baz: 123 } }`
46-48: Consider structuring the warning points as a list.The warning contains multiple important points that could be more digestible as a list.
:::warning -The field name must contain at least one dot. The original field is removed after expansion, and all values are preserved as their original types during expansion. If `path` is specified, the expanded structure is placed at that location. +- The field name must contain at least one dot +- The original field is removed after expansion +- All values are preserved as their original types during expansion +- If `path` is specified, the expanded structure is placed at that location :::docs/administration/pipelines/processors/csv.mdx (3)
56-58: Enhance the warning message with specific behavior details.Consider expanding the warning to clarify what exactly happens when
ignore_missingis true. For example:
"If the number of fields in the CSV data is less than the number of target fields:
- When
ignore_missing: false(default): the processor will fail- When
ignore_missing: true: missing fields will be set to empty strings or the specifiedempty_value"
52-54: Add specific alternatives for multi-line CSV processing.Consider enhancing the note with concrete examples of alternative approaches, such as:
"For multi-line CSV processing, consider:
- Using the
splitprocessor to break the input into lines first- Using the
foreachprocessor to iterate over lines- Pre-processing the data with external tools before ingestion"
82-82: Standardize result descriptions across examples.For consistency, consider using the same phrase for all result descriptions. Suggest standardizing to "Result:" for all examples:
- creates new fields: + Result:- parses the values: + Result:Also applies to: 116-116, 150-150
docs/administration/pipelines/processors/dissect.mdx (3)
44-45: Consider adding pattern matching examples.The explanation of pattern matching could be enhanced with inline examples showing the dot notation for nested fields, making it more immediately clear to readers.
- Patterns are matched verbatim. Use %{keyname} to capture values into named fields: nested fields can be created using dot notation in key names. The captured values are stored as strings. + Patterns are matched verbatim. Use %{keyname} to capture values into named fields (e.g., %{user.firstname} creates nested fields). The captured values are stored as strings. For example: + - Pattern: "%{user.firstname} %{user.lastname}" + - Input: "John Doe" + - Result: { "user": { "firstname": "John", "lastname": "Doe" } }
171-202: Consider consolidating error handling examples.The "Error Handling" section appears redundant with "Non-String Fields" as both demonstrate error scenarios. Consider consolidating these sections into a single "Error Handling" section with subsections for different types of errors.
66-66: Enhance the basic extraction example comment.The current comment could be more informative about the pattern structure being demonstrated.
- Extracting first name, last name, and age... + Demonstrating basic space-separated field extraction with multiple capture groups...docs/administration/pipelines/processors/geoip.mdx (1)
89-93: Consider adding database version compatibility information.While the database types are well-documented, it would be helpful to specify which versions of MaxMind databases are supported. This information could help users avoid compatibility issues.
docs/administration/pipelines/processors/decrypt.mdx (5)
54-58: Consider enhancing the warning message about unsupported algorithms.The warning about unsupported algorithms could be more specific to help users avoid errors.
Consider updating the warning to explicitly list the supported algorithms:
:::warning The encryption key must be exactly 32 bytes long for the AES-256 encryption, and both the encrypted value and IV must be base64 encoded strings. -Unsupported algorithms will result in error. +Only AES-256-GCM and AES-256-CFB algorithms are supported. Using any other algorithm will result in an error. :::
70-70: Consider adding a note about base64 encoding in the example.While the example shows base64 encoded values, it would be helpful to explicitly mention this requirement in the comment.
- When decrypting in the CFB mode... + When decrypting in the CFB mode (note: encrypted_value and iv must be base64 encoded)...
104-104: Enhance the GCM example comments for clarity.The example could better explain the combined use of GCM mode and compression.
- Using the GCM mode with compression... + Using the GCM mode with compression (provides both encryption and data compression)... - decompresses the values after decryption: + first decrypts using GCM mode, then decompresses the result:Also applies to: 123-123
139-139: Consider expanding error handling examples.While the current example shows key length errors, it would be helpful to document other common error scenarios.
Consider adding examples for:
- Invalid base64 encoding errors
- Authentication failures in GCM mode
- Invalid IV length errors
Example structure:
### Common Error Scenarios #### Invalid Base64 Encoding <example showing base64 error> #### GCM Authentication Failure <example showing auth failure> #### Invalid IV Length <example showing IV length error>Also applies to: 158-158
Line range hint
27-158: Consider adding a Troubleshooting section.The documentation would benefit from a dedicated troubleshooting section to help users diagnose and resolve common issues.
Consider adding a section like:
## Troubleshooting ### Common Issues 1. **Key Length Issues** - Symptom: "encryption key must be 32 bytes" error - Solution: Ensure key is exactly 32 bytes 2. **Base64 Encoding Problems** - Symptom: "invalid base64" error - Solution: Verify both encrypted value and IV are properly base64 encoded 3. **GCM Authentication Failures** - Symptom: "authentication failed" error - Solution: Verify the encrypted data hasn't been tampered withdocs/administration/pipelines/processors/drop.mdx (1)
34-42: Consider adding a code example in the warning.The new "Details" section effectively explains common use cases and includes an important warning. Consider enhancing the warning with a code example demonstrating a complex condition that could accidentally drop important documents.
:::warning Once dropped, a document cannot be recovered by the subsequent processors. Use caution with complex conditions to avoid accidentally dropping important documents. + +For example: +```yaml +drop: + - if: > + ctx.status == 'error' && + ctx.retries > 3 && + !ctx.critical # Be careful! This could drop important errors +``` :::docs/administration/pipelines/processors/append.mdx (1)
40-46: Consider rephrasing to avoid repetitive sentence beginnings.The "Details" section effectively explains the processor's behavior, but three consecutive sentences begin with "If". Consider rephrasing for better readability.
-If the field exists and is an array, the operation is carried out. If the field exists and is a scalar, the field is converted to an array before the operation. If the field doesn't exist, an array is created containing the provided values. +When the field exists and is an array, the operation is carried out. For scalar fields, they are converted to arrays before the operation. In cases where the field doesn't exist, an array is created containing the provided values.🧰 Tools
🪛 LanguageTool
[style] ~42-~42: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...erted to an array before the operation. If the field doesn't exist, an array is cr...(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
docs/administration/pipelines/processors/bytes.mdx (1)
40-44: Consider hyphenating "case-insensitive".The "Details" section effectively explains the processor's behavior. However, "case insensitive" should be hyphenated as "case-insensitive" according to standard English conventions.
-Allowed units are `B`, `KB`, `MB`, `GB`, `TB`, and `PB`, all case insensitive. +Allowed units are `B`, `KB`, `MB`, `GB`, `TB`, and `PB`, all case-insensitive.🧰 Tools
🪛 LanguageTool
[misspelling] ~44-~44: This word is normally spelled with a hyphen.
Context: ...,KB,MB,GB,TB, andPB, all case insensitive. Fractional values like "1.5KB" are a...(EN_COMPOUNDS_CASE_INSENSITIVE)
docs/administration/pipelines/processors/lowercase.mdx (1)
177-208: Comprehensive error handling example.The example effectively shows how to handle non-string values and capture errors, which is crucial for production environments. Consider adding a note about potential performance impact when using
on_failureprocessors extensively.docs/administration/pipelines/processors/registered_domain.mdx (1)
40-52: Excellent addition of component details and performance considerations.The Details section effectively:
- Lists all extractable components
- Provides important performance considerations for large-scale processing
- Helps users understand the processor's capabilities
Consider adding a link to the Mozilla Public Suffix List documentation for users who want to learn more about the underlying implementation.
docs/administration/pipelines/processors/redact.mdx (1)
48-59: Comprehensive Details section with important implementation notes.The new section effectively covers:
- PII protection context
- Case sensitivity considerations
- Performance implications
Consider adding examples of case-insensitive pattern modifications to help users implement this common requirement.
docs/administration/pipelines/processors/leef.mdx (2)
24-30: Well-structured introduction to LEEF processing.The description and note effectively explain:
- Core functionality
- Field extraction capabilities
- Delimiter handling
Consider adding a reference to the official LEEF format specification for users who need detailed format information.
52-88: Comprehensive examples covering key use cases.The examples effectively demonstrate:
- Basic LEEF parsing
- Custom field targeting
- Error handling
Consider adding an example showing how to handle multiple LEEF messages in a single field, as this is a common use case in log processing.
Also applies to: 90-127, 129-162
docs/administration/pipelines/processors/html_strip.mdx (1)
Line range hint
56-74: Consider adding example outputs for HTML with nested elements.While the current examples cover basic scenarios well, consider adding examples that demonstrate how the processor handles:
- HTML with deeply nested elements
- HTML with mixed inline and block elements
- HTML with special characters and entities
Also applies to: 85-114, 117-146, 148-177, 180-211
docs/administration/pipelines/processors/normalize.mdx (1)
67-69: Consider adding more details about field mapping limitations.The warning about non-reversible field mapping could be enhanced with specific examples of fields that don't have equivalents across formats.
docs/administration/pipelines/processors/convert.mdx (1)
76-78: Consider adding examples for auto-type edge cases.The note about auto-type conversion order is helpful, but consider adding examples of edge cases where the type detection might produce unexpected results.
docs/administration/pipelines/processors/encrypt.mdx (1)
58-60: Consider adding security best practices section.While the warning about key length is helpful, consider adding a section about:
- Key management best practices
- IV generation recommendations
- Secure storage of encrypted data
docs/administration/pipelines/processors/json.mdx (1)
Line range hint
133-171: Consider expanding merge example scenarios.While the current merge example is good, consider adding examples for:
- Merging arrays with duplicate values
- Deep merging of nested objects
- Handling of null values during merge
docs/administration/pipelines/processors/move.mdx (1)
58-62: Enhance the warning message about field conflicts.The warning about field conflicts could be more specific about the potential consequences.
- Also, when moving nested structures, field conflicts can occur. Consider using `exclude` patterns to protect critical fields. + Also, when moving nested structures, field conflicts can occur if source and target fields share the same names. Use `exclude` patterns to prevent overwriting critical fields.docs/administration/pipelines/processors/grok.mdx (1)
62-64: Consider adding performance monitoring guidance.The warning about performance impact could be enhanced with specific monitoring recommendations.
- Complex patterns may impact performance. Monitor matching time, and consider optimizing patterns for frequently processed fields. + Complex patterns may impact performance. Monitor pattern matching time using the `trace_match` parameter, and consider optimizing patterns for frequently processed fields by: + - Using simpler patterns where possible + - Avoiding excessive use of wildcards + - Breaking down complex patterns into multiple simpler onesdocs/administration/pipelines/processors/geo_grid.mdx (1)
65-67: Consider adding alternative approaches for polar regions.The warning about polar regions could be enhanced with alternative solutions.
- Geohash cells that contain poles _cannot_ be converted to polygons due to coordinate system limitations. + Geohash cells that contain poles _cannot_ be converted to polygons due to coordinate system limitations. For polar regions, consider: + - Using lower precision geohash cells that don't intersect poles + - Switching to geotile format which handles polar regions better + - Implementing custom handling for pole-intersecting cellsdocs/administration/pipelines/processors/foreach.mdx (2)
44-44: Fix grammatical issues in the procedure description.Address missing punctuation and improve clarity.
- * it first locates the specified field in the document, and determines if the field contains an array or a map + * It first locates the specified field in the document and determines if the field contains an array or a map.🧰 Tools
🪛 LanguageTool
[uncategorized] ~44-~44: A period might be missing here.
Context: ...nes if the field contains an array or a map * if an array of a map exists, it goes...(AI_EN_LECTOR_MISSING_PUNCTUATION_PERIOD)
50-52: Fix grammar in the warning message.Improve the warning message structure.
- Be careful when modifying the `_ingest` fields directly as they are used internally by the processor. + Be careful when modifying the `_ingest` fields directly, as they are used internally by the processor.🧰 Tools
🪛 LanguageTool
[grammar] ~50-~50: Did you mean “is” or “can be”?
Context: ... processors as appropriate. :::warning Be careful when modifying the_ingestfi...(BE_IS)
[uncategorized] ~51-~51: Possible missing comma found.
Context: ...ful when modifying the_ingestfields directly as they are used internally by the proc...(AI_HYDRA_LEO_MISSING_COMMA)
docs/administration/pipelines/processors/cef.mdx (3)
46-48: Consider moving the warning about CEF message format requirements to the top.The warning about CEF message format requirements is crucial for users. Consider moving it from line 69 to the beginning of the Details section for better visibility.
46-48: Enhance the type conversion note with specific examples.The note about automatic type conversion could be more helpful with specific examples of field types that are automatically converted.
Add examples like:
:::note -Type conversion is handled automatically for numeric values in extension fields, e.g. port numbers, counts, etc. +Type conversion is handled automatically for extension fields: +- Numeric fields (e.g., port numbers, counts): Converted to integers +- Boolean fields (e.g., success flags): Converted to boolean +- IP addresses: Preserved as strings +- Timestamps: Converted to ISO8601 format :::
164-197: Enhance the error handling example with more scenarios.The error handling example could be more comprehensive by showing different types of failures and their handling.
Add examples for:
- Missing header fields
- Invalid field values
- Malformed extension fields
docs/administration/pipelines/processors/dot_nester.mdx (2)
27-27: Align description with the processor name.The description uses "flattens" terminology while the processor is named "dot_nester". Consider revising for consistency.
-Flattens nested objects into a single-level object where keys are paths representing the original nested structure. +Nests object keys using dot notation, transforming nested structures into a single-level object with path-based keys.
56-63: Standardize format examples.The format examples use inconsistent field names. Consider using the same example fields across all formats for better comparison.
-**Default** (no format) - Preserves original field name casing, e.g. `User.FirstName.LastName` -**ASIM** - Preserves original field name casing, e.g. `User.FirstName.LastName` -**ECS** - Converts all field names to lowercase, e.g. `user.firstname.lastname` -**CIM** - Converts all field names to lowercase, e.g. `user.firstname.lastname` +**Default** (no format) - Preserves original field name casing, e.g. `User.Address.City` +**ASIM** - Preserves original field name casing, e.g. `User.Address.City` +**ECS** - Converts all field names to lowercase, e.g. `user.address.city` +**CIM** - Converts all field names to lowercase, e.g. `user.address.city`docs/administration/pipelines/processors/date.mdx (1)
69-70: Enhance timezone format examples.The timezone formats could be more helpful with specific examples.
- <TermCol>**Timezones**</TermCol> - <DefCol>`Z`, `ZZ`, `ZZZ`</DefCol> + <TermCol>**Timezones**</TermCol> + <DefCol> + - `Z`: UTC (e.g., "Z") + - `ZZ`: Offset (e.g., "+0100") + - `ZZZ`: Region (e.g., "Europe/London") + </DefCol>docs/administration/pipelines/processors/attachment.mdx (1)
58-65: Enhance supported formats list with version information.The list of supported formats could be more helpful with version information.
-The processor supports the following libraries to extract these fields: - -- Microsoft Excel (XLSX) -- Microsoft Word (DOCX) -- PDF documents -- Rich Text Format (RTF) -- OpenDocument Text (ODT) -- Plain text (TXT) +The processor supports the following formats: + +- Microsoft Excel (XLSX) - Office 2007 and later +- Microsoft Word (DOCX) - Office 2007 and later +- PDF documents - Version 1.7 and earlier +- Rich Text Format (RTF) - Version 1.9.1 +- OpenDocument Text (ODT) - Version 1.2 +- Plain text (TXT) - UTF-8 encoded
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (55)
docs/administration/pipelines/processors/append.mdx(2 hunks)docs/administration/pipelines/processors/attachment.mdx(3 hunks)docs/administration/pipelines/processors/bytes.mdx(2 hunks)docs/administration/pipelines/processors/cef.mdx(1 hunks)docs/administration/pipelines/processors/circle.mdx(2 hunks)docs/administration/pipelines/processors/community_id.mdx(2 hunks)docs/administration/pipelines/processors/compact.mdx(6 hunks)docs/administration/pipelines/processors/contains.mdx(5 hunks)docs/administration/pipelines/processors/convert.mdx(6 hunks)docs/administration/pipelines/processors/csv.mdx(7 hunks)docs/administration/pipelines/processors/date.mdx(11 hunks)docs/administration/pipelines/processors/date_index_name.mdx(8 hunks)docs/administration/pipelines/processors/decrypt.mdx(6 hunks)docs/administration/pipelines/processors/dissect.mdx(5 hunks)docs/administration/pipelines/processors/dot_expander.mdx(8 hunks)docs/administration/pipelines/processors/dot_nester.mdx(11 hunks)docs/administration/pipelines/processors/drop.mdx(9 hunks)docs/administration/pipelines/processors/encrypt.mdx(8 hunks)docs/administration/pipelines/processors/enrich.mdx(1 hunks)docs/administration/pipelines/processors/fail.mdx(9 hunks)docs/administration/pipelines/processors/final.mdx(6 hunks)docs/administration/pipelines/processors/fingerprint.mdx(9 hunks)docs/administration/pipelines/processors/foreach.mdx(5 hunks)docs/administration/pipelines/processors/geo_grid.mdx(9 hunks)docs/administration/pipelines/processors/geoip.mdx(6 hunks)docs/administration/pipelines/processors/grok.mdx(9 hunks)docs/administration/pipelines/processors/gsub.mdx(10 hunks)docs/administration/pipelines/processors/html_strip.mdx(4 hunks)docs/administration/pipelines/processors/join.mdx(6 hunks)docs/administration/pipelines/processors/json.mdx(10 hunks)docs/administration/pipelines/processors/kv.mdx(10 hunks)docs/administration/pipelines/processors/leef.mdx(1 hunks)docs/administration/pipelines/processors/lowercase.mdx(7 hunks)docs/administration/pipelines/processors/move.mdx(11 hunks)docs/administration/pipelines/processors/network_direction.mdx(7 hunks)docs/administration/pipelines/processors/normalize.mdx(9 hunks)docs/administration/pipelines/processors/pipeline.mdx(6 hunks)docs/administration/pipelines/processors/redact.mdx(6 hunks)docs/administration/pipelines/processors/registered_domain.mdx(5 hunks)docs/administration/pipelines/processors/remove.mdx(1 hunks)docs/administration/pipelines/processors/rename.mdx(1 hunks)docs/administration/pipelines/processors/reroute.mdx(1 hunks)docs/administration/pipelines/processors/script.mdx(1 hunks)docs/administration/pipelines/processors/set.mdx(1 hunks)docs/administration/pipelines/processors/sort.mdx(1 hunks)docs/administration/pipelines/processors/split.mdx(1 hunks)docs/administration/pipelines/processors/syslog.mdx(1 hunks)docs/administration/pipelines/processors/trim.mdx(1 hunks)docs/administration/pipelines/processors/uppercase.mdx(1 hunks)docs/administration/pipelines/processors/uri_parts.mdx(1 hunks)docs/administration/pipelines/processors/urldecode.mdx(1 hunks)docs/administration/pipelines/processors/user_agent.mdx(1 hunks)docusaurus.config.ts(1 hunks)sidebars.ts(2 hunks)src/components/CustomFeatures/TermTable/styles.module.css(1 hunks)
✅ Files skipped from review due to trivial changes (14)
- docs/administration/pipelines/processors/split.mdx
- docs/administration/pipelines/processors/uppercase.mdx
- docs/administration/pipelines/processors/syslog.mdx
- docs/administration/pipelines/processors/user_agent.mdx
- docs/administration/pipelines/processors/sort.mdx
- docs/administration/pipelines/processors/rename.mdx
- docs/administration/pipelines/processors/uri_parts.mdx
- docs/administration/pipelines/processors/remove.mdx
- docs/administration/pipelines/processors/community_id.mdx
- docs/administration/pipelines/processors/urldecode.mdx
- docs/administration/pipelines/processors/script.mdx
- docs/administration/pipelines/processors/set.mdx
- docs/administration/pipelines/processors/trim.mdx
- docs/administration/pipelines/processors/reroute.mdx
🧰 Additional context used
🪛 LanguageTool
docs/administration/pipelines/processors/append.mdx
[style] ~42-~42: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...erted to an array before the operation. If the field doesn't exist, an array is cr...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
docs/administration/pipelines/processors/gsub.mdx
[grammar] ~52-~52: Did you mean “is” or “can be”?
Context: ...for subsequent matches. ::: :::warning Be careful with complex regular expression...
(BE_IS)
docs/administration/pipelines/processors/bytes.mdx
[misspelling] ~44-~44: This word is normally spelled with a hyphen.
Context: ..., KB, MB, GB, TB, and PB, all case insensitive. Fractional values like "1.5KB" are a...
(EN_COMPOUNDS_CASE_INSENSITIVE)
docs/administration/pipelines/processors/pipeline.mdx
[grammar] ~46-~46: Did you mean “is” or “can be”?
Context: ...` if they are optional. ::: :::warning Be careful with recursive pipeline referen...
(BE_IS)
docs/administration/pipelines/processors/foreach.mdx
[uncategorized] ~44-~44: A period might be missing here.
Context: ...nes if the field contains an array or a map * if an array of a map exists, it goes...
(AI_EN_LECTOR_MISSING_PUNCTUATION_PERIOD)
[grammar] ~50-~50: Did you mean “is” or “can be”?
Context: ... processors as appropriate. :::warning Be careful when modifying the _ingest fi...
(BE_IS)
[uncategorized] ~51-~51: Possible missing comma found.
Context: ...ful when modifying the _ingest fields directly as they are used internally by the proc...
(AI_HYDRA_LEO_MISSING_COMMA)
docs/administration/pipelines/processors/fail.mdx
[uncategorized] ~72-~72: Use a comma before ‘but’ if it connects two independent clauses (unless they are closely connected and short).
Context: ...s set to true, errors are still logged but they don't stop the processing. ::: ##...
(COMMA_COMPOUND_SENTENCE)
docs/administration/pipelines/processors/enrich.mdx
[uncategorized] ~71-~71: Possible missing comma found.
Context: ...SV files must have headers in the first row as the column names become field names ...
(AI_HYDRA_LEO_MISSING_COMMA)
[style] ~74-~74: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...rectly into memory, the impact of using very large lookup tables must be carefully conside...
(EN_WEAK_ADJECTIVE)
docs/administration/pipelines/processors/attachment.mdx
[uncategorized] ~68-~68: A punctuation mark might be missing here.
Context: ...:note For multiple attachments, use the foreach processor. ::: #...
(AI_EN_LECTOR_MISSING_PUNCTUATION)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: Deploy VirtualMetric Docs
🔇 Additional comments (73)
docs/administration/pipelines/processors/circle.mdx (2)
26-26: LGTM! Clear and concise description.The simplified description effectively communicates the processor's purpose without redundant text.
48-56: LGTM! Well-organized units list.The units are logically organized with clear indication of the default unit. The bullet-point format improves readability.
docs/administration/pipelines/processors/pipeline.mdx (3)
23-23: LGTM! Clear and concise description.The streamlined description effectively communicates the processor's purpose while maintaining clarity.
Line range hint
52-107: Well-structured examples with improved readability!The reorganized examples with clearer titles and comments make the documentation more accessible. The progression from basic to conditional usage helps users understand the processor's capabilities incrementally.
46-48: LGTM! Clear and appropriate warning message.The warning message effectively communicates the potential risks of recursive pipeline references.
🧰 Tools
🪛 LanguageTool
[grammar] ~46-~46: Did you mean “is” or “can be”?
Context: ...` if they are optional. ::: :::warning Be careful with recursive pipeline referen...(BE_IS)
docs/administration/pipelines/processors/fail.mdx (2)
22-22: LGTM! Clear and concise description.The description effectively communicates the processor's purpose and functionality.
63-66: LGTM! Important warning about template fields.The warning effectively highlights a critical requirement for template fields.
docs/administration/pipelines/processors/gsub.mdx (2)
26-26: LGTM! Clear and concise description.The description effectively communicates the processor's functionality and use cases.
62-62: Excellent improvements to example descriptions!The modified descriptions are more action-oriented and clearer. The progression from basic to advanced examples is logical and well-structured.
Also applies to: 93-93, 124-124, 141-141, 157-157, 175-175, 191-191, 212-212
docs/administration/pipelines/processors/kv.mdx (1)
Line range hint
74-247: Well-structured and comprehensive examples!The examples section is excellent, with:
- Clear progression from basic to advanced usage
- Consistent structure across examples
- Good coverage of key features
- Clear input/output demonstrations
docs/administration/pipelines/processors/join.mdx (4)
25-25: LGTM! Clear and concise description.The description effectively communicates the processor's purpose in a single, easy-to-understand sentence.
46-48: LGTM! Important warning about empty arrays.The warning is well-placed and clearly communicates the behavior with empty arrays.
Line range hint
56-85: LGTM! Well-structured basic example.The example effectively demonstrates the basic usage with clear input, configuration, and output.
Line range hint
88-117: LGTM! Clear demonstration of empty array handling.The example effectively shows how empty arrays are processed, which complements the earlier warning.
docs/administration/pipelines/processors/enrich.mdx (3)
2-22: LGTM! Well-structured synopsis with clear parameter organization.The front matter and synopsis effectively present the processor's purpose and configuration options.
54-56: LGTM! Clear and prominent warning messages.The warning boxes effectively highlight important limitations regarding KQL conversion and memory usage considerations.
Also applies to: 73-75
79-231: LGTM! Excellent examples with clear progression.The examples section is comprehensive and well-structured:
- Covers key use cases from basic to advanced
- Shows clear input/output examples
- Includes helpful explanatory comments
- Demonstrates proper query formatting
docs/administration/pipelines/processors/network_direction.mdx (4)
27-27: LGTM! Clear and concise description.The description effectively communicates the processor's purpose in a single, well-structured sentence.
56-68: LGTM! Well-structured traffic classification.The term table clearly defines the four traffic categories with good formatting and consistent structure.
Line range hint
78-247: LGTM! Comprehensive and well-structured examples.The examples effectively cover all traffic direction scenarios (inbound, outbound, internal, external, and dynamic configuration) with consistent formatting and clear input/output demonstrations.
Line range hint
1-247: LGTM! Well-organized documentation structure.The documentation follows a logical flow with clear sections, consistent formatting, and appropriate use of admonitions for important notes and warnings.
docs/administration/pipelines/processors/dot_expander.mdx (3)
23-23: Well-structured description!The description is concise and effectively communicates the processor's purpose without redundancy.
42-44: Clear and helpful note!The note effectively communicates when to use the
ignore_failureflag.
Line range hint
52-185: Excellent example structure and progression!The examples are well-organized with:
- Clear progression from basic to advanced usage
- Consistent structure across all examples
- Helpful comments explaining each scenario
docs/administration/pipelines/processors/csv.mdx (2)
28-28: Well-structured description!The description effectively communicates the processor's purpose and key features in a clear, concise manner.
Line range hint
28-150: Excellent documentation structure and content!The documentation is well-organized, technically accurate, and provides clear examples. The addition of the Details section with important notes and warnings is particularly valuable for users.
docs/administration/pipelines/processors/dissect.mdx (2)
25-25: Well-written description!The description is clear, concise, and effectively communicates both the purpose and mechanism of the processor.
42-50: Clear technical explanations and important distinctions!The section effectively explains pattern matching behavior and clearly distinguishes it from regular expressions. The note about exact matching is particularly helpful for users coming from a regex background.
docs/administration/pipelines/processors/geoip.mdx (7)
27-29: Well-structured description!The split into two paragraphs improves readability while maintaining a logical flow from core functionality to detailed capabilities.
48-52: Good section organization!The "Details" section title and introduction align well with the documentation standards and provide a clear context for the following property tables.
54-75: Excellent property documentation!The term table format provides clear and structured documentation for each City/Country database property. The definitions are precise and helpful.
79-85: Well-structured ASN properties!The ASN properties table maintains consistency with the City/Country format while clearly documenting the network-specific fields.
87-94: Clear database type differentiation!The description of each database type effectively communicates its capabilities and limitations, helping users make informed choices.
109-109: Consistent example descriptions!The use of gerund form ("Looking up", "Extracting") in example descriptions maintains a consistent and professional tone throughout the documentation.
Also applies to: 145-145, 184-184
234-234: Improved error handling clarity!The error handling section's grammar improvements make the documentation more professional while maintaining technical accuracy.
Also applies to: 253-253
docs/administration/pipelines/processors/decrypt.mdx (2)
27-27: LGTM! Clear and concise description.The description effectively communicates the processor's purpose and supported encryption modes.
50-52: LGTM! Well-structured explanation of encryption modes.The bullet points clearly explain the tradeoffs between GCM and CFB modes, helping users make an informed choice.
docusaurus.config.ts (1)
85-85: Consider the impact on navigation usability.Setting
autoCollapseCategoriestofalsewill keep all categories expanded. While this maintains context, it might lead to excessive scrolling in the sidebar, especially with the addition of new processor documentation.Let's check the depth of the navigation structure:
✅ Verification successful
Navigation structure supports expanded categories well
The sidebar has a clean 3-level hierarchy with well-organized sections. Even with all categories expanded, the structure remains navigable due to:
- Logical grouping of related items
- Moderate nesting depth (max 3 levels)
- Focused categories with reasonable item counts
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Count the levels of nesting in the sidebar configuration rg -A 5 "type: \"category\"" sidebars.ts | grep -c "type: \"category\""Length of output: 71
Script:
#!/bin/bash # Examine the sidebar structure with context rg -A 10 'type: "category"' sidebars.tsLength of output: 2990
sidebars.ts (2)
72-72: Verify documentation files for new processors.New processors have been added to the sidebar. Let's ensure their documentation files exist and are properly linked.
Also applies to: 80-80, 84-84, 100-100
✅ Verification successful
Documentation files verified for all new processors
All new processor documentation files are present in the correct location:
- cef.mdx
- date_index_name.mdx
- dot_nester.mdx
- leef.mdx
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Check if documentation files exist for new processors for doc in "cef" "date_index_name" "dot_nester" "leef"; do if ! fd -t f "${doc}.mdx$" docs/administration/pipelines/processors/; then echo "Missing documentation for ${doc} processor" fi doneLength of output: 715
Line range hint
72-100: Check for broken links after processor removal.The "flatten" processor has been removed and replaced with "dot_nester". Let's verify there are no remaining references to the removed processor.
✅ Verification successful
No broken references found after processor replacement
All mentions of "flatten" in the codebase are either generic descriptions of data transformation or part of the new dot_nester processor documentation. No broken links or outdated references to the removed processor were found.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Search for any remaining references to the flatten processor rg -l "flatten" docs/administration/pipelines/processors/Length of output: 163
Script:
#!/bin/bash # Check context of flatten references in both files echo "=== Content from move.mdx ===" rg -C 3 "flatten" docs/administration/pipelines/processors/move.mdx echo -e "\n=== Content from dot_nester.mdx ===" rg -C 3 "flatten" docs/administration/pipelines/processors/dot_nester.mdx # Search for any other flatten references in docs echo -e "\n=== Other references in docs ===" rg -l "flatten" docs/ | grep -v "processors/move.mdx\|processors/dot_nester.mdx"Length of output: 2981
Script:
#!/bin/bash # Check context of flatten reference in post-processing.mdx rg -C 3 "flatten" docs/administration/pipelines/post-processing.mdxLength of output: 372
src/components/CustomFeatures/TermTable/styles.module.css (1)
3-6: LGTM! Improved table styling and readability.The changes enhance the table styling by:
- Adding
tbodyto selectors for better specificity- Increasing line height from 50% to 75% for improved readability
- Maintaining consistency between dark and light themes
Also applies to: 11-14
docs/administration/pipelines/processors/contains.mdx (2)
25-25: LGTM! Improved documentation clarity and structure.The documentation improvements enhance readability by:
- Simplifying the description
- Adding a prominent warning about required parameters in the Details section
Also applies to: 42-46
50-50: LGTM! Better organized examples.Example sections are now more consistently named and better organized, making it easier to find specific use cases.
Also applies to: 73-73, 97-97, 121-121
docs/administration/pipelines/processors/drop.mdx (2)
21-21: LGTM! Clear and concise description.The simplified description effectively communicates the processor's purpose.
50-50: LGTM! Improved example descriptions.The example descriptions have been updated to be more direct and clearer, making the documentation more user-friendly.
Also applies to: 66-66, 79-79, 97-97, 110-110, 134-134, 147-147, 162-162
docs/administration/pipelines/processors/compact.mdx (3)
22-22: LGTM! Clear and concise description.The simplified description effectively communicates the processor's purpose.
36-42: LGTM! Comprehensive "Details" section.The new "Details" section effectively explains empty fields and includes a helpful note about the processor's utility.
131-131: LGTM! Improved example descriptions.The example descriptions have been updated to be more direct and clearer, making the documentation more user-friendly.
Also applies to: 148-148, 164-164, 183-183
docs/administration/pipelines/processors/append.mdx (1)
24-24: LGTM! Clear and concise description.The simplified description effectively communicates the processor's purpose.
docs/administration/pipelines/processors/bytes.mdx (1)
24-24: LGTM! Clear and concise description.The simplified description effectively communicates the processor's purpose.
docs/administration/pipelines/processors/final.mdx (3)
21-23: LGTM! Clear and comprehensive description.The updated description effectively communicates both the processor's purpose and its common use cases.
36-78: LGTM! Well-structured "Details" section with practical examples.The new "Details" section effectively explains common use cases with clear examples and includes an important warning about multi-pipeline setups.
Line range hint
82-162: LGTM! Clear and practical examples.The examples effectively demonstrate various use cases of the processor with clear descriptions and expected outcomes.
docs/administration/pipelines/processors/lowercase.mdx (2)
40-52: Great addition of the Details section!The new Details section effectively communicates important aspects of the processor's behavior, particularly regarding array handling and non-string values. This helps users understand potential edge cases.
145-174: Well-structured conditional processing example.The new example effectively demonstrates how to use conditional processing with the lowercase processor. The use of
should_convertflag is a good practice for controlling processor behavior.docs/administration/pipelines/processors/date_index_name.mdx (2)
2-2: Improved clarity in descriptions.The terminology has been corrected from "indeces" to "index names", and the functionality description is now more comprehensive and accurate.
Also applies to: 29-29
50-60: Valuable addition of implementation details.The new Details section provides crucial information about:
- Time unit rounding capabilities
- ISO week handling specifics
- Date parsing failure scenarios
This helps users better understand the processor's behavior and potential pitfalls.
docs/administration/pipelines/processors/leef.mdx (1)
46-48: Important warning about message format requirements.Clear communication about the required "LEEF:" prefix and failure handling options. This helps users avoid common implementation issues.
docs/administration/pipelines/processors/html_strip.mdx (1)
40-53: LGTM! Well-structured "Details" section.The new "Details" section effectively explains the processor's functionality and includes important notes about text preservation. The warning about malformed HTML is particularly helpful for users.
docs/administration/pipelines/processors/normalize.mdx (1)
48-65: Excellent field mapping table!The table clearly shows the field name mappings across different formats (ECS, CIM, ASIM). This is very helpful for users understanding the normalization process.
docs/administration/pipelines/processors/convert.mdx (1)
48-72: Well-organized conversion types table!The TermTable format makes it easy to understand the available conversion types and their purposes.
docs/administration/pipelines/processors/encrypt.mdx (1)
50-56: Great explanation of compression benefits!The bullet points clearly explain the advantages of using compression with encryption.
docs/administration/pipelines/processors/json.mdx (1)
54-58: Clear explanation of conflict resolution strategies!The distinction between
replaceandmergestrategies is well explained.docs/administration/pipelines/processors/move.mdx (2)
27-27: LGTM! Clear and concise description.The description effectively communicates the processor's purpose without unnecessary details.
46-52: LGTM! Informative details section.The details section provides valuable information about the processor's behavior and capabilities.
docs/administration/pipelines/processors/grok.mdx (2)
26-26: LGTM! Clear and focused description.The description effectively communicates the core functionality.
46-57: LGTM! Well-structured pattern explanation.The details section effectively explains the pattern syntax and type conversion capabilities.
docs/administration/pipelines/processors/fingerprint.mdx (3)
26-26: LGTM! Comprehensive description.The description effectively explains the processor's capabilities and supported data types.
46-69: LGTM! Excellent data type handling documentation.The details section provides clear and comprehensive information about how different data types are handled.
79-107: LGTM! Well-structured use cases.The common use cases section provides practical examples with clear explanations.
docs/administration/pipelines/processors/geo_grid.mdx (1)
30-30: LGTM! Clear and concise description.The description effectively communicates the processor's purpose and supported formats.
docs/administration/pipelines/processors/foreach.mdx (2)
24-24: LGTM! Clear and concise description.The description effectively communicates the processor's purpose.
56-70: LGTM! Well-structured error handling documentation.The error handling table clearly explains available mechanisms.
New processor chapters added, multiple processor chapters proofed
Summary by CodeRabbit
Based on the comprehensive summary of changes, here are the release notes:
Documentation Updates
New Processors
Processor Documentation Improvements
Configuration Changes
Styling Enhancements
These changes focus on improving documentation clarity, adding new processor support, and refining the user experience for navigating and understanding pipeline processors.