Propagates the log topic through to the Vector send event. #33

kenlin-db · 2024-09-12T20:31:12Z

Adds a new util method for extracting the topic name from the log file upload pattern; adds it to the vector event log (so it can be propagated to the event log table).

The new util method is tested with a new unit test:

% cargo test extract_topic_name     
   Compiling vector v0.39.0-databricks-v1 (/Users/ken.lin/vector)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 24.00s
     Running unittests src/lib.rs (target/debug/deps/vector-b8c6a62aa94e6d9a)

running 2 tests
test sinks::util::vector_event::tests::extract_topic_name_fail ... ok
test sinks::util::vector_event::tests::extract_topic_name_success ... ok

… logging.

src/sinks/util/vector_event.rs

+
+// Utility function for extracting the topic name from an archived log file path.
+pub fn extract_topic_name(file_path: &str) -> String {
+    // Topic: If the file being uploaded matches the archived-log filepattern we can extract


src/sinks/util/vector_event.rs

+
+    #[test]
+    fn extract_topic_name_success() {
+        let file_path = "databricks-logs/archived-log/log-sync-internal/test-topic/date=2024-04-02/us-west-2/vector-aggregator-0/test.log";


src/sinks/util/vector_event.rs

+    fn extract_topic_name_success() {
+        let file_path = "databricks-logs/archived-log/log-sync-internal/test-topic/date=2024-04-02/us-west-2/vector-aggregator-0/test.log";
+        assert_eq!(extract_topic_name(file_path), "test-topic");
+        let file_path_structured = "databricks-logs/archived-log/log-sync-internal/structured-log/test-topic/date=2024-04-02/us-west-2/vector-aggregator-0/test.log";


kenlin-db added 3 commits September 12, 2024 13:10

Adds topic to VectorSendEventMetadata and includes it in Vector event…

dd26a81

… logging.

fix regex, test.

c3e16ce

make the structured-log/ token optional.

f2e420a

kenlin-db requested a review from andrewma2 September 12, 2024 20:31

github-actions bot added the domain: sinks label Sep 12, 2024

github-advanced-security bot found potential problems Sep 12, 2024

View reviewed changes

andrewma2 approved these changes Sep 12, 2024

View reviewed changes

kenlin-db merged commit 302ad0f into v0.39 Sep 13, 2024
41 of 49 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Propagates the log topic through to the Vector send event. #33

Propagates the log topic through to the Vector send event. #33

kenlin-db commented Sep 12, 2024

Propagates the log topic through to the Vector send event. #33

Propagates the log topic through to the Vector send event. #33

Conversation

kenlin-db commented Sep 12, 2024