v1.0.0

README.md

@@ -0,0 +1,92 @@
+# mtnlog - A simple multinode performance logger for Python
+
+## Introduction
+
+mtnlog is a simple multinode performance logger for Python. It is designed to be used in a similar way to Python's built-in logging module, but with a focus on performance logging. It provides a simple API for logging performance data, including start and end times, and allows for easy integration with other logging systems.
+
+## Installation
+
+You can install mtnlog using pip:
+
+```bash
+pip install mtnlog
+```
+
+## Usage
+
+To use mtnlog, you have two features: `JSONLogger` and `PerformanceLogger`.
+
+### JSONLogger
+
+The `JSONLogger` class is a simple logger that writes performance data to a JSON file. You can create a new `JSONLogger` instance by passing a file path to the constructor:
+
+```python
+from mtnlog import JSONLogger
+
+logger = JSONLogger(log_dir='logs') # logs is the directory where the log file will be saved
+```
+
+You can then use the `log` method to log performance data:
+
+```python
+logger.log('<your_dict>', filename='log') # your_dict is a dictionary with the data you want to log / filename is the name of the file
+```
+
+`your_dict` is a dictionary with the data you want to log.
+`filename` is the name of the file where the data will be saved
+
+### PerformanceLogger
+
+The `PerformanceLogger` class is a logger for system performance data. It logs the the time taken to execute the block, as well as the CPU, memory, and GPU usage. You can create a new `PerformanceLogger` instance by passing a file path to the constructor:
+
+```python
+from mtnlog import PerformanceLogger
+
+collector = PerformanceLogger(log_dir="<your_log_dir>", log_node="<current_node>")
+```
+
+`your_log_dir` is the directory where the log file will be saved.
+`current_node` is the number of the node you are logging.
+
+You can then use the `change_tag` method to change the tag of the log:
+
+```python
+collector.change_tag("<new_tag>")
+```
+
+`new_tag` is the new tag you want to use.
+
+To stop logging, you can use the `stop` method:
+
+```python
+collector.stop()
+```
+
+## Example
+
+Here is an example of how to use mtnlog:
+
+```python
+from mtnlog import JSONLogger, PerformanceLogger
+
+# Create a JSONLogger instance
+
+logger = JSONLogger(log_dir='logs')
+
+# Log some data
+
+logger.log({'message': 'Hello, world!'}, filename='log')
+
+# Create a PerformanceLogger instance
+
+collector = PerformanceLogger(log_dir='logs', log_node="0")
+
+# Change the tag
+
+collector.change_tag('new_tag')
+
+# Stop logging
+
+collector.stop()
+
+```

pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "mtnlog"
+version = "1.0.0"
+authors = [
+    { name = "Wongkraiwich Chuenchomphu", email = "[email protected]" },
+]
+description = "A simple performance logger for Python"
+readme = "README.md"
+requires-python = ">=3.8"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+
+[project.urls]
+Homepage = "https://github.com/kentakoong/mtnlog"
+Issues = "https://github.com/kentakoong/mtnlog/issues"

src/mtnlog.egg-info/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.1
+Name: mtnlog
+Version: 1.0.0
+Summary: A simple performance logger for Python
+Author-email: Wongkraiwich Chuenchomphu <[email protected]>
+Project-URL: Homepage, https://github.com/kentakoong/mtnlog
+Project-URL: Issues, https://github.com/kentakoong/mtnlog/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+# mtnlog - A simple multinode performance logger for Python
+
+## Introduction
+
+mtnlog is a simple multinode performance logger for Python. It is designed to be used in a similar way to Python's built-in logging module, but with a focus on performance logging. It provides a simple API for logging performance data, including start and end times, and allows for easy integration with other logging systems.
+
+## Installation
+
+You can install mtnlog using pip:
+
+```bash
+pip install mtnlog
+```
+
+## Usage
+
+To use mtnlog, you have two features: `JSONLogger` and `PerformanceLogger`.
+
+### JSONLogger
+
+The `JSONLogger` class is a simple logger that writes performance data to a JSON file. You can create a new `JSONLogger` instance by passing a file path to the constructor:
+
+```python
+from mtnlog import JSONLogger
+
+logger = JSONLogger(log_dir='logs') # logs is the directory where the log file will be saved
+```
+
+You can then use the `log` method to log performance data:
+
+```python
+logger.log('<your_dict>', filename='log') # your_dict is a dictionary with the data you want to log / filename is the name of the file
+```
+
+`your_dict` is a dictionary with the data you want to log.
+`filename` is the name of the file where the data will be saved
+
+### PerformanceLogger
+
+The `PerformanceLogger` class is a logger for system performance data. It logs the the time taken to execute the block, as well as the CPU, memory, and GPU usage. You can create a new `PerformanceLogger` instance by passing a file path to the constructor:
+
+```python
+from mtnlog import PerformanceLogger
+
+collector = PerformanceLogger(log_dir="<your_log_dir>", log_node="<current_node>")
+```
+
+`your_log_dir` is the directory where the log file will be saved.
+`current_node` is the number of the node you are logging.
+
+You can then use the `change_tag` method to change the tag of the log:
+
+```python
+collector.change_tag("<new_tag>")
+```
+
+`new_tag` is the new tag you want to use.
+
+To stop logging, you can use the `stop` method:
+
+```python
+collector.stop()
+```
+
+## Example
+
+Here is an example of how to use mtnlog:
+
+```python
+from mtnlog import JSONLogger, PerformanceLogger
+
+# Create a JSONLogger instance
+
+logger = JSONLogger(log_dir='logs')
+
+# Log some data
+
+logger.log({'message': 'Hello, world!'}, filename='log')
+
+# Create a PerformanceLogger instance
+
+collector = PerformanceLogger(log_dir='logs', log_node="0")
+
+# Change the tag
+
+collector.change_tag('new_tag')
+
+# Stop logging
+
+collector.stop()
+
+```

src/mtnlog.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+README.md
+pyproject.toml
+src/mtnlog/__init__.py
+src/mtnlog/json.py
+src/mtnlog/performance.py
+src/mtnlog.egg-info/PKG-INFO
+src/mtnlog.egg-info/SOURCES.txt
+src/mtnlog.egg-info/dependency_links.txt
+src/mtnlog.egg-info/top_level.txt

src/mtnlog.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

src/mtnlog.egg-info/top_level.txt

@@ -0,0 +1 @@
+mtnlog

src/mtnlog/__init__.py

@@ -0,0 +1,11 @@
+from .json import *
+from .performance import *
+
+__version__ = '1.0.0'
+
+__doc__ = """Performance logger for tracking resource usage."""
+
+__all__ = [
+    'JSONLogger',
+    'PerformanceLogger',
+]

src/mtnlog/json.py

@@ -0,0 +1,28 @@
+"""JSON logger class for logging arguments and results."""
+
+import json
+import os
+
+
+class JSONLogger:
+    """Arguments logger class."""
+
+    def __init__(self, log_dir):
+        os.makedirs(log_dir, exist_ok=True)
+        self.log_dir = log_dir
+
+    def serialize(self, obj):
+        """Custom serialization for non-serializable objects."""
+        if isinstance(obj, dict):
+            return {k: self.serialize(v) for k, v in obj.items()}
+        if isinstance(obj, list):
+            return [self.serialize(v) for v in obj]
+        if isinstance(obj, (int, float, str, bool, type(None))):
+            return obj
+        # Convert non-serializable objects to their string representation
+        return str(obj)
+
+    def log(self, obj, filename="log"):
+        """Logs the object."""
+        with open(f"{self.log_dir}/{filename}.json", "w", encoding='utf-8') as f:
+            json.dump(self.serialize(obj), f, ensure_ascii=False, indent=4)

src/mtnlog/performance.py

@@ -0,0 +1,81 @@
+"""Performance logger class for logging performance metrics."""
+
+import os
+import psutil
+
+import pandas as pd
+from nvitop import ResourceMetricCollector, Device
+
+
+class PerformanceLogger:
+    """Performance logger class."""
+
+    def __init__(self, log_dir, log_node):
+
+        os.makedirs(log_dir, exist_ok=True)
+
+        self.log_dir = log_dir
+        self.log_node = log_node
+        self.df = pd.DataFrame()
+        self.tag = None
+        self.filepath = None
+        self.collector = ResourceMetricCollector(Device.cuda.all()).daemonize(
+            on_collect=self.on_collect,
+            interval=1.0,
+        )
+        self.cpu_count = psutil.cpu_count(logical=False)
+        self.start_time = None
+
+    def new_res(self):
+        """Returns the directory."""
+
+        os.makedirs(f"{self.log_dir}/{self.tag}", exist_ok=True)
+
+        self.filepath = f"{self.log_dir}/{self.tag}/node-{self.log_node}.csv"
+
+    def change_tag(self, tag):
+        """Changes the tag."""
+        if self.filepath is not None:
+            self.stop()
+        self.tag = tag
+        self.new_res()
+
+    def stop(self):
+        """Stops the collector."""
+        if not self.df.empty:
+            self.df.to_csv(self.filepath, index=False)
+        self.df = pd.DataFrame()
+
+    def get_cpu_usage_per_core(self):
+        """Returns the CPU usage per core."""
+        cpu_percent = psutil.cpu_percent(interval=0.1, percpu=True)
+        return {f"cpu_core_{i+1}": percent for i, percent in enumerate(cpu_percent[:self.cpu_count])}
+
+    def clean_column_name(self, col):
+        """Cleans the column name."""
+        if col.startswith("metrics-daemon/host/"):
+            col = col[len("metrics-daemon/host/"):]
+        return col
+
+    def on_collect(self, metrics):
+        """Collects metrics."""
+
+        metrics['tag'] = self.tag
+
+        cpu_metrics = self.get_cpu_usage_per_core()
+        metrics.update(cpu_metrics)
+
+        df_metrics = pd.DataFrame.from_records([metrics])
+
+        df_metrics.columns = [self.clean_column_name(col) for col in df_metrics.columns]
+
+        if self.df.empty:
+            self.df = df_metrics
+        else:
+            for col in df_metrics.columns:
+                if col not in self.df.columns:
+                    self.df[col] = None
+
+            self.df = pd.concat([self.df, df_metrics], ignore_index=True)
+
+        return True