fsutil: add iostat() for collecting iostat on path or device.

iostat() is ported from s2/src s2iostat.py;
fsutil.iostat(): read sector size from /sys/block dir;

Other changes:

-   Refine README.md, move out config to standalone file `config.md`.

Author: drmingdrmer

README.md

@@ -111,59 +111,9 @@ jobq.run([0, 1, 2], [add1, printarg])
 
 #   Configuration
 
-`pykit` provides a way to setup config for it.
-Config module tries to import `pykitconfig` in which a user sets config.
-Example:
-
-```
-> cat pykitconfig.py
-uid = 2
-gid = 3
-
-> cat foo.py
-from pykit import fsutil
-fsutil.write_file('bar', '123') # write_file sets file uid and gid to 2 and 3.
-```
-
-##  Supported config
-
--   `uid`: specifies default user-id  when file created, directory made.
--   `gid`: specifies default group-id when file created, directory made.
--   `log_dir`: specifies default base_dir when logger created.
--   `cat_stat_dir`: specifies default stat_dir for all log cat class instances.
-
--   `zk_node_id`:
-    specifies a string to identify this host, by default it is `uuid.getnode()`.
-
-    `ZKLock` uses it to differentiate what host actually holds a lock.
-
-    In `ZKLock`:
-
-    - Different host must specify different `node_id`
-
-    - But A host could specify different `node_id` for different locks.
-
--   `zk_acl`:
-    default acl for new created zk-node.
-    Such as `(('xp', '123', 'cdrwa'), ('foo', 'bar', 'rw'))`
-
--   `zk_auth`:
-    default auth info for new connection.
-    Such as `('digest', 'xp', '123')`
-
--   `zk_hosts`:
-    default comma separated host list for new connection.
-    Such as `'127.0.0.1:2181,128.0.0.8:2181'`
-
--   `zk_lock_dir`:
-    default base dir for `ZKLock` to create lock node.
+See [config.md](config.md)
 
 
-
-
-
-See the `README.md` of sub modules for detail.
-
 #   Test
 
 Run one of following to test all, a module, a TestCase or a function.
@@ -184,41 +134,8 @@ See [Details](script/README.md)
 There are several scripts for developers.
 See [script](script).
 
-##  Config
-
-`pykit.config` is used internally by `pykit` modules.
-
-### How it works
-
-It try to load `pykitconfig` from a top level module of the project.
-
-Then it load predefined config via ```getattr(pykitconfig, key, default)```.
-
-So module can get config via `config.xxx` by import it.
-
-### How to use
-
-- Add predefined config in config.py
-
-    ```
-    uid=_get('uid')
-    gid=_get('gid')
-    ```
-
-- Import config in you own module
-
-    ```
-    from pykit import config
-    ```
-
-- Handle user define config in you code
 
-    ```
-    uid=uid or config.uid
-    gid=gid or config.gid
-    ```
 
-- Update the configuration supported config in this doc
 
 #   Author
 

config.md

@@ -0,0 +1,110 @@
+# Pykit configuration
+
+`pykit` provides a way to setup config for it.
+An end user can config `pykit` behavior without modify source code.
+
+
+## How it works
+
+Modules those support configuration will load `pykit.config`
+and `pykit.config` tries to import `pykitconfig` from `sys.path`.
+
+The `pykit.config` loads attributes defined in `pykitconfig` those it recognizes.
+
+
+## Usage
+
+
+### For pykit user
+
+For end user one just creates a file `pykitconfig.py` and adds config into it.
+E.g.:
+
+`cat pykitconfig.py`:
+
+```python
+import my_project_conf
+
+uid = 2
+gid = 3
+log_dir = my_project_conf.log_dir
+```
+
+And pykit will automatically load `pykitconfig.py` and use them.
+
+```python
+from pykit import fsutil
+fsutil.write_file('bar', '123') # write_file sets file uid and gid to 2 and 3.
+```
+
+### For pykit module developer
+
+-   Add predefined config in `config.py`, to specify the field name default
+    value.
+
+    ```python
+    # cat pykit/config.py
+    uid=_get('uid')
+    gid=_get('gid')
+
+    my_conf = _get('my_conf', "default_value")
+    ```
+
+-   Import config in you own module
+
+    ```python
+    # cat pykit/mymodule/foo.py
+    from pykit import config
+
+    print config.my_conf # "default_value"
+    ```
+
+##  Supported config
+
+-   `uid`:
+    specifies default user-id  when file created, directory made.
+
+-   `gid`:
+    specifies default group-id when file created, directory made.
+
+-   `log_dir`:
+    specifies default base_dir when logger created.
+
+-   `cat_stat_dir`:
+    specifies default stat_dir for all log cat class instances.
+
+-   `iostat_stat_path`:
+    specifies the default path to store io stat for `fsutil.iostat`.
+
+-   `zk_node_id`:
+    specifies a string to identify this host, by default it is `uuid.getnode()`.
+
+    `ZKLock` uses it to differentiate what host actually holds a lock.
+
+    In `ZKLock`:
+
+    - Different host must specify different `node_id`
+
+    - But A host could specify different `node_id` for different locks.
+
+-   `zk_acl`:
+    default acl for new created zk-node.
+    Such as `(('xp', '123', 'cdrwa'), ('foo', 'bar', 'rw'))`
+
+-   `zk_auth`:
+    default auth info for new connection.
+    Such as `('digest', 'xp', '123')`
+
+-   `zk_hosts`:
+    default comma separated host list for new connection.
+    Such as `'127.0.0.1:2181,128.0.0.8:2181'`
+
+-   `zk_lock_dir`:
+    default base dir for `ZKLock` to create lock node.
+
+-   `zk_record_dir`:
+    specifies zk node path base dir to store **records** for `zktx`.
+    It must ends with `/`.
+
+-   `zk_tx_dir`:
+    specifies the base dir of zk node path to store tx related info.

config.py

@@ -25,6 +25,7 @@ def _get(key, default=None):
 gid                  = _get('gid')
 log_dir              = _get('log_dir')
 cat_stat_dir         = _get('cat_stat_dir')
+iostat_stat_path     = _get('iostat_stat_path', '/tmp/pykit-iostat')
 zk_acl               = _get('zk_acl')                   # (('xp', '123', 'cdrwa'), ('foo', 'bar', 'rw'))
 zk_auth              = _get('zk_auth')                  # ('digest', 'xp', '123')
 zk_hosts             = _get('zk_hosts', '127.0.0.1:2181')

fsutil/README.md

@@ -34,6 +34,8 @@
 - [Stat methods](#stat-methods)
   - [fsutil.get_path_inode_usage](#fsutilget_path_inode_usage)
   - [fsutil.get_path_usage](#fsutilget_path_usage)
+  - [fsutil.iostat](#fsutiliostat)
+    - [Implementation](#implementation)
 - [Author](#author)
 - [Copyright and License](#copyright-and-license)
 
@@ -667,6 +669,71 @@ then it can not use the reserved space.
 Thus this function provides with the `available` bytes by default.
 
 
+##  fsutil.iostat
+
+**syntax**:
+`fsutil.iostat(device=None, path=None, stat_path=None)`
+
+Collect IO stat.
+
+**Synopsis**:
+
+```python
+print fsutil.iostat('/dev/sda1') # {'read': 6151, 'write': 34073, 'ioutil': 0}
+print fsutil.iostat(path='/')    # {'read': 6151, 'write': 34073, 'ioutil': 100}
+```
+
+It accepts either `device` or `path` as target to collect IO stat from:
+
+-   `device` should be a path starts with `/dev`, such as `/dev/sda1`.
+
+-   `path` is any path on a valid mounted fs. If `path` is used and `device` is
+    `None`, it uses the device on which the `path` is mounted.
+
+One must specify either `device` or `path`.
+
+
+### Implementation
+
+`/proc/diskstats` provides accumulated IO stat since a host boots up.
+Such as total count of read/write operation on a disk.
+
+This function records changes in `/proc/diskstats` and calculates the diff
+between two recorded stat as return value.
+
+`fsutil.iostat` reads instant IO stat from `/proc/diskstats` and save it in
+`stat_path`. When next time `fsutil.iostat` is called, it calculates the
+difference between the current stat from `/proce/diskstats` and the saved stat.
+
+If no previous recorded stat saved in `stat_path`, it waits a second and load
+`/proc/diskstats` again, and calculate the diff.
+
+**arguments**:
+
+-   `device`:
+    specifies from which device to collect IO stat.
+
+-   `path`:
+    specifies from which fs path to collect IO stat.
+
+-   `stat_path`:
+    specifies where to store and load IO stat.
+
+    By default it is `None`, then it uses `config.iostat_stat_path`(`/tmp/pykit-iostat`) to save
+    stat.
+
+**return**:
+a dict contains 3 field:
+```json
+{
+'read': 6151,
+'write': 34073,
+'ioutil': 0
+}
+```
+
+`read` and `write` is in byte/second.
+`ioutil` is a percentage number between 0 and 100.
 
 
 #   Author

fsutil/__init__.py

@@ -30,6 +30,12 @@
     Cat
 )
 
+from .iostat import (
+    DeviceNotFound,
+
+    iostat,
+)
+
 __all__ = [
     "FSUtilError",
     "NotMountPoint",
@@ -58,5 +64,9 @@
     "LockTimeout",
     "NoData",
     "NoSuchFile",
-    "Cat"
+    "Cat",
+
+    "DeviceNotFound",
+
+    "iostat",
 ]