initial release

Author: piotrj

.github/workflows/run.yml

@@ -134,7 +134,7 @@ jobs:
         tag_name: ${{ steps.version.outputs.version }}
         name: Release ${{ steps.version.outputs.version }}
         draft: false
-        prerelease: true
+        prerelease: false
         files: |
             librer.${{ steps.version.outputs.version }}.portable.linux.zip
             librer.${{ steps.version.outputs.version }}.portable.windows.zip

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2023 Piotr Jochymek
+Copyright (c) 2023-2024 Piotr Jochymek
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -2,14 +2,12 @@
 
 A cross-platform GUI file cataloging program with extensive customization options to suit user preferences. Highly optimized for multi-core parallel search speed, data integrity, and repository portability.
 
-**The software and this document are under development and in a Release Candidate (3) state.**
-
 ## Features:
 The primary purpose of this software is to enable users to catalog their files, especially on removable media such as memory cards and portable drives. It allows user to add metadata, referred to here as **custom data**, and later search the created records with multiple cryteria. **Custom data** consists of textual information acquired through the execution of user-chosen commands or scripts. **Custom data** may include any text data customized to meet the user's requirements, restricted only by the available memory and the software accessible for data retrieval. The retrieved data is stored in a newly created database record and can be utilized for search or verification purposes. **Liber**  allows you to search files using regular expressions, glob expressions and **fuzzy matching** on both filenames and **custom data**. Created data records can be exported and imported to share data with others or for backup purposes.
 
 ## Screenshots:
 
-main window, new record creation dialog and running **custom data** extraction:
+#### Main window, new record creation dialog and running **custom data** extraction:
 ![image info](./info/scanning.png)
 
 #### Search dialog:
@@ -23,16 +21,16 @@ Portable executable packages created with [PyInstaller](https://pyinstaller.org/
 
 https://github.com/PJDude/librer/releases
 
+## [Tutorial](./info/tutorial.md) ##
 
 ## Guidelines for crafting custom data extractors
 Custom data extractor is a command that can be invoked with a single parameter - the full path to a specific file from which data is extracted. The command should provide the expected data in any textual format to the standard output (stdout). CDE can be an executable file (e.g., 7z, zip, ffmpeg, md5sum etc.) or an executable shell script (extract.sh, extract.bat etc.). The conditions it should meet are reasonably short execution time and reasonably limited information output. The criteria allowing the execution of a particular **Custom data extractor** include the glob expression (on file name) and the file size range.
 
-## [Tutorial](./info/tutorial.md) ##    
 
 ## Usage tips:
-- don't put any destructive actions in your Custom Data Extractors scrips
+- do not put any destructive actions in your Custom Data Extractor scripts
 - try to keep as little custom data as possible, to speed up scanning and searching records and keep record files small
-- if general purpose and generally available tools produce too much and not needed text data, write a wrapper script (*.sh, *bat) executing specific tool and post-process retrieved data. Wrapper can also help when the tool expects more variable parameters or in an unusual order
+- if general purpose and generally available tools produce too much and not needed text data, write a wrapper script (*.sh, *bat) executing specific tool and post-process retrieved data.
 - You don't have to use Custom Data if you don't need to. Only the file system will be cataloged.
 
 ## Supported platforms:
@@ -43,7 +41,7 @@ Custom data extractor is a command that can be invoked with a single parameter -
 **librer** writes log files, configuration and record files in runtime. Default location for these files is **logs** and **data** subfolders of **librer** main directory.
 
 ## Technical information
-Record in librer is the result of a single scan operation and is shown as one of many top nodes in the main tree window. Contains a directory tree with collected custom data. It is stored as a single .dat file in librer database directory. Its internal format is optimized for security, fast initial access and maximum compression (just check :)) Every section is a python data structure serialized by [pickle](https://docs.python.org/3/library/pickle.html) and compressed separately by [Zstandard](https://pypi.org/project/zstandard/) algorithm. The record file, once saved, is never modified afterward. It can only be deleted upon request or exported. All record files are independent of each other.Fuzzy matching is implemented using the SequenceMatcher function provided by the [difflib](https://docs.python.org/3/library/difflib.html) package. Searching records is performed as a separate subprocess for each record. The number of parallel searches is limited by the CPU cores. 
+Record in librer is the result of a single scan operation and is shown as one of many top nodes in the main tree window. Contains a directory tree with collected custom data. It is stored as a single .dat file in librer database directory. Its internal format is optimized for security, fast initial access and maximum compression (just check :)) Every section is a python data structure serialized by [pickle](https://docs.python.org/3/library/pickle.html) and compressed separately by [Zstandard](https://pypi.org/project/zstandard/) algorithm. The record file, once saved, is never modified afterward. It can only be deleted upon request or exported. All record files are independent of each other. Fuzzy matching is implemented using the SequenceMatcher function provided by the [difflib](https://docs.python.org/3/library/difflib.html) module. Searching records is performed as a separate subprocess for each record. The number of parallel searches is limited by the CPU cores. 
 
 ###### Manual build (linux):
 ```
@@ -73,8 +71,5 @@ python3 ./src/librer.py
 - calculate the **CRC** of scanned files and use it to search for duplicates among different records, verify current data with the saved file system image
 - comparing two records with each other. e.g. two scans of the same file system performed at different times
 
-## Known issues
-For still unknown reason, Custom Data Extraction (Execution of tons of subprocesses) on the exact same hardware is much slower on Windows than on Linux.  
-
 ## Licensing
 - **librer** is licensed under **[MIT license](./LICENSE)**

src/core.py

@@ -2,7 +2,7 @@
 
 ####################################################################################
 #
-#  Copyright (c) 2023 Piotr Jochymek
+#  Copyright (c) 2023-2024 Piotr Jochymek
 #
 #  MIT License
 #
@@ -40,15 +40,13 @@
 else:
     from os import getpgid, killpg
 
-from os.path import abspath,normpath,basename,dirname
-from os.path import join as path_join
+from os.path import abspath,normpath,basename,dirname,join as path_join
 
 from zipfile import ZipFile
 from platform import system as platform_system,release as platform_release,node as platform_node
 
 from fnmatch import fnmatch
 from re import search as re_search
-from sys import getsizeof
 import sys
 from collections import defaultdict
 from pathlib import Path as pathlib_Path
@@ -125,12 +123,14 @@ def fnumber(num):
     return str(format(num,',d').replace(',',' '))
 
 def str_to_bytes(string):
-    units = {'kb': 1024,'mb': 1024*1024,'gb': 1024*1024*1024,'tb': 1024*1024*1024*1024, 'b':1}
     try:
-        string = string.replace(' ','')
-        for suffix,weight in units.items():
-            if string.lower().endswith(suffix):
+        string = string.replace(' ','').lower()
+        string_endswith = string.endswith
+        for suffix,altsuffix,weight in ( ('kb','k',1024),('mb','m',1024*1024),('gb','g',1024*1024*1024),('tb','t',1024*1024*1024*1024),('b','b',1) ):
+            if string_endswith(suffix):
                 return int(string[0:-len(suffix)]) * weight #no decimal point
+            elif string_endswith(altsuffix):
+                return int(string[0:-len(altsuffix)]) * weight #no decimal point
 
         return int(string)
     except:
@@ -196,7 +196,6 @@ def get_command(executable,parameters,full_file_path,shell):
 #'ignore','replace','backslashreplace'
 def popen_win(command,shell):
     return Popen(command, stdout=PIPE, stderr=STDOUT,stdin=DEVNULL,shell=shell,text=True,universal_newlines=True,creationflags=CREATE_NO_WINDOW,close_fds=False,errors='ignore')
-    universal
 
 def popen_lin(command,shell):
     return Popen(command, stdout=PIPE, stderr=STDOUT,stdin=DEVNULL,shell=shell,text=True,universal_newlines=True,start_new_session=True,errors='ignore')
@@ -610,7 +609,7 @@ def threaded_cde(timeout_semi_list):
 
             aborted_string = 'Custom data extraction was aborted.'
             for (scan_like_list,subpath,rule_nr,size) in self.customdata_pool.values():
-                #decoding_error=False
+
                 self.killed=False
                 self.abort_single_file_cde=False
 
@@ -636,7 +635,7 @@ def threaded_cde(timeout_semi_list):
                         subprocess = uni_popen(command,shell)
                         timeout_semi_list[0]=(timeout_val,subprocess)
                     except Exception as re:
-                        print('threaded_cde error:',re)
+                        #print('threaded_cde error:',re)
                         subprocess = None
                         timeout_semi_list[0]=(timeout_val,subprocess)
                         returncode=201
@@ -651,12 +650,6 @@ def threaded_cde(timeout_semi_list):
                         while True:
                             line = subprocess_stdout_readline().rstrip()
 
-                            #try:
-                            #except Exception as le:
-                                #print(command,le)
-                            #    line = str(le)
-                                #decoding_error = True
-
                             output_list_append(line)
 
                             if not line and subprocess_poll() is not None:
@@ -681,7 +674,7 @@ def threaded_cde(timeout_semi_list):
                 if not aborted:
                     self_header.files_cde_quant += 1
                     self_header.files_cde_size += size
-                    self_header.files_cde_size_extracted += getsizeof(output)
+                    self_header.files_cde_size_extracted += asizeof(output)
 
                 new_elem={}
                 new_elem['cd_ok']= bool(returncode==0 and not self.killed and not aborted)

src/dialogs.py

@@ -2,7 +2,7 @@
 
 ####################################################################################
 #
-#  Copyright (c) 2023 Piotr Jochymek
+#  Copyright (c) 2023-2024 Piotr Jochymek
 #
 #  MIT License
 #

src/librer.py

@@ -2,7 +2,7 @@
 
 ####################################################################################
 #
-#  Copyright (c) 2023 Piotr Jochymek
+#  Copyright (c) 2023-2024 Piotr Jochymek
 #
 #  MIT License
 #
@@ -27,9 +27,8 @@
 ####################################################################################
 
 from os import sep,system,getcwd,name as os_name
-from os.path import abspath,normpath,dirname
-from os.path import join as path_join
-from os.path import isfile as path_isfile
+from os.path import abspath,normpath,dirname,join as path_join,isfile as path_isfile
+
 from pathlib import Path
 from time import strftime,time,mktime
 from signal import signal,SIGINT
@@ -2730,22 +2729,19 @@ def tree_semi_focus(self):
             tree.see(item)
             tree.selection_set(item)
 
-            self.tree_sel_change(item,True)
+            self.tree_sel_change(item)
         else:
             self.sel_item = None
 
     @catched
-    def tree_sel_change(self,item,force=False,change_status_line=True):
+    def tree_sel_change(self,item,change_status_line=True):
         self.sel_item = item
 
         if change_status_line :
             self.status('')
 
         self_tree_set_item=lambda x : self.tree_set(item,x)
 
-        #path=self_tree_set_item('path')
-
-        #self.sel_kind = self_tree_set_item('kind')
         self.tree_select()
 
     def menubar_unpost(self):

src/png.2.py.py

@@ -2,7 +2,7 @@
 
 ####################################################################################
 #
-#  Copyright (c) 2022-2023 Piotr Jochymek
+#  Copyright (c) 2022-2024 Piotr Jochymek
 #
 #  MIT License
 #

src/record.py

@@ -2,7 +2,7 @@
 
 ####################################################################################
 #
-#  Copyright (c) 2022-2023 Piotr Jochymek
+#  Copyright (c) 2023-2024 Piotr Jochymek
 #
 #  MIT License
 #
@@ -26,14 +26,14 @@
 #
 ####################################################################################
 
-import argparse
-import os
-#import signal
-#from sys import exit
-#from subprocess import DEVNULL
-import pathlib
 import sys
 
+from os.path import dirname,join as path_join
+from os import name as os_name
+
+from pathlib import Path as pathlib_Path
+from argparse import ArgumentParser,RawTextHelpFormatter
+
 from time import sleep,perf_counter
 
 from threading import Thread
@@ -42,39 +42,28 @@
 from fnmatch import translate
 from difflib import SequenceMatcher
 
-#from pickle import load
-
-#from multiprocessing import Process, Queue
-#from queue import Empty
-
-#import base64
-#import codecs
-
-import io
-
 from json import dumps as json_dumps
+from collections import deque
 
 from core import *
 
 VERSION_FILE='version.txt'
 
 def get_ver_timestamp():
     try:
-        timestamp=pathlib.Path(os.path.join(os.path.dirname(__file__),VERSION_FILE)).read_text(encoding='ASCII').strip()
+        timestamp=pathlib_Path(path_join(dirname(__file__),VERSION_FILE)).read_text(encoding='ASCII').strip()
     except Exception as e_ver:
         print(e_ver)
         timestamp=''
     return timestamp
 
 def parse_args(ver):
-    parser = argparse.ArgumentParser(
-            formatter_class=argparse.RawTextHelpFormatter,
-            prog = 'record.exe' if (os.name=='nt') else 'record',
-            description = f"librer record version {ver}\nCopyright (c) 2023 Piotr Jochymek\n\nhttps://github.com/PJDude/librer",
+    parser = ArgumentParser(
+            formatter_class=RawTextHelpFormatter,
+            prog = 'record.exe' if (os_name=='nt') else 'record',
+            description = f"librer record version {ver}\nCopyright (c) 2023-2024 Piotr Jochymek\n\nhttps://github.com/PJDude/librer",
             )
 
-    #parser.add_argument('--foo', required=True)
-
     parser.add_argument('command',type=str,help='command to execute',choices=('load','search','info'))
 
     parser.add_argument('file',type=str,help='dat file')
@@ -143,13 +132,9 @@ def find_params_check(self,
 
     return None
 
-from collections import deque
-
-#results_queue=Queue()
 results_queue=deque()
 
 def printer():
-    #results_queue_get = results_queue.get
     results_queue_get = results_queue.popleft
 
     try:
@@ -161,7 +146,7 @@ def printer():
                 print(json_dumps(result))
             else:
                 sys.stdout.flush()
-                sleep(0.01)
+                sleep(0.001)
 
     except Exception as pe:
         print_info('printer error:{pe}')
@@ -172,12 +157,7 @@ def printer():
 def print_info(*args):
     print('#',*args)
 
-
 if __name__ == "__main__":
-    #buffer_size = 1024*1024*64
-    #sys.stdout = io.TextIOWrapper(sys.stdout.detach(), write_through=True, line_buffering=False)
-    #sys.stdout._CHUNK_SIZE = buffer_size
-
     VER_TIMESTAMP = get_ver_timestamp()
 
     args=parse_args(VER_TIMESTAMP)

src/version.pi.template.librer.txt

@@ -33,7 +33,7 @@ VSVersionInfo(
         StringStruct(u'FileDescription', u'Librer'),
         StringStruct(u'FileVersion', u'0.0.0.0'),
         StringStruct(u'InternalName', u'Librer'),
-        StringStruct(u'LegalCopyright', u'Piotr Jochymek 2023'),
+        StringStruct(u'LegalCopyright', u'Piotr Jochymek 2023-2024'),
         StringStruct(u'OriginalFilename', u'librer.exe'),
         StringStruct(u'ProductName', u'Librer'),
         StringStruct(u'ProductVersion', u'VER_TO_REPLACE')])

src/version.pi.template.record.txt

@@ -33,7 +33,7 @@ VSVersionInfo(
         StringStruct(u'FileDescription', u'Librer-Record'),
         StringStruct(u'FileVersion', u'0.0.0.0'),
         StringStruct(u'InternalName', u'record'),
-        StringStruct(u'LegalCopyright', u'Piotr Jochymek 2023'),
+        StringStruct(u'LegalCopyright', u'Piotr Jochymek 2023-2024'),
         StringStruct(u'OriginalFilename', u'record.exe'),
         StringStruct(u'ProductName', u'Librer-Record'),
         StringStruct(u'ProductVersion', u'VER_TO_REPLACE')])

src/version.py

@@ -2,7 +2,7 @@
 
 ####################################################################################
 #
-#  Copyright (c) 2023 Piotr Jochymek
+#  Copyright (c) 2023-2024 Piotr Jochymek
 #
 #  MIT License
 #