Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic review pass #3

Open
wants to merge 6 commits into
base: review
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 41 additions & 46 deletions main.py
Original file line number Diff line number Diff line change
@@ -1,68 +1,63 @@
import os
import time
import boto3
import sys

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

s3_bucket_name = ""
filename = ""

class OnMyWatch:
# Set the directory on watch
watchDirectory = "e:/Code/indellinent/test/"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use pathlib.Path for easy path management?


def __init__(self):
def __init__(self, handlers):
self.observer = Observer()

self.handlers = handlers

def run(self):
event_handler = Handler()
self.observer.schedule(event_handler, self.watchDirectory)
for handler in self.handlers:
self.observer.schedule(handler, self.watchDirectory)

self.observer.start()
try:
while True:
time.sleep(5)
except:
while self.observer.isAlive():

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since this is a "running indefinitely" process add lots of logging to ensure useful diagnostic information?

loguru is an easy way to add really simple logging to your app, but it does diverge from using the standard logging library (it seems to be fairly popular though).

self.observer.join(5)
except KeyboardInterrupt:
# user requested stop
pass
finally:
self.observer.stop()
self.observer.join()
print("Observer Stopped")

self.observer.join()



class Handler(FileSystemEventHandler):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to add type hinting. The use of type-hinting seems to becoming a norm in Python through use of static checkers such as flake8, pyright and mypy. Besides that, using type hints definitely helps with debugging and maintenance to help readers of the code understand what data is flowing where. IDE's such as PyCharm (and VSCode, I think) will also warn you when it's obvious that the wrong data type is being applied to type hints - useful for your own development as well.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just remembered that boto3 is using some kind of dynamic type stuff (it's been a couple of years) so usually I had to use # type: ignore to pacify mypy and move on.

On the up-side, using # type: ignore would indicate to your reviewers that you know about mypy and assume it will be used and (hopefully) are even using it yourself.


def __init__(self, s3_client, bucket_name, target_filename):
self.s3 = s3_client
self.bucket_name = bucket_name
self.target_filename = target_filename

@staticmethod
def on_any_event(event):
def on_any_event(self, event):

if event.event_type == 'created' and filename in event.src_path :
s3 = boto3.client(
's3',
region_name = os.environ['AWS_REGION'],
aws_access_key_id = os.environ['AWS_ACCESS_KEY_ID'],
aws_secret_access_key = os.environ['AWS_SECRET_ACCESS_KEY']
)
if event.event_type == 'created' or event.event_type == 'closed' and self.target_filename in event.src_path:
self.s3.upload_file(event.src_path, self.bucket_name, self.target_filename)
print("Watchdog received %s event - %s." % (event.event_type, event.src_path))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use logging instead of print statements? (unless, of course, you are specifically wanting console output)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for this case I wanted console output, but that is good to know for future.


s3.upload_file(event.src_path, s3_bucket_name, filename)
print("Watchdog received created event - % s." % event.src_path)
elif event.event_type == 'modified' and filename in event.src_path :
s3 = boto3.client(
's3',
region_name = os.environ['AWS_REGION'],
aws_access_key_id = os.environ['AWS_ACCESS_KEY_ID'],
aws_secret_access_key = os.environ['AWS_SECRET_ACCESS_KEY']
)

s3.upload_file(event.src_path, s3_bucket_name, filename)
print("Watchdog received modified event - % s." % event.src_path)
else:
return None


if __name__ == '__main__':
import argparse

parser = argparse.ArgumentParser(
prog="monitor-specific-file",
description="Monitor a specific file, and upload it to a specific s3 bucket",
)
parser.add_argument('s3_bucket_name')
parser.add_argument('filename')
args = parser.parse_args()

s3_bucket_name = args.s3_bucket_name
filename = args.filename

s3 = boto3.client("s3")

#Set global variables as data that's passed from command line
s3_bucket_name = sys.argv[1]
filename = sys.argv[2]
test_handler = Handler(s3, s3_bucket_name, filename)

watch = OnMyWatch()
watch.run()
watch = OnMyWatch([test_handler])
watch.run()
14 changes: 14 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[project]
name = "monitor_specific_file"
version = "0.0.1"
readme = "README.md"
requires-python = ">=3.7"

dependencies = [
"boto3",
"watchdog",
]

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"