Watch + Sender all in one

[reidsunderland]

I was curious if it was possible to combine a watch and sender into one config, with no broker.

I did get it working, but not without broker settings. I also ran into a few crashes due to other missing options that are normally defined in watch.py's default options.

I put the config in ~/.config/sr3/sender/config.conf because I wasn't sure how to get sender.py's do method called in a Flow. Maybe it's possible using flowMain.

# watch + sender?

broker   amqp://bunnymaster@localhost/
exchange xs_bunnymaster

post_broker amqp://bunnymaster@localhost/
post_exchange xs_bunnymaster2

post_baseUrl sftp://localhost/

# watch config
callback gather.file
sleep    5s
path     /tmp/watchdir/
force_polling False
post_on_start True
blockSize 0

# send config
sendTo    sftp://localhost/
mirror False
directory /tmp/sender/

acceptUnmatched True

In this type of config, I doubt that we would ever want to consume messages from a broker, but it would be useful to optionally support posting to a broker (only the posting the files sent by the sender part, not the watch).

Here are the crashes I saw without certain options in the config:

[ERROR] 1012466 sarracenia.flow gather flowCallback plugin <bound method File.gather of <sarracenia.flowcb.gather.file.File object at 0x7fd23e7179d0>> crashed: 'Config' object has no attribute 'force_polling'

[ERROR] 1012519 sarracenia.flow gather flowCallback plugin <bound method File.gather of <sarracenia.flowcb.gather.file.File object at 0x7fa920217ac0>> crashed: 'Config' object has no attribute 'post_exchange'

[ERROR] 1012533 sarracenia.flow gather flowCallback plugin <bound method File.gather of <sarracenia.flowcb.gather.file.File object at 0x7fe7de67e950>> crashed: 'Config' object has no attribute 'post_on_start'

[ERROR] 1012596 sarracenia.flow gather flowCallback plugin <bound method File.gather of <sarracenia.flowcb.gather.file.File object at 0x7ff64f8024a0>> crashed: 'Config' object has no attribute 'blockSize'

[ERROR] 1012611 sarracenia updatePaths missing post_baseUrl setting
Traceback (most recent call last):
  File "/net/local/home/sunderlandr/sr3/sarracenia/instance.py", line 276, in <module>
    i.start()
  File "/net/local/home/sunderlandr/sr3/sarracenia/instance.py", line 267, in start
    self.running_instance.run()
  File "/net/local/home/sunderlandr/sr3/sarracenia/flow/__init__.py", line 600, in run
    self.filter()
  File "/net/local/home/sunderlandr/sr3/sarracenia/flow/__init__.py", line 1041, in filter
    url = self.o.variableExpansion(m['baseUrl'], m)
KeyError: 'baseUrl'

[petersilva]

This is very cool!

[petersilva]

I was thinking about this... and the downside is that the worklist is in memory so there is a bit of a risk if things crash and are restarted. the worklist will be gone, and might have already gone through duplicate suppression, but not yet sent.

not a show stopper... but thing to be aware of.

[petersilva]

every student/person I ever had starting out with zero knowledge of sr3... when asked to create a configuration, expected this sort of thing to work... it's kind of a PoLA violation that it doesn't work.

[andreleblanc11]

Creating a new component for this might make sense.