(63) - Configure nostr-sync server with cronjobs

[nbenmoody]

Description/Goals
I want to create a server to run a few specific sync jobs on a daily basis.

Summary of Changes

• I wrote up a new inventory/role/playbook to clone the nostr-sync-scripts repo and add the specific scripts to the crontab.

Tasks Before Merging

• Create the sync-server and update the inventory with it as the host.
• Run the playbook against it.

[mplorentz]

Hey Ben, we usually link the ticket number in the PR description, it creates a bidrectional link. I'll create one here:
https://github.com/verse-pbc/issues/issues/63

[nbenmoody]

Hey Ben, we usually link the ticket number in the PR description, it creates a bidrectional link. I'll create one here: verse-pbc/issues#63

Notes taken, thanks man!

[mplorentz]

I pushed some fixes to this branch. I think this should get the script running but we can let it run overnight to see for sure. They will fail but they should execute and leaves some logs in /var/log/syslog.

[mplorentz]

I looked at the logs this morning and I'm confused about what I saw. I pushed another small fix and I'm going to look again tomorrow morning.

[mplorentz]

Ok these scripts are running now. I updated them to run every hour because it will keep the content fresh and the one I fixed (news_sync) only takes a second to execute when it is run frequently. Maybe in the future we should play with running them even more frequently.

The other two scripts fail on every run, but that's not a problem with these ansible-scripts, so we'll deal with that elsewhere.

@nbenmoody even though you opened this originally can you take a look at the PR and the syslog and merge if you are satisfied?

[nbenmoody]

@rabble - I think @mplorentz's update just matches what we manually did last week, but he has increased the grow_fedi script to run every hour also, instead of each day like we had it. Is that what we are wanting now? I can push a quick update, if not, just let me know!

[nbenmoody]

Okay, @rabble confirmed this was good, in Slack.

However, I did find some errors on these scripts when they are running:

the grow_fedi script is hitting this frequently:

Jan 11 15:30:01 sync fedi_sync[579609]: No pubkeys found in matching_nhex.txt.
Jan 11 16:00:01 sync CRON[579862]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 11 16:00:01 sync grow_fedi_nhex[579865]: Traceback (most recent call last):
Jan 11 16:00:01 sync grow_fedi_nhex[579865]:   File "/home/admin/nostr-sync-scripts/grow_fedi_nhex.py", line 244, in <module>
Jan 11 16:00:01 sync grow_fedi_nhex[579865]:     main()
Jan 11 16:00:01 sync grow_fedi_nhex[579865]:   File "/home/admin/nostr-sync-scripts/grow_fedi_nhex.py", line 236, in main
Jan 11 16:00:01 sync grow_fedi_nhex[579865]:     blocklist = load_blocklist("_unified_tier0_blocklist.csv")
Jan 11 16:00:01 sync grow_fedi_nhex[579865]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 11 16:00:01 sync grow_fedi_nhex[579865]:   File "/home/admin/nostr-sync-scripts/grow_fedi_nhex.py", line 29, in load_blocklist
Jan 11 16:00:01 sync grow_fedi_nhex[579865]:     with open(file_path, newline='') as csvfile:
Jan 11 16:00:01 sync grow_fedi_nhex[579865]:          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 11 16:00:01 sync grow_fedi_nhex[579865]: FileNotFoundError: [Errno 2] No such file or directory: '_unified_tier0_blocklist.csv'
Jan 11 16:30:01 sync CRON[580372]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/fedi_sync.py 2>&1 | logger -t fedi_sync)

However, it also seems to be alright sometimes?

Jan 11 16:00:01 sync grow_fedi_nhex[579865]:          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 11 16:00:01 sync grow_fedi_nhex[579865]: FileNotFoundError: [Errno 2] No such file or directory: '_unified_tier0_blocklist.csv'
Jan 11 17:00:01 sync CRON[580463]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 11 18:00:01 sync CRON[581770]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 11 19:00:01 sync CRON[582278]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 11 20:00:01 sync CRON[583341]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 11 21:00:01 sync CRON[583848]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 11 22:00:01 sync CRON[584529]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 11 23:00:01 sync CRON[585999]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 12 00:00:01 sync CRON[586625]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)
Jan 12 01:00:01 sync CRON[587222]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/grow_fedi_nhex.py 2>&1 | logger -t grow_fedi_nhex)

[nbenmoody]

...and then the fedi_sync script is tossing this out:

Jan 11 16:30:01 sync fedi_sync[580375]: No pubkeys found in matching_nhex.txt.

The news_sync script appears to be working alright:

Jan 12 00:45:01 sync CRON[587117]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/news_sync.py 2>&1 | logger -t news_sync)
Jan 12 00:45:03 sync news_sync[587121]: Syncing Nostr users
Jan 12 00:45:03 sync news_sync[587121]: - Number of native Nostr users fetched: 10
Jan 12 00:45:03 sync news_sync[587121]: - Number of notes copied to news.nos.social: 0
Jan 12 00:45:03 sync news_sync[587121]: - Duration: 0.4 seconds
Jan 12 00:45:03 sync news_sync[587121]: Syncing Fediverse users
Jan 12 00:45:03 sync news_sync[587121]: - Number of Fediverse users fetched: 8
Jan 12 00:45:03 sync news_sync[587121]: - Number of notes copied to news.nos.social: 0
Jan 12 00:45:03 sync news_sync[587121]: - Duration: 1.9 seconds
Jan 12 00:45:03 sync news_sync[587121]: Timestamp saved successfully.
Jan 12 01:45:01 sync CRON[587658]: (root) CMD (/home/admin/sync_venv/bin/activate && /home/admin/sync_venv/bin/python /home/admin/nostr-sync-scripts/news_sync.py 2>&1 | logger -t news_sync)
Jan 12 01:45:03 sync news_sync[587663]: Syncing Nostr users
Jan 12 01:45:03 sync news_sync[587663]: - Number of native Nostr users fetched: 10
Jan 12 01:45:03 sync news_sync[587663]: - Number of notes copied to news.nos.social: 0
Jan 12 01:45:03 sync news_sync[587663]: - Duration: 0.7 seconds
Jan 12 01:45:03 sync news_sync[587663]: Syncing Fediverse users
Jan 12 01:45:03 sync news_sync[587663]: - Number of Fediverse users fetched: 8
Jan 12 01:45:03 sync news_sync[587663]: - Number of notes copied to news.nos.social: 0
Jan 12 01:45:03 sync news_sync[587663]: - Duration: 1.4 seconds
Jan 12 01:45:03 sync news_sync[587663]: Timestamp saved successfully.

[mplorentz]

I am tracking the above issues in https://github.com/verse-pbc/issues/issues/18