No logging handlers configured

[garrettr]

In 59ba7aa (now in master and develop), I added some support for logging with the Python standard library logging module. I also added code to set up handlers (places where logs go) and attach them to the loggers, which was in config.py. Unfortunately, I appear to have accidentally forgotten to copy these changes to example_config.py, which means they were not committed to the repository. That means that no logging is recorded anywhere. Note that in development, Flask is in debug mode (app.run(debug=True)), so the Flask app's logs are printed to stderr. Other logging (done with the log objects defined at the top of, currently, source.py and store.py) is not being recorded. debug == False in production, so no logs are recorded in production.

This is a minor issue because there is very little logging being performed at the moment, and it is mostly for extremely rare/unlikely cases (such as duplicate codename generation). Nonetheless, this needs to be fixed (and hotfixed) because it will make debugging legitimate errors more difficult, especially in production.

[garrettr]

There are some other issues related to logging that should be addressed here as well.

1. Python's logging module is thread-safe, but not multiple process safe. Since we currently use apache mpm worker, we have multiple Python interpreters running the application concurrently. This could cause problems with contention, possibly leading to crashes or log file corruption.
   a. We could limit the number of worker processes to 1 (currently set to 2 in {document,app}.apache2.conf). Since we do not expect high load in general, this may be sufficient as a temporary fix.
   b. There are lots of solutions for this problem, all of which are based around running a central log server that receives logs from all running processes. Example
2. Since logs could contain sensitive information, they need to rotated (and the old copies securely deleted) as quickly as possible, to minimize the chance of such data being compromised in a server seizure while still maintaining utility for debugging issues when they arise. I suggest a window of 8-24 hours. This can be easily accomplished with logrotate.

[kyleolivo]

Looking into this for the hackathon.

[garrettr]

For #310, here's resolutions for the additional issues raised in my above comment:

1. For process safety, use SysLogHandler. There's a commented example in Logging hotfix #310's config.py.
   • another, similar, approach. primary benefit is being able to set the app name in syslog, which you can't do with SysLogHandler in Python 2.7.
2. Log rotation and deletion is being tracked in Change default apache logrotate config #304

@dolanjs what do you think?

[garrettr]

Fixed in master with #310.