Changes to work with latest environment and libraries (#630)

* Remove jinja2.ext.with_ .

It's a no-op since 2.9 and removed since 3.1 .
https://jinja.palletsprojects.com/en/3.0.x/extensions/#with-statement

* Lock aiohttp version to 3.7.4 since 3.8.0+ introduced ill behavior.
aio-libs/aiohttp#5572

* Use misaka for markdown rendering since hoedown is incompatible with latest Python.

* Fix aiomongo to work with pymongo 3.5 .

* Unlimit Python version and limit MongoDB version.

* Allow range for aiohttp version.

* Add Python 3.9 to test.yml .

* Bump Python version requirement to 3.6 since cffi won't install easily on 3.5 .

* Update README.md .

* Use escape and Markup from markupsafe since it's removed from jinja2 since 3.1.0 .

Author: iceboy233

.github/workflows/test.yml

@@ -74,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.5, 3.6, 3.7, 3.8]
+        python-version: [3.6, 3.7, 3.8, 3.9]
     services:
       rabbitmq:
         image: rabbitmq
@@ -115,4 +115,3 @@ jobs:
 
       - name: Unit tests
         run: python -m unittest
-        continue-on-error: ${{ matrix.python-version == '3.8' }}

README.md

@@ -43,8 +43,8 @@ There is pre-built docker package at `vijos/vj4`. This is maintained by [@moesoh
 
 #### Prerequisites
 
-* [Python 3.5.3+, <3.8](https://www.python.org/downloads/)
-* [MongoDB 3.0+](https://docs.mongodb.org/manual/installation/)
+* [Python 3.6+](https://www.python.org/downloads/)
+* [MongoDB 3.0+, <3.6](https://docs.mongodb.org/manual/installation/)
 * [Node.js 10.0+](https://nodejs.org/en/download/package-manager/)
 * [RabbitMQ](http://www.rabbitmq.com/)
 
@@ -61,10 +61,10 @@ You don't need root privilege to run `yarn`. It installs stuffs in the project d
 
 You may want to use [tuna](https://pypi.tuna.tsinghua.edu.cn/) if you are in China.
 
-Some requirements may need `Python.h`. In Debian/Ubuntu you can use
+Some requirements may need `Python.h` and `ffi.h`. In Debian/Ubuntu you can use
 
 ```bash
-apt install python3-dev
+apt install python3-dev libffi-dev
 ```
 
 to resolve this.

requirements.txt

@@ -1,7 +1,7 @@
-aiohttp>=3
+aiohttp>=3.6.0,<3.8.0
 jinja2>=2.9.0
 sockjs>=0.7.0,<0.10.0
-hoedown
+misaka
 accept
 aioamqp
 git+https://github.com/LukeXuan/aiosmtplib # custom aiosmtlib with ssl support
@@ -11,6 +11,6 @@ httpagentparser
 geoip2
 GitPython
 PyYAML
-aiomongo
+git+https://github.com/iceboy233/aiomongo
 yarl
 aiohttp-sentry

vj4/template.py

@@ -25,7 +25,6 @@ class Environment(jinja2.Environment):
   def __init__(self):
     super(Environment, self).__init__(
         loader=jinja2.FileSystemLoader(path.join(path.dirname(__file__), 'ui/templates')),
-        extensions=[jinja2.ext.with_],
         auto_reload=options.debug,
         autoescape=True,
         trim_blocks=True,

vj4/util/misc.py

@@ -1,32 +1,32 @@
 import base64
 import hashlib
-import hoedown
 import jinja2
 import markupsafe
+import misaka
 import re
 from urllib import parse
 
 from vj4.util import options
 
 
-MARKDOWN_EXTENSIONS = (hoedown.EXT_TABLES |  # Parse PHP-Markdown style tables.
-                       hoedown.EXT_FENCED_CODE |  # Parse fenced code blocks.
-                       hoedown.EXT_AUTOLINK |  # Automatically turn safe URLs into links.
-                       hoedown.EXT_NO_INTRA_EMPHASIS |  # Disable emphasis_between_words.
-                       hoedown.EXT_MATH |  # Parse TeX $$math$$ syntax, Kramdown style.
-                       hoedown.EXT_SPACE_HEADERS |  # Require a space after '#' in headers.
-                       hoedown.EXT_MATH_EXPLICIT |  # Instead of guessing by context, parse $inline math$ and $$always block math$$ (requires EXT_MATH).
-                       hoedown.EXT_DISABLE_INDENTED_CODE)  # Don't parse indented code blocks.
-MARKDOWN_RENDER_FLAGS = (hoedown.HTML_ESCAPE |  # Escape all HTML.
-                         hoedown.HTML_HARD_WRAP)  # Render each linebreak as <br>.
+MARKDOWN_EXTENSIONS = (misaka.EXT_TABLES |  # Parse PHP-Markdown style tables.
+                       misaka.EXT_FENCED_CODE |  # Parse fenced code blocks.
+                       misaka.EXT_AUTOLINK |  # Automatically turn safe URLs into links.
+                       misaka.EXT_NO_INTRA_EMPHASIS |  # Disable emphasis_between_words.
+                       misaka.EXT_MATH |  # Parse TeX $$math$$ syntax, Kramdown style.
+                       misaka.EXT_SPACE_HEADERS |  # Require a space after '#' in headers.
+                       misaka.EXT_MATH_EXPLICIT |  # Instead of guessing by context, parse $inline math$ and $$always block math$$ (requires EXT_MATH).
+                       misaka.EXT_DISABLE_INDENTED_CODE)  # Don't parse indented code blocks.
+MARKDOWN_RENDER_FLAGS = (misaka.HTML_ESCAPE |  # Escape all HTML.
+                         misaka.HTML_HARD_WRAP)  # Render each linebreak as <br>.
 
 
 FS_RE = re.compile(r'\(vijos\:\/\/fs\/([0-9a-f]{40,})\)')
 
 
 def nl2br(text):
-  markup = jinja2.escape(text)
-  return jinja2.Markup('<br>'.join(markup.split('\n')))
+  markup = markupsafe.escape(text)
+  return markupsafe.Markup('<br>'.join(markup.split('\n')))
 
 
 def fs_replace(m):
@@ -36,7 +36,7 @@ def fs_replace(m):
 
 def markdown(text):
   text = FS_RE.sub(fs_replace, text)
-  return markupsafe.Markup(hoedown.html(
+  return markupsafe.Markup(misaka.html(
       text, extensions=MARKDOWN_EXTENSIONS, render_flags=MARKDOWN_RENDER_FLAGS))