Use python3 instead of php.

[bernhardreiter]

Using python3 to serve the manager would get rid of a major dependency ("php").
Given that large frameworks always has defects it is desireable to remove php
as python3 is already coming with intelqm.

Technically: Should be a reasonable effort because most stuff happens in Javascript anyway.
Would need a python3 webframe work to move, which needs to be selected.
(E.g. from stuff like Django to flask and bottle.)

[aaronkaplan]

On 30 Jun 2016, at 15:51, Bernhard E. Reiter [email protected] wrote:

Using python3 to serve the manager would get rid of a major dependency ("php").
Given that large frameworks always has defects it is desireable to remove php
as python3 is already coming with intelqm.

Technically: Should be a reasonable effort because most stuff happens in Javascript anyway.
Would need a python3 webframe work to move, which needs to be selected.
(E.g. from stuff like Django to flask and bottle.)

For whatever it's worth, I prefer small light frameworks such as flask.
Django is quite an overkill for this type of task IMHO.

My 2 cents,
a.

[SYNchroACK]

@aaronkaplan you're 100% right IMHO.

+1

[bernhardreiter]

The decision for a web framework will have an effect on several intelmq components, so it should be taken centrally. I've created certtools/intelmq#708 for it.

[bernhardreiter]

When fixing #111 is makes sense to also switch from php to python3. As https://github.com/Intevation/intelmq-fody-backend is working nicely with https://www.hug.rest/
this is the first choice for a micro-framework to use as there is some synergy. If there is a need to switch to a different one like falcon it would be much less efforts.

Rough plan:

• Understand the 280 lines of php and reimplement in python3/hug.
• Make sure the template.org dynamic is kept (there is not much)
• update and test installation instructions with web server
• update debian packages,
• test installation

Is is allowed to change the endpoints and their parameters if researching shows that something can be done better, because the serving and html pages and javascript parts are delivered together and considered internal implementation detail.

[aaronkaplan]

I don't quite agree and I am also not sure if it fits with future architectures.
Please also take a look at fastAPI.

[bernhardreiter]

@aaronkaplan thanks for commenting!

Can you be more specific about which parts you do not agree?
So far I understand that the choice of the python3 microframework maybe suboptimal.

Choice of microframework hug versus fastAPI

We'll take a look at FastAPI.

Explanation of the current suggestion hug:
We did no new reevaluation of the progress of Python JSON Https service microframeworks, because
a) It is not the main part of the work and not a strong dependency
b) There is some synergy, which makes the work less expensive and both easier to operate when using hug for both fody-backend and intelmq-manager (backend). Otherwise we look at replacing fody-backend microframework as well.
c) there are no special highspeed requirements of the intelmq-manager backend, the performance problem is more with how intelmq itself is able to come up with the necessary infos.

Small hint: fastapi itself lists hug as a major influence and good step forward, so it is not bad in the eyes of the fastapi developers. :)
#80 (comment)
(The question will be, are the new methods, specially some of the schema works in the past was not always fitting micro approaches, especially not as we are not trying to do an api in the string sence, but an internal data exchange.)

[bernhardreiter]

In certtools/intelmq#708 (comment) there is a look about the strength and weaknesses of FastAPI versus hug as general web framework choice.

In conclusion FastAPI has the potential to surpass hug, because it takes more modern approaches. Currently it is not as mature and has some unneeded complexity for an internal interface (which is the main topic in this issue).

So there is some more work in using FastAPI, here for the deployment:

Deployment

FastAPI is newer, it needs an ASGI server and those again need an TLS capable HTTP server/proxy. Some of the dependencies are not considered production stable (yet).

To check the packaging level in Debian (as example):

• Not packaged: https://www.starlette.io/, fastapi, traefik (a recommended HTTPS proxy)
• Packaged https://tracker.debian.org/pkg/pydantic, https://tracker.debian.org/pkg/python-uvicorn

For hug there are packages (though python-hug is not yet in Debian Stable):

• https://tracker.debian.org/pkg/python-hug
• https://tracker.debian.org/pkg/python-falcon

Conclusion

The microframework choice is not the main focus of this issue. Both FastAPI and hug would work.
FastAPI has more future potential, but for the time being causes more work on people that would like to deploy it.

We from Intevation would be fine to do either variant, so it is up to you @aaronkaplan, @wagner-certat and our customers to decide what you prefer.

[pettai]

I don't quite agree and I am also not sure if it fits with future architectures.

@aaronkaplan could you please be more specific on why hug et al isn't fit?

[bernhardreiter]

Sunet contracted us (from Intevation) do resolve this issue, timeframe is within two month.

[ghost]

The main problem I see is that we have multiple Python-based web applications in IntelMQ. Yet it's fody and and webinput, now the manager gets converted too. That will make the set-up procedure much harder , it's already a pain with only fody and webinput. Maybe the manager could act as an proxy/relay here to keep/make it manageable.

[bernhardreiter]

From what I've read and the existance of the issue itself, we agree that moving away from php is a good thing. :)

Proliferation of python3 microframeworks is a possible negative aspect. However gaining experience
is a plus.

If we move to hug, it is one technology less and we will find something to see if we keep the urls between the manager and fody and other apis separated.

Moving to FastAPI would give us more experience with a potentially raising microserving technology,
so we can make a good recommendation for certtools/intelmq#708 one day.
(Here we also find something to keep the URL parts separated.)

@aaronkaplan What is your opinion now, that you've read the various cons and pros?

[bernhard-herzog]

I'm going to start working on this. In order to keep the effort focused
the broad strategy is to limit the change to replacing PHP with Python
and keep everything else the same as much as possible.

I'm going to to use hug as framework for reasons already mentioned in
other comments in this issues: we (Intevation) already use it in fody
and it's easy to install on at least one platform we care about
(Ubuntu). I'm going to keep most of the code independent of the
framework so that it should be reasonably easy to use a different
framework at some later point.

[aaronkaplan]

Hi @bernhard-herzog , @bernhardreiter ,

so... I'd like to schedule a quick call with you regarding the IntelMQ 3.0 architecture.
First of all, to get your feedback and secondly on how you can integrate that with hug.

ok?

In principle, for the next version I don't care which RESTful API framework you want to use.
The only thing is that I would like to make sure we support the same API endpoints as future IntelMQ bots.
So, since this is a longer discussion , I would recommend to sync via a jitsi (or whatever) call.

Could you please mail me a few suggestions when you would have time (I can only do afternoons Mo-Thursday, FR the whole day).

Thanks + Best,
a.

[bernhard-herzog]

I've now actually started on this with a bit of prototyping of a
solution. So far I've got a simple hug-based API that I use as a
replacement for controller.php. The HTML for the management interface is
still served by PHP but all API requests that would go to controller.php
now go to my API prototype. That prototype can only handle a few simple
queries, such as ?scope=check and ?scope=queues but those are handled
all the way via sudo to intelmqctl and back to the JS in the browser.

[bernhardreiter]

About the IntelMQ 3 architecture and the small mumble conf: We've talked about https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architecture-3.0.md and it does not affect this point, as the timeline is much longer and it is all in the idea and feedback collection phase. So all green light for getting this done quickly and with hug3. :)

@bernhard-herzog plans to have our prototype in a branch within the next days (when get around doing it).

[aaronkaplan]

On 21.04.2020, at 12:49, Bernhard E. Reiter ***@***.***> wrote: About the IntelMQ 3 architecture and the small mumble conf: We've talked about https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architecture-3.0.md and it does not affect this point, as the timeline is much longer and it is all in the idea and feedback collection phase. So all green light for getting this done quickly and with hug3. :) @bernhard-herzog plans to have our prototype in a branch within the next days (when get around doing it).

confirmed.

[bernhard-herzog]

I've just pushed the first part of my implementation to
https://github.com/Intevation/intelmq-manager/tree/python-backend

[bernhardreiter]

technical considerations

Static parts for HTML, javascript, css for the single page web application could also be served outside of hug3 by a webserver, and only optionally like https://github.com/hugapi/hug/blob/develop/examples/static_serve.py . So preparation of the spa could happen as pre-procession. However I'm not sure if we get this done as part of this issue.

[bernhard-herzog]

Python-backend is feature complete

I've finished the port of the PHP backend and can now run
IntelMQ-Manager without PHP.

The main thing left to do is installation and setup. This includes the
usual python packaging things like setup.py but also corresponding
documentation. I plan to do this this week, at least enough of it so
that other people can more easily test this.

One thing that is missing is a timeout for the intelmqctl subprocesses.
This is not easy to implement due to sudo being involved, and doesn't
seem to work in the PHP version either (#195).

[ghost]

Wow, that is great news!

Could the timeout maybe easier be implemented in intelmqctl itself?

[bernhard-herzog]

Could the timeout maybe easier be implemented in intelmqctl itself?

I've replied to this in #195 (comment)

[bernhard-herzog]

I've updated docs/INSTALL-DEV in our python-backend branch so that other developers can easily try it out.

[bernhard-herzog]

Some of next steps:

Feed-back

Has anybody tried it? Did it work? What do we need to change?

Code organisation

The Python code is in the intelmq_manager subdirectory. The PHP code
in intelmq-manager has been removed, but the static files, like style
sheets, images, JavaScript files, etc. is still there. This at least
looks a little strange (only one character difference in the names, the
'_' was necessary so that the directory name could be a package name).

Possible solutions:

• rename intelmq-manager to static

• rename intelmq-manager to intelmq_manager/static

Documentation

In particular, the installation instructions obviously have to be
updated. The current instructions cover quite a range of systems. Do we
still support all of them, particularly the older ones?

I've developed the python port on Ubuntu 18.04LTS, so the new
dependencies, hug and mako, don't have to be particularly new (the
minimum versions given in the setup.py file are those of 18.04).
However, I don't know whether older versions of hug would also work.
Older versions of mako should be fine as the templates only use features
that have been in mako for a very long time

[ghost]

Has anybody tried it? Did it work? What do we need to change?

Works for me! I tried it on our development instance and haven't seen any bug yet :)

Possible solutions:

* rename `intelmq-manager` to `static`

* rename `intelmq-manager` to `intelmq_manager/static`

Do you prefer one over the other? I'm fine with both variants.

In particular, the installation instructions obviously have to be
updated. The current instructions cover quite a range of systems. Do we
still support all of them, particularly the older ones?

No, as 18.04 works, that's fine. For example we do not need to support Debian 8 and Ubuntu <= 16.04. I can add instructions for Debian, CentOS and openSUSE myself.

I've developed the python port on Ubuntu 18.04LTS, so the new
dependencies, hug and mako, don't have to be particularly new (the
minimum versions given in the setup.py file are those of 18.04).

👍

[bernhardreiter]

Personally my tendency is renaming to 'static' to separate the python code module more clearly.

[bernhard-herzog]

Has anybody tried it? Did it work? What do we need to change?

Works for me! I tried it on our development instance and haven't seen any bug yet :)

Great!

Possible solutions:

* rename `intelmq-manager` to `static`

* rename `intelmq-manager` to `intelmq_manager/static`

Do you prefer one over the other? I'm fine with both variants.

After looking into this a bit, I went with intelmq_manager/static
because that fits Python's setuptools/distutils better. Putting data
files into the Python package is the way these tools prefer.

[bernhardreiter]

My test was successful, I did an overview test, covering:

• clicking all menu items, see useful results
• start the bot-net once, saw some updates, stopped it again
• change some bot positions in the network diagram, saving okay
• changed one bot configuration -> saved

hints for improvement and packaging

• make sure in documentation that 'INTELMQ_MANAGER_CONTROLLER_CMD is not used anymore. E.g remove it from ./intelmq/bin/intelmqctl.py as output of intelmqctl debug --get-environment-variables
• make sure that positions can be saved

positions.conf

This needs a writable manager/positions.conf file, even if it just an empty one and the documentation

# if you use the intelmq manager (adapt the webservers' group if needed):
chown intelmq.www-data /opt/intelmq/etc/*.conf

from intelmq installation instructions https://github.com/certtools/intelmq/blob/develop/docs/Developers-Guide.md#development-environment may not be enough to ensure this, so packageing might. Maybe a solution is to have the manager be able to create the file if it does not exist (which we can put in a new issue).

[bernhard-herzog]

The actual porting is done. What's missing is documentation and
packaging updates. It seems to me that that might be easier after this
is merged, because then others can more easily contribute. I've
therefore created a pull request now.

[bernhardreiter]

If the plan is to release IntelMQ Manager 2.3.0 with the new python backend module (https://github.com/certtools/intelmq-api)
and because the documentation and packaging there is done (according to the message). It is only the release that is missing, isn't it?

(So basically we could close here, because Issues are closed when they are in the code and not when they are release?)

[ghost]

Yes, we can close here. Before doing the release just we need more testing

[bernhardreiter]

What kind of testing would be needed? (Just asking, if it is known, someone could estimate if they could help.)

[ghost]

What kind of testing would be needed? (Just asking, if it is known, someone could estimate if they could help.)

Just using it in practice to check if there are some glitches. We need to do that internally and I intend to write a mail to the mailinglists as well to announce this major change (and ask for testing it & feedback).