Gekko protected with username+password

[rcoenen]

How do I get Japonicus to authenticate against Gekko's api?

I am getting

http://localhost:3000/api/scansets
{}
Traceback (most recent call last):
  File "./japonicus-run", line 13, in <module>
    japonicus.JaponicusSession(evaluation.gekko, settings, options)
  File "/home/ubuntu/gekko/japonicus/japonicus/japonicus.py", line 161, in __init__
    web=self.web_server
  File "/home/ubuntu/gekko/japonicus/japonicus/evolution_generations.py", line 139, in Generations
    conf
  File "/home/ubuntu/gekko/japonicus/japonicus/evolution_generations.py", line 71, in grabDatasets
    minDays=conf.backtest.deltaDays
  File "/home/ubuntu/gekko/japonicus/evaluation/gekko/dataset.py", line 17, in selectCandlestickData
    DataSetPack = getAllScanset(GekkoURL)
  File "/home/ubuntu/gekko/japonicus/evaluation/gekko/dataset.py", line 10, in getAllScanset
    return RESP['datasets']
TypeError: 'bool' object is not subscriptable

Accessing http://localhost:3000/api/scansets yields a 401 Unauthorized
I have tried to get Gekko to skip authentication for localhost but that is not so easy to get done.
Is there a way to get Japonicus to authenticate with username + password?

[Gab0]

Hey, I've never seen this authentication thing going on gekko... Isn't this changeable on gekko config?
japonicus cannot authenticate those api calls...

[rcoenen]

I simply added my own basic authentication using the http-auth Node module, because I am running Gekko on an EC2 instance and I do not want the bot to be open and unprotected...

[rcoenen]

general remark: setting up Gekko + Japonicus on EC2 and having some security (username + password) is extremely complicated, fiddly and undocumented. Be warned.

[Gab0]

Did you manage to make it work? I didn't even know about it lol.
Maybe if you have iptables or something blocking every request except for the ones from your IP it's easier...

[rcoenen]

Not yet. I can't really figure out how to apply basic auth when incoming request is external IP but skip authentication when it's local. This solution could work since have Gekko and Japonicus on the sand host... but it's not ideal even if I got it working. Best would bet to get Japonicus to authenticate with username & password

[Gab0]

Is the auth user/pass sent inside the json data of each request to gekko? Maybe you can tweak japonicus/evaluation/gekko/API.py to make it happen.

[rcoenen]

yep!!! ok my first ever piece of Python... I added , auth=(user, passw) to all request.get / request.post in japonicus/evaluation/gekko/API.py total hack with u/p hardcoded in but.. for now... works!

[Gab0]

Cool bro...
Its {"auth": (user, passw)} in the json or with separated keys? I might add it to Settings and stuff.

[rcoenen]

ehrr it's just two string values, one string named user containing the username and another string called passw containing the password

[IAMtheIAM]

Hey @rcoenen Can you share your API.py file (with password removed) so I can get an idea of how you did this. I'm trying to do the exact same thing!

general remark: setting up Gekko + Japonicus on EC2 and having some security (username + password) is extremely complicated, fiddly and undocumented. Be warned.

I agree completely. it was super complicated and difficult to get it running on a VPS. I was able to get it running with nginx using docker-letsencrypt-nginx-proxy-companion, which made it even more hard to configure. But, now that it's done, its nice and automatically configures the SSL certificates for any new docker container I start which will be exposing ports to the public. I recommend it, even with a steep learning curve (mainly due to nginx).

[rcoenen]

@IAMtheIAM Yep nginx does not help, and what also doe snot help: OpenSSL does not want to generate self-signed certificates for bare EC2 instances so... ugh

Anyway, the changes I made to API.py are super simple, see screenshot below of the complete DIFF. Also attached my file renamed to API.py.txt as it won't allow me to attach a .py file directly

API.py.txt

[IAMtheIAM]

Thanks! @rcoenen

[IAMtheIAM]

Hey @rcoenen @Gab0 I got the Basic auth working, but I'm having an issue with Nginx Timeout. Exactly 60 seconds after I send my backtest request from my local machine to my server running Gekko, I get 504 Gateway Time-out

I tried adding inside the nginx.conf file the following:

  proxy_connect_timeout       8600s;
  proxy_send_timeout          8600s;
  proxy_read_timeout          8600s;
  send_timeout                8600s;

However it still times out. Do you know which setting to change to get the request to complete without timing out?

[rcoenen]

Sorry I have up on Nginx because reasons above (no SSL in free tier EC2, so no added benefit over dumb plain text authentication without Nginx in the way) I ended up spending all my time on DevOps stuff that I'm not really interested in rather than learning to trade crypto and build strategies so... Sorry can't help

…

On Wed, Jul 10, 2019, 12:02 IAMtheIAM ***@***.***> wrote: Hey @rcoenen <https://github.com/rcoenen> @Gab0 <https://github.com/Gab0> I got the Basic auth working, but I'm having an issue with Nginx Timeout. Exactly 60 seconds after I send my backtest request to my server running Gekko, I get 504 Gateway Time-out I tried adding inside the nginx.conf file the following: proxy_connect_timeout 8600s; proxy_send_timeout 8600s; proxy_read_timeout 8600s; send_timeout 8600s; However it still times out. Do you know which setting to change to get the request to complete without timing out? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#184?email_source=notifications&email_token=AAFYAKDU3D357OSZKVDOG7TP6YBYVA5CNFSM4H5ME44KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZT6BZY#issuecomment-510124263>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFYAKDPVE7DLDIOQXP66W3P6YBYVANCNFSM4H5ME44A> .

[IAMtheIAM]

Actually I just figured it out. I was adding the config to the wrong file, and it was being overwritten by another file. So now it works fine :-D

If you use LetsEncrypt, you can get free SSL with any host.
But yea, it takes time to learn how to configure nginx. Actually you could configure LetsEncrypt SSL without any nginx, just install certbot and it will manage SSL config and renewal for you automatically (it's the recommended method for installing SSL from letsEncrypt

https://certbot.eff.org/
https://letsencrypt.org/getting-started/

[IAMtheIAM]

Cool bro...
Its {"auth": (user, passw)} in the json or with separated keys? I might add it to Settings and stuff.

@Gab0 If we were to integrate the auth=() into the codebase, how would you recommend configuration? which .toml file would you suggest putting the username/password into and how would we pass that into API.py It would be useful because I think more people will start using the Basic Authentication for Gekko.

[Gab0]

@IAMtheIAM That would be great... at _Global.toml would be a good place, but I guess it would involve a bit more than adding the option to the setting file to make everything work cleanly on auth. You may try though, as I will be able to work on the next version after july 25th only.