Unable to create Sender. Could not access custom SpeckleServer.

[htlcnn]

I'm running a custom SpeckleServer on my virtual machine. I'm able to login to Web UI and create Projects, Streams there.
I installed SpeckleRevit and use it with Revit 2018. I was able to create Sender and sent data to hestia server.
For my custom SpeckleServer, I logged in successfully in SpeckleRevit, but when I added a new sender and choose that server, SpeckleRevit said Could not access that server (is it online?) or no server Selection.
Is there any specific configuration to be able to send data to custom SpeckleServer?

[didimitrie]

Hello @htlcnn , i think this might be related to speckleworks/SpeckleUi#7 - is your local server https?

[htlcnn]

@didimitrie no, my server is not https. Is https required?

[didimitrie]

I'm afraid so - we will try to remove this limitation, or at least allow for localhost:xxxx urls.

[htlcnn]

Thanks. I'll add https to my server using Let's encrypt self-signed certificate.

[htlcnn]

@didimitrie I've added a self-signed certificate to my SpeckleServer. This time SpeckleRevit fails to find the server at the first step. It says There seems to be no speckle server there:

Here's api response from https://10.15.0.30/api (json beautified):

{
    "isSpeckleServer": true,
    "serverName": "BIM UDIC SpeckleServer",
    "version": "v1.9.8",
    "api": {
        "projects": [{
            "method": "POST",
            "route": "https://10.15.0.30/api/projects"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/projects"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/projects/admin"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/projects/:projectId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId/addstream/:streamId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/projects/:projectId/removestream/:streamId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId/adduser/:userId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/projects/:projectId/removeuser/:userId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId/upgradeuser/:userId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId/downgradeuser/:userId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/projects/:projectId"
        }],
        "clients": [{
            "method": "POST",
            "route": "https://10.15.0.30/api/clients"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/clients"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/clients/:clientId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/clients/:clientId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/clients/:clientId"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/clients"
        }],
        "streams": [{
            "method": "POST",
            "route": "https://10.15.0.30/api/streams"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/admin"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/streams/:streamId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/streams/:streamId"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/streams/:streamId/clone"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/diff/:otherId"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/delta/:otherId"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/streams/:streamId/delta"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/objects"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/clients"
        }],
        "accounts": [{
            "method": "POST",
            "route": "https://10.15.0.30/api/accounts/register"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/accounts/login"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/accounts"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/accounts/admin"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/accounts"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/accounts/:userId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/accounts/:userId"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/accounts/search"
        }],
        "comments": [{
            "method": "GET",
            "route": "https://10.15.0.30/api/comments"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/comments/assigned"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/comments/:resourceType/:resourceId"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/comments/:resourceType/:resourceId"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/comments/:commentId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/comments/:commentId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/comments/:commentId"
        }],
        "objects": [{
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/objects"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/objects"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/objects/derive"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/objects/:objectId"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/objects/getbulk/"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/objects/:objectId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/objects/:objectId/properties/"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/objects/:objectId"
        }]
    },
    "plugins": [{
        "name": "Speckle Admin",
        "version": "1.0.3",
        "desc": "YEAH! Speckle has a frontend yo!",
        "serveFrom": "/",
        "serveSource": "plugins/SpeckleAdmin/dist",
        "author": "Speckle Project Contributors",
        "contact": "[email protected]",
        "homepage": "https://speckle.works",
        "git": "https://github.com/speckleworks/SpeckleAdmin",
        "sourceDir": "plugins/SpeckleAdmin",
        "canonicalUrl": "https://10.15.0.30/"
    }, {
        "name": "Sample Plugin",
        "version": "0.1.0",
        "desc": "This plugin tests plugins",
        "serveFrom": "/sample-plugin",
        "author": "DAS",
        "contact": "[email protected]",
        "homepage": "https://dimitrie.org",
        "git": "https://github.com/speckleworks/SpeckleViewer",
        "sourceDir": "plugins/speckle-sample-plugin",
        "canonicalUrl": "https://10.15.0.30/sample-plugin"
    }],
    "jnMask": "######-##"
}

[didimitrie]

Sometimes the debouncer that sends the test request is out of sync, and causes the button to be disabled. If you add a space, or add /api does it work?

If not, the response seems correct, and it probably means there's a problem with self signed certs - I would debug that by capturing the requests with fiddler and seeing if any problems are evident there...

[htlcnn]

Adding a space or /api doesn't work. I'll try using fiddler. Thanks for the hint.

[htlcnn]

This is the first time I used fiddler. SpeckleRevit still fails until I enable HTTPS debugging in fiddle and click Yes in a popup that seems to ask about accepting self-signed certificate (sorry for my bad glance). Now I'm able to add my speckle server. But I still can not add new Sender. It still says Could not access to that server.
One thing I notice is that the redirect URL from SpeckleRevit is different from SpeckleAdmin (web app).
For SpeckleRevit, redirect URL is always https://localhost:5050. That doesn't exist in my .env.
For SpeckleAdmin, redirect URL is https://10.15.0.30/#/signin/callback

[didimitrie]

https://localhost:5050
Are you sure it's https? It should be http. That's where the sign in popup redirects to with the api token when using the account manager - it should be different than the web ui.

Can you post a screenshot of the errors/logs form the console from the revit plugin? To open it click on the ck in the title; it should launch chromium's dev tools. I still believe this is to do with the self signed cert not working properly, and thus requests getting blocked...

[htlcnn]

Yes it was http://localhost:5050.
Here's the screenshot from console when I'm not logged in:

Screenshot after I clicked CREATE SENDER:

Also a window pops up after I logged in:

[htlcnn]

This is fiddle's popup. SpeckleRevit's Add New Server button lights up only after I click Yes.

[didimitrie]

Simplest solution to get stuff going: deploy the server on DigitalOcean on a $5/month VM, you'll get some free credit when you start to keep it going for 5 months i think :)

Otherwise, more complicated answer: seems like there might be something else hogging port 5050: https://stackoverflow.com/questions/29714055/system-net-httplistenerexception-failed-to-listen-on-prefix-http-localhost8

based on the last error...

[didimitrie]

Another quick question (should've asked first!): does it work with the public test server? If so, then I can most definitively point the finger at the self signed SSL cert.

I'll close the issue for now, as it's not a problem with the revit client per se, but let's keep talking here, and maybe we figure your config out...

[htlcnn]

Yes, choosing hestia works.

I have DO instances up and running some production apps. I just want to test on my VM first, and also see if sending data and loading viewer would be faster than using hestia server.

The image saying about port 5050 conflict was my mistake. I added the same server after it was added successfully into SpeckleRevit. I deleted all servers and added it again (still had to keep fiddler on and accept certificate manually).

This is my overall Server config:

• nginx as reverse proxy, with self-signed cert. conf file:

server {
    listen 80;
    server_name 10.15.0.30;
    location /.well-known {
            alias /home/speckle/.well-known;
    }

    location / {
        # redirect to https
        return 301 https://$host$request_uri;
    }
}

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

# theoretically this can be a range of severs to facilitate 
# horizontal scaling.
upstream speckleserver {
    server 127.0.0.1:3000;
}

server {
    # And: as far as I know websockets are not supported by http2 yet.
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;

    server_name 10.15.0.30;

    include snippets/self-signed.conf;
    include snippets/ssl-params.conf;

    #dummy certificates, remove after letsencrypt certificate is generated.

    #ssl_certificate /etc/letsencrypt/live/speckle.example.com/fullchain.pem;
    #ssl_certificate_key /etc/letsencrypt/live/speckle.example.com/privkey.pem;

    #ssl_stapling on;
    #ssl_stapling_verify on;

    # log access + errors
    access_log /var/log/nginx/speckle.log combined;

    # maintain the .well-known directory alias for renewals
    location /.well-known {
        alias /home/speckle/.well-known;
    }

    # This part might need a bit of tuning later on.
    # I'm not 100% sure about the wide range of timeout options available.
    location / {
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_pass http://speckleserver;
        proxy_read_timeout 86400;
    }
}

• speckleserver .env:

# This is a base file for configuring the speckle server. Steps:
# 2. Copy it to .env ( `cp .env-base .env`)
# 2. Configure the .env file with your settings.
# 3. Start the speckle server!
#
# NOTE: If these are set through a different process, eg docker, pm2, or command line
# they will not be overwritten.

# SERVER_NAME: The server name is important, as it will help users differentiate
# between multiple accounts.
SERVER_NAME="BIM UDIC Speckle Server"

# CANONICAL_URL: The url address (if you have any) where this server exists. For example,
# a canonical url can be "https://hestia.speckle.works".
CANONICAL_URL="http://10.15.0.30:3000"

# PUBLIC_STREAMS: Wether all streams created should be public (true) or private (false)
# by default.
PUBLIC_STREAMS=false

# PLUGIN_DIRS: Dirs to scan for speckle plugins (comma separated). For example, if you install
# some other modules in `./test` and `./wonderful`, you could use PLUGIN_DIRS="./test,./wonderful"
PLUGIN_DIRS="./node_modules/@speckle,./plugins"

# PORT: The port you want the speckle server to run on. Make sure the port is accesible
# if you have any firewalls set up (ie, ufw).
PORT=3000

# IP: The ip to listen to
# Defaults to all IPs bound to this machine, and all IPv6 IPs if IPv6 is enabled.
# If you want to bind to IPv4 only, or a specific IPv6/v4 address, uncomment the line below and specifiy your preferred address.
# IP=127.0.0.1

# MAX_PROC: The maximum amount of service workers to start. If this is not specified,
# Speckle will fork out as many as the machine's processor cores. To specify it,
# just uncomment the line below.
MAX_PROC=1

# REQ_SIZE: The request size protects your server from being flooded with too much data.
REQ_SIZE=10mb

# This is used in the encryption of the api access tokens.
# Please change it to something random!
SESSION_SECRET="xxx"

#
# DBs
#

# Mongodb URI: this is the uri string that the server will use to connect to your mongo instance.
# The default string below will work with an out-of-the-box local mongodb deployment.
# For more info, see: https://docs.mongodb.com/manual/reference/connection-string/
MONGODB_URI="mongodb://localhost:27017/speckle"

# Redis URI: this is the uri string that the server will use to connect to your redis instance.
# The default string should work with an out-of-the-box local redis deployment.
# For more info, see http://www.iana.org/assignments/uri-schemes/prov/redis
REDIS_URL="redis://localhost:6379"


#
# Housekeeping
#

# If you want your api calls to return pretty json, set this to true. It will cause
# the reponses to be ±10% bigger.
INDENT_RESPONSES=false

# If you want user email addresses to be publicly visible, set this to true.
EXPOSE_EMAILS=false

# The first user to register gets a server admin role.
# A server admin is allowed access to all resources on this server.
FIRST_USER_ADMIN=true

#
# Email sending
# This is future functionality that will be used for password resets, email
# confirmation, etc. when using the default auth strategy. It will also be used
# to send notifications (not in the near future though).
#

# SMTP server
# This section is used to send emails out. You can use the provider of your choice,
# as long as it exposes an SMTP server that you can use. Some of these providers are:
# sparkpost, mailgun, sendgrid.
SMTP_HOST="xxx"
SMTP_PORT=587
SMPT_USERNAME="[email protected]"
SMPT_PASSWORD="xxx"

# Will populate the `from:` field in any emails this server sends. Please note, some
# providers will require to verify your domain first.
EMAIL_SENDER="[email protected]"

#
# Authentication Strategies
#

# Set to false to disable register routes
PUBLIC_REGISTRATION=true

# Local
# Local is the default auth strategy for speckle (email + password).
# If this strategy is disabled, the api routes for registering and logging in will not work.
USE_LOCAL=true

# Auth0
# https://auth0.com
USE_AUTH0=false
AUTH0_CLIENT_ID="CHANGE_ME"
AUTH0_DOMAIN="CHANGE_ME"
AUTH0_CLIENT_SECRET="CHANGE_ME"

# Azure AD
# https://azure.microsoft.com/en-gb/services/active-directory/
USE_AZUREAD=false
AZUREAD_ORG_NAME="CHANGE_ME"
AZUREAD_CLIENT_ID="CHANGE_ME"
AZUREAD_IDENTITY_METADATA="CHANGE_ME"
AZUREAD_CLIENT_SECRET="CHANGE_ME"

# Github
# https://github.com/jaredhanson/passport-github
USE_GITHUB=false
GITHUB_CLIENT_ID="CHANGE_ME"
GITHUB_CLIENT_SECRET="CHANGE_ME"
GITHUB_CALLBACK="CHANGE_ME"

# Whitelisted domains
# Add here any other places that you trust this server to redirect to once authentication is done.
REDIRECT_URLS="http://localhost:5050,https://localhost:8080,https://app.speckle.systems,http://10.15.0.30:3000"

# Wether to allow non http*s* redirect urls.
# "localhost" redirect urls will always work with or without TLS.
ALLOW_INSECURE_REDIRECTS=true

#
# Telemetry
# see: https://discourse.speckle.works/t/community-consultation-time-telemetry/410
# This helps us keep track anonymously of the number of speckle servers deployed.
# Disabling this will make Speckle sad :(
#
TELEMETRY=true

#
# JN Mask
# Set a mask to enforce on the job number field in the admin.
# Format as: https://v15.vuetifyjs.com/en/components/text-fields#masks
#
JNMASK="######-##"

[htlcnn]

Ducking around, I found this issue: Navigating to URL with self-signed certificate shows empty screen

I searched SpeckleUi and found this line:
https://github.com/speckleworks/SpeckleUi/blob/05aa7d2480cac21ae56166a81112c6c55158c449/SpeckleUiBase/SpeckleUiWindow.xaml.cs#L53
If SpeckleRevit is using CefSharp < v75.1.141, I think we could add settings.IgnoreCertificateErrors = true;
If SpeckleRevit is using CefSharp >= v75.1.141, there's a proposal here: https://stackoverflow.com/a/45077988

Please tell me how could I rebuild SpeckleRevit and test these proposals.
Thanks.

[didimitrie]

Speckle Revit is a little PITA when it comes to building and testing. For reference, it's using CefSharp < 75, as Revit comes by default with version 57, so we can't force a newer version from the client side. Make sure that cefsharp 57 is being used! Otherwise things break.

Building and testing it: you'll need to replace the SpeckleUIBase nuget reference in the SpeckleRevit solution with a local clone of SpeckleUi, which you can modify and test.

There's also a quite a few flags that cef can be initialised with, not sure if this will be a better solution - probably not.

[htlcnn]

@didimitrie I tried rebuilding SpeckleUi after adding settings.IgnoreCertificateErrors = true; to SpeckleUiWindow. SpeckleUiTester still Could not access the server. Also, rebuilt SpeckleRevit.dll didn't work, showing a blank page instead. The pain began.

I tried another method and managed to install Let's Encrypt certificate to my server. I created a subdomain and point it (A record) to my Virtual machine's VPN IP (Let's Encrypt cert was issued to my local server successfully using DNS challenge):

But SpeckleRevit still Could not access that server. Chrome-devtools screenshot:

[htlcnn]

It seems that JWT Authorization failed after successful login. I tried to use Postman to send GET request to /api/streams with my token, also got 401. How could I test locally on SpeckleServer's cli?

[didimitrie]

Using something like Postman or Insomnia. There's some instructions here: https://speckle.systems/docs/developers/api-specs

[htlcnn]

I used Postman and used Speckle OpenApi Specs. I edited baseUrl and added my token to Header (key: Authorization, value: JWT... - my token I got from SpeckleAdmin).
Response was Unauthorized

[htlcnn]

I must have done something that messed the SpeckleServer up, so it didn't accept some types of request.

The first time, I installed SpeckleServer and setup smtp info. Unfortunately, smtp info was incorrect (I used password instead of app-password for gmail account), so the first register user (also admin user) never received verification email. I thought maybe non-verified user was not allow to create Senders, and I couldn't find resend verification email endpoint in SpeckleServer, so I fixed smtp info and registered another user, login to first user and set 2nd user Admin.

The second time, I removed SpeckleServer folder WITHOUT dropping mongodb database. My efforts weren't succeeded, with all the images above.

The third time, I stop SpeckleServer, drop mongodb database, npm install and run again. Now I'm able to create sender and send data to SpeckleServer.

Thank you very much for your help!

[didimitrie]

No worries @htlcnn! I didn't do much, was more acting like a sounding board. Happy things worked out :)

As a suggestion as to what potentially went wrong when redeploying the speckle server and then getting 401s with existing tokens (ie, not dropping the db): the session secret probably changed (it's in the .env file). That's used to salt passwords and sign the JWT tokens, and if it changes, basically all auth fails!

---

For the future/if you have time: it would be great to share or summarise your issue on the forum as it's more visible there, and more people can benefit :)

[htlcnn]

I'll make a thread on the forum.
Actually I changed secret in .env without thinking it would be related to database. So in that old situation, does registering another new user work? I wonder how I could still login using the web ui, but not via api, I thought they use the same endpoint.

[didimitrie]

I'll make a thread on the forum.
Actually I changed secret in .env without thinking it would be related to database. So in that old situation, does registering another new user work? I wonder how I could still login using the web ui, but not via api, I thought they use the same endpoint.

Theoretically yes. Checking the server code, I was wrong: password salting is handled by bcrypt, as it should be. Only JWT tokens for the api are affected.

Logging in the admin works as you get a fresh api token each time. The client side are not regenerated on logins, so removing then adding the account again would have not worked...

[htlcnn]

Forum thread created: https://discourse.speckle.works/t/speckleserver-deployment-notes-on-ubuntu-18-04/532
It's my pleasure to discuss with you. Thanks again @didimitrie