codeunderhood.com still online?

[vsoch]

Hi! I wanted to check out codeunderhood.com but it's just infinitely spinning (and I suspect will time out). Is this a temporary thing or is it permanently offline? If the latter, do you have a container build that would make it easy to preview locally, or someone using it that I could look at? Thanks!

[robinp]

Hi @vsoch! The backend is down since I don't have the bandwidth to keep it updated, but good point, should update the readme and add docker.

Are you interested with a specific backend in mind, or just generally curious? The main branch used to support Kythe serving tables (for accurate crossreferencing), but that is not an approach I'm actively pursuing. There's a branch which works on top of Zoekt search indices, and also has some more UI features.

I think the latter might make more sense to see, as it is a path more accessible to developers (Zoekt indices are really easy to set up). If that sounds right, will set up some Dockers sometime the week.

[vsoch]

I’m not familiar with the backend, but I would support what is simplest and your intuition says is right! I’d be happy to test and give feedback if you want to add Docker in the coming weeks.

[robinp]

Hey @vsoch - please have a look at the README of https://github.com/TreeTide/underhood#getting-started-super-quickly.

After cloning, should be able to

$ cd docker
$ docker-compose -f docker-compose-demo.yml up

and then visit http://localhost:8080/ to (Ctrl+)click around in the demo index. Or use plain docker-compose.yml with your own index. See the README for nav usage using keys.

Ping if doesn't work.

[vsoch]

Woot! Will try tonight and let you know how it goes!

[vsoch]

Worked perfectly! So a question for you - what are the limits of scaling this? E.g., I see in the demo Dockerfile you are cloning each repo and then running the indexer. In practice I think that would work for a small number of repos, especially if you have to keep them around! How would it work for say, repos in the thousands, each with anywhere between 1 and 25 releases? Would the source code need to come from its own database, and then the index the same? I'm trying to figure out if it will work for a few use cases I have in mind. Thank you!

[robinp]

Hey @vsoch. The limiting factor would be the Zoekt backend, but I don't expect that problematic for that scale (and even later, it can be horizontally sharded). See https://github.com/sourcegraph/zoekt/blob/master/doc/faq.md#how-much-resources-does-zoekt-require (there seems a limit of 64 branches to index per repo).

Zoekt itself is fast to index, and the backend can pick up reindexed shards on the fly.

Note: the UI doesn't currently differentiate between results from different branches. That is supposedly a small fix, but would need to happen for the multi-branch usecase. Now the filetree displays both branches, but results don't have the indication, also they load the wrong source (from a random branch).

Note 2: if the repos/branches change frequently, the UI doesn't automatically pick that up and refresh yet. See #61 .

Note 3: while this works fairly well, there are no tests either on the UI or the gateway, to set your expectations.

Edit: re "where source needs to come from", I'm not sure what you mean. Zoekt has small indexer tools to index a given directory, or git repo etc. There is maybe also a small server to keep pulling updates and reindex changing repos on the fly? The serving is pretty agnostic to how the Zoekt shards are constructed.

[vsoch]

Thank you @robinp for providing the development environment and answering my questions! We definitely will consider trying out underhood when the time comes. Thank you again!

[sergei-dyshel]

@robinp

The main branch used to support Kythe serving tables (for accurate crossreferencing), but that is not an approach I'm actively pursuing.

Looks like UnderHood is the only viable alternative to deprecated sample UI in Kythe. How much work is needed to support Kythe again?

[kamahen]

I wrote a simple UI for supporting Kythe facts (not using the serving tables, but instead reading the JSON facts and converting them to another form). It's only been used with the facts that I generated, and I haven't done any work on it for a while. If you want more details, I can supply them (it's a fairly simple Javascript front end that queries (using JSON) a backend written in Prolog (I used Prolog because it could compute the transitive relationships dynamically rather than requiring them to be computed ahead of time, the way the Kythe server did things).

I've used the Underhood browser, built with the sources from github. It took a few iterations to get working (mostly because I was unfamiliar with nx), but I think the necessary stuff was added to its documentation. If not, I can search my emails to find the information and put together some notes.

[robinp]

Hey - indeed, the readme should give pointers. Nix is a (mostly) deterministic package manager, so while it will likely seem to download half the internet, the result should be usable. It seems 192e550 is a commit before the Zoekt work started, could have a look there.

See https://github.com/TreeTide/underhood#getting-started-quickly about compiling the frontend and backend, also how to run on some kythe index. (Well, it worked somewhat with the Kythe version of the time - if the API didn't change much, could still work).