Logging of httpserver events?

[mskyttner]

I run a container which has this (superneat) extension installed and wonder if there is any logging available?

Currently I'm starting the service using this (compose.yaml):

services:
  duckserve:
    image: duckserve
    command: bash -c "DUCKDB_HTTPSERVER_FOREGROUND=1 duckdb -init serve.sql"
    volumes:
      - ./serve.sql:/serve.sql:ro
    ports:
      - "9999:9999"

The serve.sql init script I'm testing this setup with has various statements and end with: select httpserve_start('0.0.0.0', 9999, ''); ...

The service runs fine and is fast, and now I wonder if log messages can made visible to the container? Written to some kind of stream or to an in-memory duckdb table like 'httpserver_log' or both?

Thankful for any ideas/hints!

[lmangani]

Hello @mskyttner thanks for opening this issue and for the kind words! Logging is not really implemented but we'd be happy to consider it. Since the server is embedded within DuckDB, producing stdout seems too brutal - perhaps we can add an optional file target for stdout/err from the service. What information would be interesting and/or what other logs should we target for emulation?

[mskyttner]

On the xmas wishlist I guess I'd have a log written to a file in this format https://en.wikipedia.org/wiki/Common_Log_Format so for example a container can display it in a standard way. In one case I remember that an alpine-based container image with the cron service behaved nicely in that regard when it was started with a command like bash -c "startup.sh && crond -f -l 8 -L /proc/1/fd/1 2> /proc/1/fd/1" so maybe something which would allow a pattern like that would be good.

Having an in-memory duckdb table created with log data in the CLF format could maybe be nice - making it easy for other servers in the flock to check each others' logs :)

[lmangani]

I also considered writing to a table but this might quickly become a hog but there's a link to be made there, unless we point it at a separate database file but then some GC would be necessary and things would get complex. We have a "flock" function and some discovery magic in the works to create auto-forming distributed tables and this might leverage it once work out the basics. I think we'll start with a file first and we can extend it as we go.

[lmangani]

A generic receiver is step one: https://github.com/yhirose/cpp-httplib?tab=readme-ov-file#logging

svr.set_logger([](const auto& req, const auto& res) {
  your_logger(req, res);
});

Here's an untested firestarter: main...request-logger

Are you able and/or interested in building and testing locally?

[mskyttner]

Wow, yes, I can make an attempt, but it was a while ago I built duckdb locally and I haven't yet built extensions with the new mechanism for doing this, so I'd need to figure out how to set it up properly.

[lmangani]

Assuming you have the Linux build tools installed this is all it takes:

git clone -b request-logger https://github.com/quackscience/duckdb-extension-httpserver 
git submodule update --recursive --init
OVERRIDE_GIT_DESCRIBE="v1.1.3" GEN=ninja make

and then use the local build which already includes the extension

./build/release/duckdb

Mind i did NOT test the PR so it might fail at build-time too! I'd be able to try this in the next days so this is just optional in case you want to have fun and contribute while learning a few tricks together - nobody's an expert in quackscience! 😉

[lmangani]

@mskyttner i opened PR #17 with a couple options to test with

[lmangani]

@mskyttner I got around doing some basic testing and this PR seems to do the basics: #17

If you'd like to modify the output format, feel free to dig in. I'll await some testing and feedback before merging ✌️

[lmangani]

Feature implemented and pushed to community repo in version v.0.1.4 @mskyttner
duckdb/community-extensions#217

[lmangani]

Closing as implemented and ready to be easily extended. Let us know your feedback @mskyttner