Functions worker configuration for bare-metal cluster

[rob-kvietkauskas-ign]

Hello! 👋
Could someone assist me with advices/tips on function worker configuration in bare-metal cluster? I have a cluster of 6 machines (3 for brokers and bookies, 3 for zookeepers). Zookeeper cluster is functioning fine, same goes for Bookies (I successfully ran sanity check and simpletest). Brokers also work fine. I am trying to run functions workers with brokers, but after I perform setup steps described in documentation, I end up having bookkeeper (ledger allocation) errors, which in turn seem to be caused by incorrect configuration, every time I try to create (source or sink) connector. I have created an issue, but haven't received any replies yet so I am trying my luck in Discussion section. I encounter same errors in different Pulsar versions. I have deployed four separate clusters – two running version 2.9.1. and the other two – running version 2.11.1. The result is the same in both cases.

[rob-kvietkauskas-ign]

I have managed to register both source (both file and Debezium PostgreSQL Source) and (custom) sink connectors in 2.11.1 cluster. Stream storage service was disabled (as it is by default). I have set functionsWorkerEnablePackageManagement=true and enablePackagesManagement=true, restarted brokers and finally was able to run the connectors! I tried to do the same on 2.9.1, but result did not change at all – I still got ledger allocation errors.

[rob-kvietkauskas-ign]

I have finally managed to register both source (both file and Debezium PostgreSQL Source) and (custom) sink connectors in 2.9.1 bare-metal cluster. Once I added file:// schema prefix to path of the archive, sources create ... and sinks create calls (providing same configuration as before, using same nar files as before) ended up successfully.

Clearly, there's some kind of configuration, not mentioned/obfuscated in documentation, that's causing connector registration (upload) failures.

[tisonkun]

Could you elaborate a bit what archive you configured? IIRC we use Path type for archive path and it should not take file:// input

[rob-kvietkauskas-ign]

Pulsar instance is deployed same directory on (each broker machine) to /opt/pulsar/apache-pulsar-<version>;
Connectors (NAR files) are respectively stored in /opt/pulsar/apache-pulsar-<version>/connectors;
Connector configurations (YAML files) are respectively stored in /opt/pulsar/apache-pulsar-<version>/connectors/configuration;

Until last thursday, every time I tried to create (register) a connector (independent on type of it), I was performing following (working directory – /opt/pulsar/apache-pulsar-<version>/bin) for source – same way it was described in examples:

./pulsar-admin sources create \
--archive /opt/pulsar/apache-pulsar-2.9.1/connectors/pulsar-io-debezium-postgres-2.9.1.nar \
--name postgres-test \
--destination-topic-name source-connector-topic \
--source-config-file /opt/pulsar/apache-pulsar-2.9.1/connectors/configuration/debezium-postgres-source-config.yaml

and following for sink connectors:

./pulsar-admin sinks create \
--sink-config-file /opt/pulsar/apache-pulsar-2.9.1/connectors/configuration/sink-connector.yaml \
--name "sink-connector" \
--archive /opt/pulsar/apache-pulsar-2.9.1/connectors/custom-cdc-sink-connector.nar \
--parallelism 3 \
--inputs persistent://public/default/db_name.schema_name.table_observed_by_source_name

In bare-metal cluster such calls ended up with internal pulsar errors (related to ledger allocation failures). localrun calls with same parameters in same bare-metal clusters were successful. Only create was failing. On thursday, after reading same pulsar-admin sources create documentation line () about --archive parameter for a thousand time, I diceded to add file:// schema prefix to previously supplied --archive parameter value and that finally ended up in successfull create call (connector got registered successfully and was started automatically afterwards). The line was:

    -a, --archive
      The path to the NAR archive for the Source. It also supports url-path 
      [http/https/file (file protocol assumes that file already exists on 
      worker host)] from which worker can download the package.

I performed some more testing both in bare-metal cluster running 2.11.1 and 2.9.1. Even though I was able to create connectors without file:// schema in 2.11.1 cluster with functionsWorkerEnablePackageManagement enabled, connectors didn't start successfully after restart of the broker nodes – they ended up with internal errors. After deleting connectors and then recreating them while using file:// schema prefix, (same) connectors (with same configure) were recovering successfully after restart.

As I have mentioned before – I wasn't able to successfuly enable functionsWorkerEnablePackageManagement on 2.9.1 bare-metal cluster, so latest actions in 2.9.1 bare-metal cluster were performed with functionsWorkerEnablePackageManagement disabled.