😰 Trying to automate git backup using shell (stdin for custom shell in WSL)

[SrGeneroso]

Copied from #89 👇🏻

Hi! I'm quite new to the plugin and most likely I'm streching the capabilities of
it. The thing, just for giving context, is that I want remove the git plugin and use shell commands instead. I use WSL and I like to use SSH for the repos within.

I'm wondering, if it would be possible to propmt in the moment when the passphrase is needed.

I thought first to use a prompt to get the password and pass it, which could be theoretically possible using something like sshpass, but I think it would be cooler to don't have such a potentially insecure tool installed as quoting from an article

SSH uses direct TTY access to ensure that the password is indeed issued by an interactive keyboard user. sshpass runs SSH in a dedicated TTY, fooling SSH into thinking it is getting the password from an interactive user.

I know what I'm doing is not very secure as well, at least could be, but anyways, this is how I learn things and maybe there is something in Shell-Commands I don't know yet.

The command I try to make work is this one
wsl -d Ubuntu-22.04 --cd "/mnt/e/00 MD Vault/Obsidian/" --exec git add . && git commit -m "backup at {{date:"HH:mm"}}" && git push origin main
It launches wsl in the cli by default, in this case cmd, navigates to the path and executes the git commands.
From

I guess stdin might be the solution, but following the thread might not be possible? I tried echo "{{_pass}}" | git push origin main and didn't work either. Might be related to the TTY access mentioned before.
So..., stdout handling wouldn't work either...
I don't know, maybe this kind off topic, I was about to open a discussion instead but this might be nearer to the real subject. Let me know if I should repost it there.

In any case, the ideal situation would be that the cli is showing and I can type the passphrase there.

but also it would be really cool to use the prompt. What do you think?

[SrGeneroso]

@Taitava suggested 👇🏻 (Transposed from #89 )

Hi! Thank you for your message! 🙂

Yeah, I think it might be better to repost this in a new discussion. This discussion #89 is a bit long already 🙂. I do agree that having a possibility to open up an interactive CLI window would help a lot here, so you could put that in the new discussion's name 👍 .

Meanwhile, you could also try to:

Invoke WSL using a custom shell. This way you don't need to include the wsl -d Ubuntu-22.04 --cd "/mnt/e/00 MD Vault/Obsidian/" --exec part in your actual shell command. Just git add . && git commit -m "backup at "{{date:HH:mm}} && git push origin main is enough.
Maybe the stdin might work better now with the custom shell being used? At least it would be simpler, because then there would be no Windows CMD in the middle, possibly interfering. I'm not at all certain that this would still work. If it doesn't, can you then try the sshpass, even if you will not end up using it permanently?
Let's continue talking in the new discussion when you create it. 🙂

[SrGeneroso]

So... Indeed there was something I missed, the custom shells. Let's give that a try.
Actually the WSL guide is awesome, the path translator and everything shows how cool this plugin is.

Anyway, following the guide... this is important,

It's important to know that WSL doesn't have a -c parameter, i stumbled on that for a while.
Ok, what's next... Woah, the path translator, I wouldn't get that regex in a year, but it's quite nice and simple, I might yoink it for some AHK scripts, but the best part is the test button.

Next, setup of the command. Let's go. I've tried
git add . && git commit -m "backup at {{date:HH:mm}}" && git push origin main <<< "$(echo {{_pass}})
and
git add . && git commit -m "backup at {{date:HH:mm}}" && echo {{_pass}}) | git push origin main
and you may notice that I'm not very skillful with bash 🥺 but let's keep trying.
Before I go bananas, let's try the sshpass to confirm this could be possible.

First let's try to use directly outside obsidian,

Ok, is doable, sshpass -v -P 'pass' -p 'pass' git push origin main butsome gotchas,
it hanged for a while, so I added -v to see what was happening, the thing is, by default the command it's awaitting a prompt matching 'assword' (hehehe, ass) and ssh was asking for a passphrase...
this might be just for me as my setup already has the ssh keys but I protect them by a passphrase, in any case, change the string to match using -P 'pass' and try again and... there it is.

Now let's try that with sheel commands in obsidian.

I change the command to something like this
git add . && git commit -m "backup at {{date:HH:mm}}" && sshpass -v -P 'pass' -p '{{_pass}}' git push origin main
and launch it. Remember I have a prompt asking for my pass and storing it as _pass.
And... fail whit sshpass not recognized as a command...

So wait... why is that? Oh boy, I forgot to change the enviroment of the command,
Since I'm in windows that's the one I changed with the custom shell I created before.

Another try and...

It did it, yeah 😸...
Now, can it be done without sshpass? I'll try later

[SrGeneroso]

A couple of notes.

In custom shell enviroments... I have a default WSL profile which is the one I want to use, but, you might have another, to select the correct one you could edit the shell arguments to target a specific one

You can get the names of your installed distros with wsl-l in cmd.

In preactions prompts:
It would be nice to have a text input of type password to not flash those kind of things visually. As you migh notice, I use gif with sharex to post steps and is messy to edit gifs, so that could be a nice addition.

In output:
Would be nice to have a visualizer of what is happening

Another two unrelated and already covered:
The hidden setting for removing re_execute and the one for max_visible_lines_in_shell_command_fields should promote to actual settings, they are awesome, although I think max_visible_lines should be set to auto based on the content. Just being picky here.

Be able to export and import #151 could be helpful while debuggin and when sharing into the git discussion. I'm not sure how propmts and enviroments could be implement though but I really like that idea.

[Taitava]

It would be nice to have a text input of type password to not flash those kind of things visually

@SrGeneroso A password field is now released in SC 0.21.0. 👍

[SrGeneroso]

Now to use the stdin to remove sshpass
The thing is I'm not a pro bash user so this might be why I'm strugglin, but here are some questions.
When do the stdin execute?
If you have a single command it waits until you have a text prompt and the uses it?
I'm chaining commands and I'm not sure how to tell stdin "come on, it's your turn. Theoreticly is the first time an input is required but I wonder if this process runs on TTY raw mode and that's why stdin doesn't work. And by this process I gues I mean the ssh agent?
Anyways, I've found alternatives to SSHPASS as EXPECT and PASSH

[SrGeneroso]

Another thought, I'm using wsl and git is using credential.helper=libsecret what about that? I have to assume then there is no ssh agent and is just git, which propmt a tty and uses that as stdin but, is still subprocess?. Maybe is more simple than I thought. Let's try only one command, I'll manually commit and just push with the command.

Mmm, no luck, let's try first in bash to if it works with maybe with...
Pipes: echo 'pass' | git push origin main ❌
Here-Strings: git push origin main <<< "pass" ❌❌

So there is currently no way. Let's check how is stdin working in the plugin.
So, here is the file: obsidian-shellcommands/src/ShellCommandExecutor.ts and in line 261 we have this:

// Pass stdin content (if defined)
                if (undefined !== shell_command_parsing_result.stdinContent) {
                    // Stdin content is defined
                    debugLog("Stdin content is present in parsing result. Will write it to the process.");
                    if (null === child_process.stdin) {
                        // noinspection ExceptionCaughtLocallyJS: The exception is caught locally below, but it's ok because it's then rethrown as the error message does not match '/spawn\s+ENAMETOOLONG/i'.
                        throw new Error("Shell command execution process does not have a standard input stream (stdin).");
                    }
                    child_process.stdin.write(shell_command_parsing_result.stdinContent);
                    child_process.stdin.end();
                } else {
                    debugLog("No stdin content is present in parsing result.");
                }

Alright, so, we have a child_process which comes from electron/node which may depend of our enviroment configuraiton, in any case that is working, I assume. and we write to it the content of our stdin from the plugin...
Mmm, how can I test this? Btw, there are a lot of debug logs, how can i see those?

[SrGeneroso]

ok, hidden settings to the rescue. debug set to true. Let's play a little.
Here seems to be the magic

Mmm, interesting, how can i set Shell to true? I'm loosing the track. So... here we are, I wonder... Could I set a query string and look into stdout and when it matches the string then write into stdin?
Might be interesting since there is a realtime feature.

Mmm, time to look into that, as I said, this is how I learn, is like punching yourself in the face but in the end you are smarter.

[SrGeneroso]

I think is time to fork, haven't tried to use custom plugins in obsidian, let's see how it goes.
I had to update
npm update rollup npm update typescript npm update @rollup/plugin-typescript
but seems to be able to pnpm dev now.

Also, recreating the setup i had I noticed that when creating a prompt in preactions, if you create a new variable in target variable it doesn't show in variables tab and the name is reserved. Might need to do an issua about that later.

Now, i have the plugin clone from the fork and debug is true. Let's see if I can log something.
Alright!, you have to reload obsidian every time, but this is some advance.

So. Let's get something working. first I want to log when the output matches something first let's activate realtime and let's find when the stdout is called.
Seem to be in handleBufferedOutput or no, wait, might be handleBufferedOutput, let's try to debug from there.

Nope, I think I'm not getting in here. Also, the code has great comments, it helps a lot.
I restarted dev and reopen obsidian a couple times and seems like I'm there now.

Now, how do I output things.

Alright... but why is all in one block? and why it doesn't work when using a command that awaits the stdin? It works with the regular git add . && git commit ... because that ends, but it doesn't work with git push because it has a prompt for a passphrase.

Lets go back to private async executeShellCommand

        // Define output channels
        // Check that the shell command is not empty
        // Check that the working directory exists and is a folder
        // Check that the working directory exists and is a folder
        IF
        // Working directory is OK
        // Pass possible PATH augmentations to the Shell - if the Shell supports them.
       // Execute the shell command
                // Pass stdin content (if defined)
                // Common error handling regardless of output handling mode
                // Define output encoding
               // Define a terminator
               // Hook into child_process for output handling (this is where weswitch on case realtime)
              // Display a notification of the execution (if wanted).
              
              //Follow with this from the hook above (this.handleRealtimeOutput(child_process, shell_command_parsing_result, outputHandlers, processTerminator);)
                      // Prepare output channels
                     // Define an output handler
                     // Hook into output streams' (such as stdout and stderr) output retrieving events.
                      // Note that there might be just one stream, e.g. only stderr, if stdout is ignored. In the future, there might also be more than two streams, when custom streams are implemented.

[Taitava]

Sorry it took me this long to get back to you! I'll try to cover all the questions you posted in this single message. Please let me know, if I missed someting.

Actually the WSL guide is awesome, the path translator and everything shows how cool this plugin is.

Thank you a lot for your kind words! 🙂

It's important to know that WSL doesn't have a -c parameter, i stumbled on that for a while.

Good point, I think I will add an explicit mention to the WSL guide that no -c parameter should be used.

Ok, what's next... Woah, the path translator, I wouldn't get that regex in a year, but it's quite nice and simple, I might yoink it for some AHK scripts, but the best part is the test button.

Nice feedback again 🙂. I'll write a detailed description of the regex. It's good to explain how the translator code works to clear things up.

Another try and...
It did it, yeah 😸... Now, can it be done without sshpass? I'll try later

Nice that it works at least with sshpass! 🙂

---

A couple of notes.

In custom shell enviroments... I have a default WSL profile which is the one I want to use, but, you might have another, to select the correct one you could edit the shell arguments to target a specific one

Thank you for this tip! I'll add it to the documentation.

In preactions prompts: It would be nice to have a text input of type password to not flash those kind of things visually.

An excellent idea! 😎 It never crossed my mind. I'll definitely add a password field. I put it on a todo list here. I have some other field types in the works, too, and this one is so simple that I can include it at the same time I release other field types.

In output: Would be nice to have a visualizer of what is happening

Sorry, can you tell me more about what you mean by this visualizer?

Another two unrelated and already covered: The hidden setting for removing re_execute and the one for max_visible_lines_in_shell_command_fields should promote to actual settings, they are awesome,

I'll take promoting these to actual settings in the UI into consideration. The hard part is to decide which tab they should belong to. I mean, they are related to shell commands, so they could be placed at the bottom of that tab. But the tab gets already easily crowded at least if a user has a lot of shell commands defined. And I'm trying to avoid adding a new tab, as the visual space is limited. :)

although I think max_visible_lines should be set to auto based on the content. Just being picky here.

What do you mean by this? I think the current default behavior is that shell command fields get as tall as there are lines in the shell command. It has one issue though: it doesn't count soft-wraps, and might therefore be too short.

Be able to export and import #151 could be helpful while debuggin and when sharing into the git discussion. I'm not sure how propmts and enviroments could be implement though but I really like that idea.

Yeah, that's currently the biggest missing piece in this otherwise near complete puzzle. My dream is to make all configurable elements/settings exportable/importable, so that a shell command that relies on e.g. specific prompts, custom variables, and a custom shell, could include all its "dependencies" in a single block of text.

---

When do the stdin execute?
If you have a single command it waits until you have a text prompt and the uses it?

Yes, if I recall correctly, stdin content is given to the executed process when it asks for it.

I'm chaining commands and I'm not sure how to tell stdin "come on, it's your turn. Theoreticly is the first time an input is required but I wonder if this process runs on TTY raw mode and that's why stdin doesn't work. And by this process I gues I mean the ssh agent?
Anyways, I've found alternatives to SSHPASS as EXPECT and PASSH

Sorry, I'm not an expert too, and don't know much about TTY raw mode. Actually, I'm not too familiar with stdin in the first place. It's not the most used feature of this plugin, I guess, so I'm not very experienced with it.

---

Another thought, I'm using wsl and git is using credential.helper=libsecret what about that? I have to assume then there is no ssh agent and is just git, which propmt a tty and uses that as stdin but, is still subprocess?. Maybe is more simple than I thought. Let's try only one command, I'll manually commit and just push with the command.

Sorry, I'm not familiar with Git's credential helpers / libsecret, either.

Alright, so, we have a child_process which comes from electron/node which may depend of our enviroment configuraiton, in any case that is working, I assume. and we write to it the content of our stdin from the plugin... Mmm, how can I test this?

Probably you could have a shell command like echo -e "$(cat)". It should output the content of stdin. Or did you mean something else with testing?

---

Mmm, interesting, how can i set Shell to true?

shell is false when using a Custom shell. (But true when using the simple default shells built in to this plugin). Node.js's child_process does not let customise the shell's arguments, so I decided not to use the child_process's shell feature with custom shells. Instead, child_process is used to launch a process without a shell, which then executes the custom shell's binary file. So even as we start with no shell, we'll end up invoking one ourselves, with a possibility to introduce whatever arguments to the shell we want.

Could I set a query string and look into stdout and when it matches the string then write into stdin? Might be interesting since there is a realtime feature.

Sorry, I did not get the question. :)

Mmm, time to look into that, as I said, this is how I learn, is like punching yourself in the face but in the end you are smarter.

Yeah, learning by reverse engineering. :) One thing that the documentation of this plugin lacks is a technical overview of the shell command execution flow. It has gotten quite complex over time.

---

Also, recreating the setup i had I noticed that when creating a prompt in preactions, if you create a new variable in target variable it doesn't show in variables tab and the name is reserved. Might need to do an issua about that later.

Ok, there are some bugs in this plugin that settings views don't always get updated after changes. yea, if you can, please post an issue about it.

Now, i have the plugin clone from the fork and debug is true. Let's see if I can log something. Alright!, you have to reload obsidian every time, but this is some advance.

You can use Hot-Reload plugin for Obsidian to reload the plugin every time you build the main.js file. I'm using it and it works fantastically well.

So. Let's get something working. first I want to log when the output matches something first let's activate realtime and let's find when the stdout is called. Seem to be in handleBufferedOutput or no, wait, might be handleBufferedOutput, let's try to debug from there.

If you have the realtime setting on, handleBufferedOutput() should not be called, as that's called when realtime is off.

Also, the code has great comments, it helps a lot.

Thanks! :)

Now, how do I output things. Alright... but why is all in one block?

Probably Git outputted stuff in one block, or so fast that this plugin was not fast enough to read the output before all of it became available. The realtime feature usually gets output content piece by piece when there is noticeable pauses in the output. I guess you executed just git status? That command is so fast that its output probably comes as a single piece. As an opposite example, I guess git push should give you output in multiple pieces when realtime mode is on.

and why it doesn't work when using a command that awaits the stdin? It works with the regular git add . && git commit ... because that ends, but it doesn't work with git push because it has a prompt for a passphrase.

Sorry, I'm not following. What does not work? Is there any output?

Lets go back to private async executeShellCommand

        // Define output channels
        // Check that the shell command is not empty
        // Check that the working directory exists and is a folder
        // Check that the working directory exists and is a folder
        IF
        // Working directory is OK
        // Pass possible PATH augmentations to the Shell - if the Shell supports them.
       // Execute the shell command
                // Pass stdin content (if defined)
                // Common error handling regardless of output handling mode
                // Define output encoding
               // Define a terminator
               // Hook into child_process for output handling (this is where weswitch on case realtime)
              // Display a notification of the execution (if wanted).
              
              //Follow with this from the hook above (this.handleRealtimeOutput(child_process, shell_command_parsing_result, outputHandlers, processTerminator);)
                      // Prepare output channels
                     // Define an output handler
                     // Hook into output streams' (such as stdout and stderr) output retrieving events.
                      // Note that there might be just one stream, e.g. only stderr, if stdout is ignored. In the future, there might also be more than two streams, when custom streams are implemented.

I have to say that I appreciate that you have taken the time to go through the source code of this plugin, trying to learn how it works! Not an easy task, I admit. 😎

[Taitava]

I published the documentation changes now.

Btw I also added a --shell-type login argument to the Shell arguments section in the WSL custom shell documentation. I guess it's not needed in your situation, but might help in situations where a command is not found.