Install missing plugins using CLI option

[FinnHering]

Description

Hey there,
I noticed that #1042 removed the functionality to automatically download plugins if they're missing. This is a big bummer if you manage your ulauncher config using a dotfile manager and sync those across multiple devices. I understand the concerns that automatically doing changes to the extension folder could be seen as an anti-feature. How about we allow the user to run a similar operation using a CLI option? I would be willing to work on that, as it doesn't seem too hard.
Implementing this also fixes #1018, not completely, but it would allow the user to build a script to do just that. :)

Let me know what you think!

Communication guidelines

• I have read and agree with Ulauncher's communication guidelines and checked the existing feature requests before suggesting this.

Feature isn't possible to support as an extension.

• This feature request cannot satisfactory be supported as a third party extension.

[friday]

I'm not against adding this back as a cli command, but I want to support all the obvious extension management commands (add/install, remove/uninstall, update, show, etc), and also commands for debugging extensions without copying them to the extension directory. And for migrating extension manifests between API versions APIv2 to v3).

Adding all of that to the current ulauncher command and the existing default python CLI parser (which is very limited) would probably create a mess. Imo it's better to make a new ulauncher-extension (or preferably something shorter) command and add a more powerful parser like https://click.palletsprojects.com/ for that. If you're up for helping with that it'd be very welcome. I think supporting this particular issue with ulauncher-extension upgrade with an optional flag --restore might make sense for example.

Not expecting you to do all that of course, just clarifying the context I think is right for this.

[FinnHering]

Sounds awesome!
So initially – I propose – we'd have the following options available:

Extension development specific commands:

• dev create [name]: Creates a new extension named "[name]" at folder ./[name] using a boilerplate project
• dev migrate [versionTo]: Migrates extension inside PWD from current manifest version to "[versionTo]" manifest
• dev debug: Runs an isolated ulauncher instance with only the extension in PWD and debug logging

---

Extension management related commands:

• add/install git url OR userName/projectName OR id]: installs (and runs?) extension if not already exists
• rm/remove/uninstall [git url OR userName/projectName OR id]: removes (and stops) extension
• show: lists all installed extensions in a human-readable format
• upgrade [--restore]: upgrades all installed plugins listed in extensions.json. On --restore remove all extensions that are not in extensions.json and update.

P.S.: Consider me a python noob. I can develop basic python scripts and maybe a bit more without the codebase looking like a mess. But I never really build anything at this scale. So it would probably be best if you set up a basic project structure and architecture to your liking and I start building on top of that. :)

[friday]

I won't have much time for Ulauncher for a while and I have other priorities when I do.

So if you want to get started I think you should just try it. Clone the ulauncher repo. Add a new empty file for the command in the "bin" directory, install. Set the execution permissoin and check out how to use click and use their guides to add the more basic commands similar to https://manpages.ubuntu.com/manpages/focal/man1/gnome-extensions.1.html

[FinnHering]

Will do. Thank you for your time! I'll start working on it here.

[FinnHering]

@friday I'm almost done. Unfortunately, the upgrade might get tricky. The problem is that the current code uses (github's) API to check for and do updates, which triggers a rate limit pretty fast. So one option is to simply not add an "upgrade all" feature or simply use git. What do you think?
The code (including the /broken/ upgrade function) is here: https://github.com/FinnHering/Ulauncher/blob/v6/bin/ulauncher-extensions

Feedback on the code is very welcome!

[friday]

Awesome @FinnHering! I'll try to have a look at it this weekend.

Imo it doesn't need to be "perfect" as long as it works. We can add things to it like for example additional commands, flags and progress bars etc.

About the GitHub rate limit, this is a problem with using the GitHub public API like we do. The limits are pretty high, but if you have a lot of extensions and run the commands a lot I imagine you might hit the limits. Some users reported rate limit errors too. Hard to get around, but I never had time to look into it.

[FinnHering]

Great! :D

I've also enabled autocomplete where I could (uninstall and update). If you want to try that out, see: https://click.palletsprojects.com/en/8.1.x/shell-completion/#enabling-completion
and add the bin directory to PATH :)

[FinnHering]

FYI: just added the version command and the restore command. I figured that it would probably be best to keep restore functionality out of the upgrade command, since it is basically a whole different feature. :)

[FinnHering]

I also just tried out the upgrade functionality through my mobile phone's internet to ensure that previous requests don't screw with my rate limit, and I still got rate limited. I will need to optimize that functionality, since I first run a ext_downloader.check_update on every extension and then update the extensions that have updates available using ext_downloader.update which result in many redundant API calls.

I refactored the update command to only use ext_downloader.update to reduce the API calls significantly.

But for basic git operations like checking for new commits (updates), tags (ulauncher compatibility) you could just use plain ol' git instead of the API. The only drawback is that you now need to parse gits output, which is a pain compared to nice JSONs. But there is a rather popular library that could take care of it: https://github.com/gitpython-developers/GitPython
This would also remove the need to do rather messy things like that: https://github.com/Ulauncher/Ulauncher/blob/v6/ulauncher/modes/extensions/ExtensionRemote.py#L132

But this is not the scope of this issue. If you want to, I could assist in moving some functionality over to “plain git” using the GitPython lib.

[friday]

Apparently the rate limit is 60 requests per ip per hour for unauthenticated requests: https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#requests-from-personal-accounts. I thought it was more. They really hid that in the foot notes.

I think that's quite reasonable for regular use, and I never hit the rate limit myself (even when working changes to the ExtensionRemote) but it's also possible that other apps use the GitHub API, or for someone to have over 30 extensions installed, so I agree it'd better to move away from the Git API.

I already considered switching to pure git or pygit2. GitPython is semi-abandoned by the author and there are no system packages for Ubuntu, so we can't use it, and it's better to use more common dependencies. So I think using pure git would be the best option. It's not complicated, and you can always wrap that in a neat small utility class/function. That way is better for support on smaller distros that lacks a rich package ecosystem (like CentOS), and you could get the progress indicators for free if you redirect the stdout from git (assuming we don't want control of the exact output).

Using git is different from downloading from the API though. With the API we're literally downloading only the files from one specific moment, without the metadata (.git directory), but a regular git you need the metadata first and you want to avoif having to download all the history every time. For example https://github.com/tchar/ulauncher-albert-calculate-anything has a lot of history and big files too. You can make it faster with --depth 1, but the better way would probably be to do a bare clone and check out from there.

Tagging @troycurtisjr too in case he has any input on this.

[troycurtisjr]

It would have to be Git over http, right? Does that not hit the same limits?

[FinnHering]

As far as as I know GitHub doesn’t rate limit git. I will do a little stress test this evening to confirm that

[friday]

It would have to be Git over http, right? Does that not hit the same limits?

GitHub REST API is separate from the GitHub website and using Git to clone from GitHub. The API rate limits only apply to request to https://api.github.com/ We need to make those now to know is a package is up to date, but after that downloading it shouldn't be rate limited.

What do you mean by Git over http? http[s]://github.com/owner/repo[.git] instead of the SSH address?

I think you can use both without authentication, and both require setting up a SSH locally?

[FinnHering]

I don’t think that git needs ssh if you use http.

[friday]

Nice. It seems like you are correct. I haven't used Git over HTTPS in years, so I forgot. So we should use HTTPS then. (HTTP is not an option, it just redirects to HTTPS)

[FinnHering]

I wrote an issue for the git topic (#1162)