Writing a FatRat Extension

This page is for people who write extensions. To install and manage extensions someone else wrote, see Extensions.

A FatRat plugin is a Python module that teaches FatRat about links it does not otherwise understand — typically a file host, where the link the user has is a web page and the real download URL has to be scraped out of it. The plugin does the scraping; the downloading stays FatRat's job, so a plugin's transfers get segmenting, speed limits, resume and the rest without asking.

The shape of one

import fatrat

@fatrat.download_plugin(name="Example.com", url_pattern=r"https?://example\.com/f/\w+")
class Example(fatrat.DownloadPlugin):
    async def process_link(self, url):
        page = await self.fetch(url)
        token = page.search(r'name="token" value="([^"]+)"')
        if not token:
            raise fatrat.PluginError("no download token on the page")

        await self.wait(30)
        answer = await self.solve_captcha(page.urljoin("/captcha.png"))

        final = await self.fetch(url, data={"token": token, "captcha": answer})
        self.start_download(fatrat.DownloadUrl(url=final.json()["link"], referrer=url))

Plugin code runs on FatRat's main thread, so it must never block it: anything slow is awaited. Everything you can await is something FatRat does for you — a page fetch, a countdown, a captcha — and the coroutine resumes when it is done.

What a download plugin can use

Two more methods are worth overriding: final_check(path), which gets the finished file so you can reject the HTML error page a host served with a 200, and on_failed(), to drop state when a transfer fails. Anything you raise out of process_link fails the transfer with that text, so write those messages for a user rather than for a log.

Packaging

A plugin is a directory with a plugin.json manifest next to the module:

{
  "schema": 1,
  "name": "example",
  "version": "1.0.0",
  "api": "1",
  "entry": "example.py",
  "license": "GPL-3.0-only",
  "config": [
    {"type": "text", "key": "example/user", "title": "Username"},
    {"type": "password", "key": "example/password", "title": "Password"}
  ]
}

The config list is your settings page: FatRat renders it in the desktop settings dialog and in the web interface, and the values land in the settings keys you name.

Testing a plugin without FatRat

FatRat ships a small development kit, fatrat_devkit, that runs a plugin under plain python3 (3.11 or newer): no FatRat build, no restart, no desktop. It stands in for FatRat's side of the API, so your process_link() really runs — pages are fetched over the network, a countdown can be skipped, a captcha is asked for on the terminal — and what the plugin hands back is printed instead of downloaded. It is installed with FatRat under share/fatrat/data/python; in a source checkout it is data/python.

export PYTHONPATH=/usr/share/fatrat/data/python     # or .../fatrat/data/python in a checkout

python3 -m fatrat_devkit check    ./my-repo
python3 -m fatrat_devkit match    ./my-repo 'https://example.com/f/abc123'
python3 -m fatrat_devkit download ./my-repo 'https://example.com/f/abc123'

PATH is a plugin directory (with a plugin.json) or a whole repository (with a fatrat-plugins.json).

The commands that run a plugin take these options:

It exits 0 when the plugin succeeded, 1 when it failed (with the message the user would see; an unexpected exception also prints its traceback) and 2 when it could not be loaded.

Unit tests

For tests that must not touch the network, give the same machinery canned responses. Import fatrat_devkit before fatrat — the plugin base classes are chosen when fatrat is first imported — then drive your plugin class with a Host:

from fatrat_devkit import Host, MockTransport      # first
from example import Example                        # then your plugin

def test_resolves_the_link():
    host = Host(transport=MockTransport({
        "https://example.com/f/abc": '<input name="token" value="T">',
        ("POST", "https://example.com/f/abc"): '{"link": "https://cdn.example.com/abc.bin"}',
    }), settings={"example/user": "me"})

    result = host.run_download(Example, "https://example.com/f/abc")

    assert result.ok, result.error
    assert result.download.url == "https://cdn.example.com/abc.bin"
    assert result.requests[1].form["token"] == "T"

A canned response is a body, a dict with status, body and headers, a callable that receives the Request, an exception to raise, or a list of those answered in turn. An HTTP status of 400 or more fails the fetch with NetworkError, as it does in FatRat. The returned Result carries ok, error, download, file_name, vars, waits, requests and log; the other plugin kinds have run_extract, run_search, run_account_status and run_upload. Without pytest it works the same under python3 -m unittest.

The devkit is an approximation, not FatRat: pages are fetched with Python's urllib rather than FatRat's network stack (so proxy settings, the exact user agent and redirect handling can differ), FatRat's captcha dialogs are replaced by the terminal, and url_pattern is matched with Python's re where FatRat uses Qt's regular expressions. Before you publish, install the plugin into FatRat once and try it for real.

Dependencies

Most plugins need nothing but the standard library and fatrat. When yours genuinely needs a library, list it in the manifest:

  "dependencies": ["feedparser"]

FatRat will not install those packages as a side effect of installing your plugin: it names them and asks first, because they come from PyPI rather than from the repository the user chose. They are then fetched with pip into a _vendor directory inside your plugin, so two plugins may depend on different versions of the same package, and removing a plugin removes its dependencies with it. Keep the list short: every entry is one more thing the user has to trust.

Publishing

There is no central plugin index. You publish a repository — a git repository, or any directory — with a fatrat-plugins.json at its root listing what it offers, and users add it by URL under Settings → Extensions. FatRat records the exact commit it installed, so updates are something a user chooses.

The FatRat source ships a complete example under examples/plugin-repo/: a repository manifest and a commented dummy plugin that exercises fetching, a countdown, a captcha and final_check. Adding that directory as a repository is also the way to develop a plugin, since a local repository is used where it lies.

A plugin runs with your full user privileges — the same trust you extend to anything else you install from a git repository. FatRat shows the source and the revision before it installs anything, and a plugin can be switched off again under Settings → Extensions, or all of them at once with fatrat --no-python.