Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 daniels
|
8cb82ee8c4 | ||
|
daniels
|
2f996600c6 | ||
|
daniels
|
67335a37b9 | ||
|
daniels
|
259f5e7ee2 |
@@ -5,3 +5,4 @@ __pycache__/
|
|||||||
*.pyc
|
*.pyc
|
||||||
*.db
|
*.db
|
||||||
*.partial
|
*.partial
|
||||||
|
AGENTS.local.md
|
||||||
|
|||||||
@@ -1,80 +0,0 @@
|
|||||||
# Importarr Agent Instructions
|
|
||||||
|
|
||||||
Importarr is owned as a Linux-ops-managed service repository. Treat this checkout as the source of truth for application code, deployment files, and local service installs.
|
|
||||||
|
|
||||||
## Repository Source Of Truth
|
|
||||||
|
|
||||||
- Work from `/srv/opencode-workspace/importarr` for Importarr code and deploy changes.
|
|
||||||
- Do not edit host-local legacy scripts as the normal workflow:
|
|
||||||
- `/usr/local/sbin/importarr-status.py`
|
|
||||||
- `/usr/local/sbin/manual-media-import.py`
|
|
||||||
- If an emergency hotfix is made outside the repo, backport it here immediately and reinstall from the repo.
|
|
||||||
|
|
||||||
## Seamless Feature Workflow
|
|
||||||
|
|
||||||
When asked to implement an Importarr feature, fix, UI change, deployment change, or operational behavior change:
|
|
||||||
|
|
||||||
1. Inspect `git status --short --branch` before editing.
|
|
||||||
2. Implement the smallest correct repo change.
|
|
||||||
3. Run the narrowest useful verification, normally:
|
|
||||||
```sh
|
|
||||||
.venv/bin/pytest -q
|
|
||||||
```
|
|
||||||
If shell deploy scripts changed, also run:
|
|
||||||
```sh
|
|
||||||
sh -n deploy/systemd-install.sh && sh -n deploy/repo-upgrade.sh
|
|
||||||
```
|
|
||||||
4. Inspect `git diff` and ensure no secrets, raw `.env`, tokens, databases, or private material are included.
|
|
||||||
5. Commit and push completed Importarr changes by default unless the user explicitly asks not to publish or verification is blocked.
|
|
||||||
6. Install/restart from the repository so the running local service matches the repo:
|
|
||||||
```sh
|
|
||||||
make upgrade-local
|
|
||||||
```
|
|
||||||
7. Verify the live service:
|
|
||||||
```sh
|
|
||||||
make verify-live
|
|
||||||
```
|
|
||||||
8. If the live install fails, inspect `systemctl status importarr.service` and `journalctl -u importarr.service`; fix the repo, commit/push the fix, reinstall, and verify again.
|
|
||||||
|
|
||||||
## Install Model
|
|
||||||
|
|
||||||
- The service virtualenv lives at `/opt/importarr/venv`.
|
|
||||||
- The systemd unit runs `/opt/importarr/venv/bin/importarr` as the `importarr` system user.
|
|
||||||
- The package is installed from the repository into the venv using normal wheel/package install, not editable install.
|
|
||||||
- Do **not** use editable install for the system service: the unprivileged `importarr` user may not be able to read `/srv/opencode-workspace/importarr`, causing `ModuleNotFoundError` at startup.
|
|
||||||
- `/opt/importarr/repo-upgrade.sh` is the pull-and-reinstall helper for machines that should follow pushed `main`.
|
|
||||||
|
|
||||||
## Local Commands
|
|
||||||
|
|
||||||
```sh
|
|
||||||
make test
|
|
||||||
make install-systemd
|
|
||||||
make upgrade-local
|
|
||||||
make repo-upgrade
|
|
||||||
make verify-live
|
|
||||||
```
|
|
||||||
|
|
||||||
`make repo-upgrade` is for pulling already-pushed changes with `git pull --ff-only`. It refuses to run with uncommitted repo changes.
|
|
||||||
|
|
||||||
## Runtime Defaults On This Host
|
|
||||||
|
|
||||||
- Local URL: `http://127.0.0.1:8095/`
|
|
||||||
- Health: `http://127.0.0.1:8095/health`
|
|
||||||
- Status: `http://127.0.0.1:8095/api/status`
|
|
||||||
- Systemd service: `importarr.service`
|
|
||||||
- Env file: `/etc/importarr/importarr.env`
|
|
||||||
- SQLite state: `/var/lib/importarr/importarr.db`
|
|
||||||
|
|
||||||
## Secret Handling
|
|
||||||
|
|
||||||
- Prefer `*_FILE` settings for secrets, for example:
|
|
||||||
- `IMPORTARR_SAB_API_KEY_FILE`
|
|
||||||
- `IMPORTARR_AUTH_TOKEN_FILE`
|
|
||||||
- `IMPORTARR_RADARR_API_KEY_FILE`
|
|
||||||
- `IMPORTARR_SONARR_API_KEY_FILE`
|
|
||||||
- Never commit real env files, API keys, tokens, private keys, service databases, or backup data.
|
|
||||||
- Template files may list variable names with placeholder values or commented examples only.
|
|
||||||
|
|
||||||
## Linux Ops Follow-Through
|
|
||||||
|
|
||||||
For changes that materially alter the live service setup, ports, routes, monitoring, backup coverage, or host ownership, also follow the linux-ops documentation/systems-overview update rules. Do not mix unrelated pre-existing uncommitted changes from `linux-ops-docs` or `systems-overview` into Importarr commits.
|
|
||||||
@@ -1,9 +1,11 @@
|
|||||||
PYTHON ?= .venv/bin/python
|
PYTHON ?= .venv/bin/python
|
||||||
PIP ?= .venv/bin/pip
|
PIP ?= .venv/bin/pip
|
||||||
SERVICE ?= importarr.service
|
SERVICE ?= importarr.service
|
||||||
LIVE_URL ?= http://127.0.0.1:8095
|
IMPORTARR_PREFIX ?= /opt/importarr
|
||||||
|
IMPORTARR_REPO_DIR ?= $(CURDIR)
|
||||||
|
IMPORTARR_URL ?= http://127.0.0.1:8765
|
||||||
|
|
||||||
.PHONY: test install-systemd install-from-repo upgrade-local repo-upgrade verify-live
|
.PHONY: test install-systemd install-from-repo repo-upgrade verify
|
||||||
|
|
||||||
test:
|
test:
|
||||||
$(PYTHON) -m pytest
|
$(PYTHON) -m pytest
|
||||||
@@ -12,16 +14,12 @@ install-systemd:
|
|||||||
sudo -n sh deploy/systemd-install.sh
|
sudo -n sh deploy/systemd-install.sh
|
||||||
|
|
||||||
install-from-repo:
|
install-from-repo:
|
||||||
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
|
sudo -n $(IMPORTARR_PREFIX)/venv/bin/pip install --upgrade $(IMPORTARR_REPO_DIR)
|
||||||
|
|
||||||
upgrade-local:
|
|
||||||
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
|
|
||||||
sudo -n systemctl restart $(SERVICE)
|
|
||||||
|
|
||||||
repo-upgrade:
|
repo-upgrade:
|
||||||
sudo -n sh /opt/importarr/repo-upgrade.sh
|
sudo -n IMPORTARR_PREFIX=$(IMPORTARR_PREFIX) IMPORTARR_REPO_DIR=$(IMPORTARR_REPO_DIR) sh deploy/repo-upgrade.sh
|
||||||
|
|
||||||
verify-live:
|
verify:
|
||||||
curl -fsS $(LIVE_URL)/health
|
curl -fsS $(IMPORTARR_URL)/health
|
||||||
curl -fsS $(LIVE_URL)/api/status
|
curl -fsS $(IMPORTARR_URL)/api/status
|
||||||
curl -fsS $(LIVE_URL)/api/preview
|
curl -fsS $(IMPORTARR_URL)/api/preview
|
||||||
|
|||||||
@@ -2,17 +2,17 @@
|
|||||||
|
|
||||||
Importarr is an Arr-style service for importing manually categorized SABnzbd downloads after SAB reports final completion. It owns one SAB category, defaults to `manual`, and refuses to import transient Direct Unpack paths or jobs still in SAB queue/post-processing.
|
Importarr is an Arr-style service for importing manually categorized SABnzbd downloads after SAB reports final completion. It owns one SAB category, defaults to `manual`, and refuses to import transient Direct Unpack paths or jobs still in SAB queue/post-processing.
|
||||||
|
|
||||||
## New-machine install
|
## Install
|
||||||
|
|
||||||
Importarr is intended to feel like a small Arr service: deploy the container or systemd service, edit one env file, point SABnzbd category `manual` at the same completed-download path, then open the web UI.
|
Importarr is intended to feel like a small Arr service: deploy the container or systemd service, edit one env file, point SABnzbd category `manual` at the same completed-download path, then open the web UI.
|
||||||
|
|
||||||
### Docker Compose, recommended
|
### Docker Compose, recommended
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
mkdir -p /opt/importarr/config
|
mkdir -p importarr/config
|
||||||
cd /opt/importarr
|
cd importarr
|
||||||
curl -fsSLO https://gitea.delphas.dk/daniels/importarr/raw/branch/main/deploy/docker-compose.example.yml
|
curl -fsSLO https://example.com/importarr/deploy/docker-compose.example.yml
|
||||||
curl -fsSLo importarr.env https://gitea.delphas.dk/daniels/importarr/raw/branch/main/deploy/importarr.env.example
|
curl -fsSLo importarr.env https://example.com/importarr/deploy/importarr.env.example
|
||||||
${EDITOR:-vi} importarr.env
|
${EDITOR:-vi} importarr.env
|
||||||
docker compose -f docker-compose.example.yml --env-file importarr.env up -d
|
docker compose -f docker-compose.example.yml --env-file importarr.env up -d
|
||||||
```
|
```
|
||||||
@@ -26,25 +26,14 @@ docker build -t importarr:local .
|
|||||||
### systemd / pip install
|
### systemd / pip install
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
git clone https://gitea.delphas.dk/daniels/importarr.git
|
git clone https://example.com/importarr.git
|
||||||
cd importarr
|
cd importarr
|
||||||
sudo sh deploy/systemd-install.sh
|
sudo sh deploy/systemd-install.sh
|
||||||
sudo ${EDITOR:-vi} /etc/importarr/importarr.env
|
sudo ${EDITOR:-vi} /etc/importarr/importarr.env
|
||||||
sudo systemctl start importarr.service
|
sudo systemctl start importarr.service
|
||||||
```
|
```
|
||||||
|
|
||||||
The installer creates the `importarr` system user when needed, installs a virtualenv at `/opt/importarr/venv`, and installs the package from the checked-out repository. The repository is therefore the source of truth: pull or edit the repo, reinstall/restart from the repo, and the service runs the package built from that code.
|
The installer creates the `importarr` system user when needed, installs a virtualenv, and installs the package from the checked-out repository. Override install paths with `IMPORTARR_*` variables if the defaults do not fit your environment.
|
||||||
|
|
||||||
For local upgrades from a checked-out repo on dgsserver1, use the repo workflow instead of editing live scripts:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cd /srv/opencode-workspace/importarr
|
|
||||||
.venv/bin/python -m pytest
|
|
||||||
git status --short --branch
|
|
||||||
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
|
|
||||||
sudo -n systemctl restart importarr.service
|
|
||||||
make verify-live
|
|
||||||
```
|
|
||||||
|
|
||||||
For a machine that should stay current with the repository, use the installed repo-upgrade helper:
|
For a machine that should stay current with the repository, use the installed repo-upgrade helper:
|
||||||
|
|
||||||
@@ -54,7 +43,7 @@ sudo -n sh /opt/importarr/repo-upgrade.sh
|
|||||||
|
|
||||||
The helper refuses to run when the checkout has uncommitted changes, then performs `git pull --ff-only`, reinstalls the package from the repo, restarts `importarr.service`, and prints service status. Use it after changes have been committed and pushed to `main`.
|
The helper refuses to run when the checkout has uncommitted changes, then performs `git pull --ff-only`, reinstalls the package from the repo, restarts `importarr.service`, and prints service status. Use it after changes have been committed and pushed to `main`.
|
||||||
|
|
||||||
Release-worthy changes should be committed, tagged with SemVer (`v0.1.1`, `v0.2.0`, ...), pushed with tags, then reinstalled from the tagged checkout or artifact. Do not hand-edit `/usr/local/sbin/importarr-status.py` or `/usr/local/sbin/manual-media-import.py` except for a documented emergency hotfix that is immediately backported here.
|
Release-worthy changes should be committed, tagged with SemVer (`v0.1.1`, `v0.2.0`, ...), pushed with tags, then installed from the tagged checkout or artifact.
|
||||||
|
|
||||||
### Required setup
|
### Required setup
|
||||||
|
|
||||||
@@ -96,46 +85,15 @@ pytest
|
|||||||
uvicorn importarr.main:app --reload
|
uvicorn importarr.main:app --reload
|
||||||
```
|
```
|
||||||
|
|
||||||
## Migration notes for dgsserver1
|
## Operations
|
||||||
|
|
||||||
Export the existing script settings into `IMPORTARR_*` env vars, add historical folders as explicit manual batches, run a dry-run/inspection through `/api/preview`, then switch the systemd service or Compose route after the ready set matches expectations.
|
Importarr intentionally does not document private deployment topology, hostnames, reverse proxies, monitoring, backups, or operator workflows in this repository. Keep those details in your own ops runbooks.
|
||||||
|
|
||||||
On dgsserver1, the packaged service is the only intended active entrypoint after cutover. Keep `manual-media-import.timer` disabled unless a repo-managed worker/timer replaces it later.
|
For a systemd install, prefer a normal package install from the checked-out repo over an editable install so the service user does not need read access to your development checkout.
|
||||||
|
|
||||||
## Repository-as-install workflow
|
If the service fails, standard systemd diagnostics are usually enough:
|
||||||
|
|
||||||
Importarr should not drift into host-local scripts. Treat the checked-out repository as the install source:
|
|
||||||
|
|
||||||
1. Make changes in `/srv/opencode-workspace/importarr`.
|
|
||||||
2. Run tests: `make test`.
|
|
||||||
3. Commit and push the repo change.
|
|
||||||
4. Install/restart from the same repo: `make upgrade-local` for local changes, or `make repo-upgrade` to pull the latest pushed `main` and restart.
|
|
||||||
5. Verify the live service: `make verify-live`.
|
|
||||||
|
|
||||||
Do not edit `/usr/local/sbin/importarr-status.py`, `/usr/local/sbin/manual-media-import.py`, or files copied out of the repo as the normal workflow. If an emergency live hotfix is unavoidable, backport it to this repository immediately and run the repo install workflow again.
|
|
||||||
|
|
||||||
### Local install lessons learned
|
|
||||||
|
|
||||||
- The live systemd service runs as the unprivileged `importarr` user.
|
|
||||||
- Do not install the system service with `pip install --editable /srv/opencode-workspace/importarr`; that can fail at startup if the service user cannot read the workspace checkout.
|
|
||||||
- The supported local service install is a normal package install from the repo into `/opt/importarr/venv`:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
|
sudo systemctl --no-pager --full status importarr.service
|
||||||
sudo -n systemctl restart importarr.service
|
sudo journalctl -u importarr.service -n 120 --no-pager
|
||||||
```
|
|
||||||
|
|
||||||
- `deploy/systemd-install.sh`, `make upgrade-local`, and `/opt/importarr/repo-upgrade.sh` already use this supported model.
|
|
||||||
- After every implementation task that should affect the live local service, run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
make upgrade-local
|
|
||||||
make verify-live
|
|
||||||
```
|
|
||||||
|
|
||||||
- If `make verify-live` fails, check:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
sudo -n systemctl --no-pager --full status importarr.service
|
|
||||||
sudo -n journalctl -u importarr.service -n 120 --no-pager
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ services:
|
|||||||
- "8765:8765"
|
- "8765:8765"
|
||||||
volumes:
|
volumes:
|
||||||
- ./config:/config
|
- ./config:/config
|
||||||
- /srv/scrypted/sabnzbd-data/downloads/manual:/data/downloads/manual
|
- /path/to/downloads/manual:/data/downloads/manual
|
||||||
- /path/to/movies:/data/movies
|
- /path/to/movies:/data/movies
|
||||||
- /path/to/tv:/data/tv
|
- /path/to/tv:/data/tv
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|||||||
@@ -1,16 +1,16 @@
|
|||||||
IMPORTARR_SAB_URL=http://127.0.0.1:8080
|
IMPORTARR_SAB_URL=http://sabnzbd:8080
|
||||||
# Prefer *_FILE for secrets. Plain env vars still work for local/dev installs.
|
# Prefer *_FILE for secrets. Plain env vars still work for local/dev installs.
|
||||||
# IMPORTARR_SAB_API_KEY=change-me
|
# IMPORTARR_SAB_API_KEY=change-me
|
||||||
# IMPORTARR_SAB_API_KEY_FILE=/etc/importarr/sab-api-key
|
# IMPORTARR_SAB_API_KEY_FILE=/etc/importarr/sab-api-key
|
||||||
IMPORTARR_SAB_CATEGORY=manual
|
IMPORTARR_SAB_CATEGORY=manual
|
||||||
IMPORTARR_DOWNLOAD_ROOT=/srv/scrypted/sabnzbd-data/downloads/manual
|
IMPORTARR_DOWNLOAD_ROOT=/data/downloads/manual
|
||||||
IMPORTARR_MOVIES_ROOT=/srv/media/movies
|
IMPORTARR_MOVIES_ROOT=/data/movies
|
||||||
IMPORTARR_TV_ROOT=/srv/media/tv
|
IMPORTARR_TV_ROOT=/data/tv
|
||||||
IMPORTARR_STATE_PATH=/var/lib/importarr/importarr.db
|
IMPORTARR_STATE_PATH=/config/importarr.db
|
||||||
IMPORTARR_LOG_LEVEL=info
|
IMPORTARR_LOG_LEVEL=info
|
||||||
# IMPORTARR_AUTH_TOKEN=change-me
|
# IMPORTARR_AUTH_TOKEN=change-me
|
||||||
# IMPORTARR_AUTH_TOKEN_FILE=/etc/importarr/auth-token
|
# IMPORTARR_AUTH_TOKEN_FILE=/etc/importarr/auth-token
|
||||||
IMPORTARR_BIND_HOST=0.0.0.0
|
IMPORTARR_BIND_HOST=0.0.0.0
|
||||||
IMPORTARR_BIND_PORT=8095
|
IMPORTARR_BIND_PORT=8765
|
||||||
IMPORTARR_POLL_SECONDS=60
|
IMPORTARR_POLL_SECONDS=60
|
||||||
IMPORTARR_REPO_DIR=/srv/opencode-workspace/importarr
|
# IMPORTARR_REPO_DIR=/path/to/importarr
|
||||||
|
|||||||
@@ -1,16 +1,15 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=Importarr manual media importer
|
Description=Importarr manual media importer status UI
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
Wants=network-online.target
|
Wants=network-online.target
|
||||||
|
|
||||||
[Service]
|
[Service]
|
||||||
EnvironmentFile=/etc/importarr/importarr.env
|
EnvironmentFile=-/etc/importarr/importarr.env
|
||||||
ExecStart=/opt/importarr/venv/bin/importarr
|
EnvironmentFile=-/opt/importarr/build.env
|
||||||
|
ExecStart=/opt/importarr/venv/bin/importarr-status
|
||||||
Restart=on-failure
|
Restart=on-failure
|
||||||
RestartSec=5s
|
RestartSec=5s
|
||||||
User=importarr
|
User=root
|
||||||
Group=importarr
|
|
||||||
StateDirectory=importarr
|
|
||||||
|
|
||||||
[Install]
|
[Install]
|
||||||
WantedBy=multi-user.target
|
WantedBy=multi-user.target
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Critical alert when manual media importer fails
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=/usr/bin/logger -t importarr "manual-media-import.service failed; check journalctl -u manual-media-import.service"
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Import manual SABnzbd media into Jellyfin library roots
|
||||||
|
Wants=network-online.target
|
||||||
|
After=network-online.target
|
||||||
|
OnFailure=manual-media-import-failure.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
EnvironmentFile=-/etc/importarr/importarr.env
|
||||||
|
EnvironmentFile=-/opt/importarr/build.env
|
||||||
|
ExecStart=/opt/importarr/venv/bin/manual-media-import
|
||||||
|
TimeoutStartSec=30min
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Run manual media importer periodically
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=5min
|
||||||
|
OnUnitActiveSec=15min
|
||||||
|
AccuracySec=1min
|
||||||
|
Persistent=true
|
||||||
|
Unit=manual-media-import.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
+10
-2
@@ -12,9 +12,10 @@ if [ -f "$ENV_FILE" ]; then
|
|||||||
. "$ENV_FILE"
|
. "$ENV_FILE"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
REPO_DIR=${IMPORTARR_REPO_DIR:-/srv/opencode-workspace/importarr}
|
PREFIX=${IMPORTARR_PREFIX:-/opt/importarr}
|
||||||
|
REPO_DIR=${IMPORTARR_REPO_DIR:-$(pwd)}
|
||||||
SERVICE=${IMPORTARR_SERVICE:-importarr.service}
|
SERVICE=${IMPORTARR_SERVICE:-importarr.service}
|
||||||
VENV=${IMPORTARR_VENV:-/opt/importarr/venv}
|
VENV=${IMPORTARR_VENV:-$PREFIX/venv}
|
||||||
|
|
||||||
if [ ! -d "$REPO_DIR/.git" ]; then
|
if [ ! -d "$REPO_DIR/.git" ]; then
|
||||||
echo "Importarr repo not found at $REPO_DIR" >&2
|
echo "Importarr repo not found at $REPO_DIR" >&2
|
||||||
@@ -31,5 +32,12 @@ fi
|
|||||||
git fetch --prune origin
|
git fetch --prune origin
|
||||||
git pull --ff-only
|
git pull --ff-only
|
||||||
"$VENV/bin/pip" install --upgrade "$REPO_DIR"
|
"$VENV/bin/pip" install --upgrade "$REPO_DIR"
|
||||||
|
GIT_SHA="$(git rev-parse --short=12 HEAD 2>/dev/null || printf development)"
|
||||||
|
BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
cat > "$PREFIX/build.env" <<EOF
|
||||||
|
IMPORTARR_GIT_SHA=$GIT_SHA
|
||||||
|
IMPORTARR_BUILD_DATE=$BUILD_DATE
|
||||||
|
EOF
|
||||||
systemctl restart "$SERVICE"
|
systemctl restart "$SERVICE"
|
||||||
|
systemctl restart manual-media-import.timer
|
||||||
systemctl --no-pager --full status "$SERVICE"
|
systemctl --no-pager --full status "$SERVICE"
|
||||||
|
|||||||
@@ -6,13 +6,9 @@ if [ "$(id -u)" -ne 0 ]; then
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
install -d -m 0755 /etc/importarr /var/lib/importarr
|
install -d -m 0755 /etc/importarr /var/lib/importarr /run/manual-media-import
|
||||||
install -d -m 0755 /opt/importarr
|
install -d -m 0755 /opt/importarr
|
||||||
REPO_DIR="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)"
|
REPO_DIR="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)"
|
||||||
if ! id importarr >/dev/null 2>&1; then
|
|
||||||
useradd --system --home /var/lib/importarr --shell /usr/sbin/nologin importarr
|
|
||||||
fi
|
|
||||||
chown importarr:importarr /var/lib/importarr
|
|
||||||
python3 -m venv /opt/importarr/venv
|
python3 -m venv /opt/importarr/venv
|
||||||
/opt/importarr/venv/bin/pip install --upgrade pip
|
/opt/importarr/venv/bin/pip install --upgrade pip
|
||||||
/opt/importarr/venv/bin/pip install --upgrade "$REPO_DIR"
|
/opt/importarr/venv/bin/pip install --upgrade "$REPO_DIR"
|
||||||
@@ -24,7 +20,19 @@ if ! grep -q '^IMPORTARR_REPO_DIR=' /etc/importarr/importarr.env; then
|
|||||||
printf '\nIMPORTARR_REPO_DIR=%s\n' "$REPO_DIR" >> /etc/importarr/importarr.env
|
printf '\nIMPORTARR_REPO_DIR=%s\n' "$REPO_DIR" >> /etc/importarr/importarr.env
|
||||||
fi
|
fi
|
||||||
install -m 0644 "$REPO_DIR/deploy/importarr.service" /etc/systemd/system/importarr.service
|
install -m 0644 "$REPO_DIR/deploy/importarr.service" /etc/systemd/system/importarr.service
|
||||||
|
install -m 0644 "$REPO_DIR/deploy/manual-media-import.service" /etc/systemd/system/manual-media-import.service
|
||||||
|
install -m 0644 "$REPO_DIR/deploy/manual-media-import.timer" /etc/systemd/system/manual-media-import.timer
|
||||||
|
install -m 0644 "$REPO_DIR/deploy/manual-media-import-failure.service" /etc/systemd/system/manual-media-import-failure.service
|
||||||
install -m 0755 "$REPO_DIR/deploy/repo-upgrade.sh" /opt/importarr/repo-upgrade.sh
|
install -m 0755 "$REPO_DIR/deploy/repo-upgrade.sh" /opt/importarr/repo-upgrade.sh
|
||||||
|
GIT_SHA="$(git -C "$REPO_DIR" rev-parse --short=12 HEAD 2>/dev/null || printf development)"
|
||||||
|
BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
cat > /opt/importarr/build.env <<EOF
|
||||||
|
IMPORTARR_GIT_SHA=$GIT_SHA
|
||||||
|
IMPORTARR_BUILD_DATE=$BUILD_DATE
|
||||||
|
EOF
|
||||||
systemctl daemon-reload
|
systemctl daemon-reload
|
||||||
systemctl enable importarr.service
|
systemctl enable importarr.service
|
||||||
echo "Edit /etc/importarr/importarr.env, then run: systemctl start importarr.service"
|
systemctl enable manual-media-import.timer
|
||||||
|
systemctl restart importarr.service
|
||||||
|
systemctl start manual-media-import.timer
|
||||||
|
echo "Importarr installed from $REPO_DIR. Future upgrades: sudo -n sh /opt/importarr/repo-upgrade.sh"
|
||||||
|
|||||||
+20
-5
@@ -4,6 +4,7 @@ import os
|
|||||||
import shutil
|
import shutil
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import Callable
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -13,6 +14,10 @@ class ImportResult:
|
|||||||
bytes: int
|
bytes: int
|
||||||
|
|
||||||
|
|
||||||
|
class ImportCancelled(Exception):
|
||||||
|
"""Raised when an import is cancelled at a safe copy boundary."""
|
||||||
|
|
||||||
|
|
||||||
class Importer:
|
class Importer:
|
||||||
def __init__(self, movies_root: Path, tv_root: Path):
|
def __init__(self, movies_root: Path, tv_root: Path):
|
||||||
self.movies_root = movies_root
|
self.movies_root = movies_root
|
||||||
@@ -22,14 +27,24 @@ class Importer:
|
|||||||
target_root = self.tv_root if _looks_like_tv(source) else self.movies_root
|
target_root = self.tv_root if _looks_like_tv(source) else self.movies_root
|
||||||
return _unique_path(target_root / source.name)
|
return _unique_path(target_root / source.name)
|
||||||
|
|
||||||
def import_file(self, source: Path) -> ImportResult:
|
def import_file(self, source: Path, should_cancel: Callable[[], bool] | None = None) -> ImportResult:
|
||||||
target = self.target_for(source)
|
target = self.target_for(source)
|
||||||
target.parent.mkdir(parents=True, exist_ok=True)
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
partial = target.with_name(target.name + ".partial")
|
partial = target.with_name(target.name + ".partial")
|
||||||
with source.open("rb") as src, partial.open("wb") as dst:
|
try:
|
||||||
shutil.copyfileobj(src, dst, length=1024 * 1024)
|
with source.open("rb") as src, partial.open("wb") as dst:
|
||||||
dst.flush()
|
while True:
|
||||||
os.fsync(dst.fileno())
|
if should_cancel and should_cancel():
|
||||||
|
raise ImportCancelled("import cancelled")
|
||||||
|
chunk = src.read(1024 * 1024)
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
dst.write(chunk)
|
||||||
|
dst.flush()
|
||||||
|
os.fsync(dst.fileno())
|
||||||
|
except ImportCancelled:
|
||||||
|
partial.unlink(missing_ok=True)
|
||||||
|
raise
|
||||||
if partial.stat().st_size != source.stat().st_size:
|
if partial.stat().st_size != source.stat().st_size:
|
||||||
raise IOError("partial copy size mismatch")
|
raise IOError("partial copy size mismatch")
|
||||||
partial.rename(target)
|
partial.rename(target)
|
||||||
|
|||||||
+105
-8
@@ -12,7 +12,7 @@ from pydantic import BaseModel
|
|||||||
|
|
||||||
from .build_info import build_info
|
from .build_info import build_info
|
||||||
from .config import Settings
|
from .config import Settings
|
||||||
from .importer import Importer
|
from .importer import ImportCancelled, Importer
|
||||||
from .sabnzbd import SabnzbdClient
|
from .sabnzbd import SabnzbdClient
|
||||||
from .readiness import classify_history_item
|
from .readiness import classify_history_item
|
||||||
from .scanner import scan_videos
|
from .scanner import scan_videos
|
||||||
@@ -33,6 +33,10 @@ class RunNowRequest(BaseModel):
|
|||||||
force: bool = False
|
force: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class QueueControlRequest(BaseModel):
|
||||||
|
mode: str
|
||||||
|
|
||||||
|
|
||||||
def require_write_auth(authorization: Annotated[str | None, Header()] = None) -> None:
|
def require_write_auth(authorization: Annotated[str | None, Header()] = None) -> None:
|
||||||
if not settings.auth_token:
|
if not settings.auth_token:
|
||||||
return
|
return
|
||||||
@@ -53,6 +57,7 @@ def index(request: Request) -> HTMLResponse:
|
|||||||
@app.get("/api/status")
|
@app.get("/api/status")
|
||||||
def status() -> dict[str, object]:
|
def status() -> dict[str, object]:
|
||||||
history = state.list_history()
|
history = state.list_history()
|
||||||
|
control = control_status()
|
||||||
return {
|
return {
|
||||||
"app": "Importarr",
|
"app": "Importarr",
|
||||||
"build": build_info(),
|
"build": build_info(),
|
||||||
@@ -66,13 +71,81 @@ def status() -> dict[str, object]:
|
|||||||
"manual_batches": len(state.list_manual_batches(active_only=True)),
|
"manual_batches": len(state.list_manual_batches(active_only=True)),
|
||||||
"imported_total": sum(1 for row in history if row["status"] == "imported"),
|
"imported_total": sum(1 for row in history if row["status"] == "imported"),
|
||||||
"failed_total": sum(1 for row in history if row["status"] == "failed"),
|
"failed_total": sum(1 for row in history if row["status"] == "failed"),
|
||||||
"current": None,
|
"current": control["current"],
|
||||||
|
"control": control,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def control_status() -> dict[str, object]:
|
||||||
|
mode = state.get_app_state("queue_mode", "running") or "running"
|
||||||
|
current = state.get_app_state("current_job")
|
||||||
|
cancel_requested = (state.get_app_state("cancel_requested", "false") or "false") == "true"
|
||||||
|
return {
|
||||||
|
"queue_mode": mode,
|
||||||
|
"queue_accepting_new_jobs": mode == "running",
|
||||||
|
"cancel_requested": cancel_requested,
|
||||||
|
"current": current,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def queue_accepting_new_jobs() -> bool:
|
||||||
|
return (state.get_app_state("queue_mode", "running") or "running") == "running"
|
||||||
|
|
||||||
|
|
||||||
|
def cancel_requested() -> bool:
|
||||||
|
return (state.get_app_state("cancel_requested", "false") or "false") == "true"
|
||||||
|
|
||||||
|
|
||||||
|
def consume_cancel_request() -> bool:
|
||||||
|
if not cancel_requested():
|
||||||
|
return False
|
||||||
|
state.set_app_state("cancel_requested", "false")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def set_current_job(name: str | None) -> None:
|
||||||
|
state.set_app_state("current_job", name or "")
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/control/queue")
|
||||||
|
def set_queue_control(payload: QueueControlRequest, _: None = Depends(require_write_auth)) -> dict[str, object]:
|
||||||
|
if payload.mode not in {"running", "paused", "stopped"}:
|
||||||
|
raise HTTPException(status_code=400, detail="mode must be running, paused, or stopped")
|
||||||
|
state.set_app_state("queue_mode", payload.mode)
|
||||||
|
if payload.mode == "running":
|
||||||
|
state.set_app_state("cancel_requested", "false")
|
||||||
|
return control_status()
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/control/start")
|
||||||
|
def start_queue(_: None = Depends(require_write_auth)) -> dict[str, object]:
|
||||||
|
state.set_app_state("queue_mode", "running")
|
||||||
|
state.set_app_state("cancel_requested", "false")
|
||||||
|
return control_status()
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/control/pause")
|
||||||
|
def pause_queue(_: None = Depends(require_write_auth)) -> dict[str, object]:
|
||||||
|
state.set_app_state("queue_mode", "paused")
|
||||||
|
return control_status()
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/control/stop")
|
||||||
|
def stop_queue(_: None = Depends(require_write_auth)) -> dict[str, object]:
|
||||||
|
state.set_app_state("queue_mode", "stopped")
|
||||||
|
return control_status()
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/control/cancel-current")
|
||||||
|
def cancel_current(_: None = Depends(require_write_auth)) -> dict[str, object]:
|
||||||
|
state.set_app_state("cancel_requested", "true")
|
||||||
|
return control_status()
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/manual-batches")
|
@app.get("/api/manual-batches")
|
||||||
def manual_batches() -> list[dict[str, object]]:
|
def manual_batches() -> list[dict[str, object]]:
|
||||||
sync_manual_queue()
|
if queue_accepting_new_jobs():
|
||||||
|
sync_manual_queue()
|
||||||
rows = []
|
rows = []
|
||||||
for batch in state.list_manual_batches():
|
for batch in state.list_manual_batches():
|
||||||
videos = [item for item in state.list_queue_items() if item["batch_id"] == batch["id"]] if batch["status"] == "active" else []
|
videos = [item for item in state.list_queue_items() if item["batch_id"] == batch["id"]] if batch["status"] == "active" else []
|
||||||
@@ -107,9 +180,10 @@ async def jobs() -> dict[str, object]:
|
|||||||
|
|
||||||
@app.get("/api/preview")
|
@app.get("/api/preview")
|
||||||
async def preview() -> dict[str, object]:
|
async def preview() -> dict[str, object]:
|
||||||
await sync_queue()
|
if queue_accepting_new_jobs():
|
||||||
|
await sync_queue()
|
||||||
jobs = queue_jobs()
|
jobs = queue_jobs()
|
||||||
return {"sab_status": "ok", "jobs": jobs, "would_import": sum(1 for row in jobs if row["state"] in {"ready", "manual_batch"})}
|
return {"sab_status": "ok", "jobs": jobs, "would_import": sum(1 for row in jobs if row["state"] in {"ready", "manual_batch"}), "control": control_status()}
|
||||||
|
|
||||||
|
|
||||||
async def sync_queue() -> None:
|
async def sync_queue() -> None:
|
||||||
@@ -156,6 +230,8 @@ def manual_batch_jobs() -> list[dict[str, object]]:
|
|||||||
|
|
||||||
|
|
||||||
def sync_manual_queue() -> None:
|
def sync_manual_queue() -> None:
|
||||||
|
if not queue_accepting_new_jobs():
|
||||||
|
return
|
||||||
root = settings.download_root.resolve()
|
root = settings.download_root.resolve()
|
||||||
for batch in state.list_manual_batches(active_only=True):
|
for batch in state.list_manual_batches(active_only=True):
|
||||||
seen: set[str] = set()
|
seen: set[str] = set()
|
||||||
@@ -185,37 +261,58 @@ async def _import_ready_sab_jobs(importer: Importer, force: bool = False) -> int
|
|||||||
return 0
|
return 0
|
||||||
imported = 0
|
imported = 0
|
||||||
for item in data.get("history", {}).get("slots", []):
|
for item in data.get("history", {}).get("slots", []):
|
||||||
|
if consume_cancel_request():
|
||||||
|
break
|
||||||
readiness = classify_history_item(item, active, settings.sab_category, settings.download_root, force_status=force)
|
readiness = classify_history_item(item, active, settings.sab_category, settings.download_root, force_status=force)
|
||||||
if readiness.storage is None or (not readiness.ready and not force):
|
if readiness.storage is None or (not readiness.ready and not force):
|
||||||
continue
|
continue
|
||||||
for video in scan_videos(readiness.storage):
|
for video in scan_videos(readiness.storage):
|
||||||
|
if consume_cancel_request():
|
||||||
|
return imported
|
||||||
|
set_current_job(str(video.path))
|
||||||
try:
|
try:
|
||||||
result = importer.import_file(video.path)
|
result = importer.import_file(video.path, should_cancel=consume_cancel_request)
|
||||||
state.add_history(result.source, result.target, "imported", result.bytes)
|
state.add_history(result.source, result.target, "imported", result.bytes)
|
||||||
state.mark_queue_item("sab", str(video.path), "imported")
|
state.mark_queue_item("sab", str(video.path), "imported")
|
||||||
imported += 1
|
imported += 1
|
||||||
|
except ImportCancelled:
|
||||||
|
state.add_history(video.path, video.path, "cancelled", 0, "cancelled")
|
||||||
|
state.mark_queue_item("sab", str(video.path), "skipped", "cancelled")
|
||||||
|
return imported
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
state.add_history(video.path, video.path, "failed", 0, exc.__class__.__name__)
|
state.add_history(video.path, video.path, "failed", 0, exc.__class__.__name__)
|
||||||
state.mark_queue_item("sab", str(video.path), "failed", exc.__class__.__name__)
|
state.mark_queue_item("sab", str(video.path), "failed", exc.__class__.__name__)
|
||||||
|
finally:
|
||||||
|
set_current_job(None)
|
||||||
return imported
|
return imported
|
||||||
|
|
||||||
|
|
||||||
def _import_manual_batches(importer: Importer) -> int:
|
def _import_manual_batches(importer: Importer) -> int:
|
||||||
sync_manual_queue()
|
if queue_accepting_new_jobs():
|
||||||
|
sync_manual_queue()
|
||||||
imported = 0
|
imported = 0
|
||||||
for batch in state.list_manual_batches(active_only=True):
|
for batch in state.list_manual_batches(active_only=True):
|
||||||
path = Path(batch["path"])
|
path = Path(batch["path"])
|
||||||
items = [item for item in state.list_queue_items() if item["source_type"] == "manual" and item["batch_id"] == batch["id"]]
|
items = [item for item in state.list_queue_items() if item["source_type"] == "manual" and item["batch_id"] == batch["id"]]
|
||||||
for item in items:
|
for item in items:
|
||||||
|
if consume_cancel_request():
|
||||||
|
return imported
|
||||||
source = Path(item["source_path"])
|
source = Path(item["source_path"])
|
||||||
|
set_current_job(str(source))
|
||||||
try:
|
try:
|
||||||
result = importer.import_file(source)
|
result = importer.import_file(source, should_cancel=consume_cancel_request)
|
||||||
state.add_history(result.source, result.target, "imported", result.bytes)
|
state.add_history(result.source, result.target, "imported", result.bytes)
|
||||||
state.mark_queue_item("manual", item["source_id"], "imported")
|
state.mark_queue_item("manual", item["source_id"], "imported")
|
||||||
imported += 1
|
imported += 1
|
||||||
|
except ImportCancelled:
|
||||||
|
state.add_history(source, source, "cancelled", 0, "cancelled")
|
||||||
|
state.mark_queue_item("manual", item["source_id"], "skipped", "cancelled")
|
||||||
|
return imported
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
state.add_history(source, source, "failed", 0, exc.__class__.__name__)
|
state.add_history(source, source, "failed", 0, exc.__class__.__name__)
|
||||||
state.mark_queue_item("manual", item["source_id"], "failed", exc.__class__.__name__)
|
state.mark_queue_item("manual", item["source_id"], "failed", exc.__class__.__name__)
|
||||||
|
finally:
|
||||||
|
set_current_job(None)
|
||||||
if not scan_videos(path):
|
if not scan_videos(path):
|
||||||
state.complete_manual_batch(batch["id"])
|
state.complete_manual_batch(batch["id"])
|
||||||
return imported
|
return imported
|
||||||
|
|||||||
@@ -55,6 +55,17 @@ class State:
|
|||||||
)
|
)
|
||||||
self.conn.commit()
|
self.conn.commit()
|
||||||
|
|
||||||
|
def get_app_state(self, key: str, default: str | None = None) -> str | None:
|
||||||
|
row = self.conn.execute("select value from app_state where key = ?", (key,)).fetchone()
|
||||||
|
return row["value"] if row else default
|
||||||
|
|
||||||
|
def set_app_state(self, key: str, value: str) -> None:
|
||||||
|
self.conn.execute(
|
||||||
|
"insert into app_state(key, value) values (?, ?) on conflict(key) do update set value=excluded.value",
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
self.conn.commit()
|
||||||
|
|
||||||
def add_manual_batch(self, path: Path) -> dict[str, Any]:
|
def add_manual_batch(self, path: Path) -> dict[str, Any]:
|
||||||
self.conn.execute("insert or ignore into manual_batches(path) values (?)", (str(path),))
|
self.conn.execute("insert or ignore into manual_batches(path) values (?)", (str(path),))
|
||||||
self.conn.commit()
|
self.conn.commit()
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
body{font-family:system-ui,sans-serif;margin:0;background:#111827;color:#e5e7eb}header,main{max-width:1100px;margin:auto;padding:1rem}.topbar{display:flex;justify-content:space-between;gap:1rem;align-items:center;background:#0f172a}.build{text-align:right}.build strong{font-size:1.2rem}.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(12rem,1fr));gap:1rem}.cards article,.panel{background:#1f2937;border-radius:.75rem;padding:1rem;margin-top:1rem}strong{display:block;font-size:2rem}span,small,dd{color:#9ca3af}table{width:100%;border-collapse:collapse;background:#1f2937;margin-top:1rem}th,td{padding:.6rem;border-bottom:1px solid #374151;text-align:left}input,button{padding:.6rem;border-radius:.4rem;border:1px solid #374151}button{background:#38bdf8;color:#082f49;font-weight:700}.inline-form{display:flex;gap:.5rem;flex-wrap:wrap}.inline-form input[name=path]{min-width:min(100%,28rem);flex:1}.info{display:grid;grid-template-columns:10rem 1fr;gap:.4rem 1rem}.info dt{font-weight:700}.info dd{margin:0;overflow-wrap:anywhere}.state{background:#0f172a;border:1px solid #374151;border-radius:999px;padding:.15rem .5rem}
|
body{font-family:system-ui,sans-serif;margin:0;background:#111827;color:#e5e7eb}header,main{max-width:1100px;margin:auto;padding:1rem}.topbar{display:flex;justify-content:space-between;gap:1rem;align-items:center;background:#0f172a}.build{text-align:right}.build strong{font-size:1.2rem}.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(12rem,1fr));gap:1rem}.cards article,.panel{background:#1f2937;border-radius:.75rem;padding:1rem;margin-top:1rem}strong{display:block;font-size:2rem}span,small,dd{color:#9ca3af}table{width:100%;border-collapse:collapse;background:#1f2937;margin-top:1rem}th,td{padding:.6rem;border-bottom:1px solid #374151;text-align:left}input,button{padding:.6rem;border-radius:.4rem;border:1px solid #374151}button{background:#38bdf8;color:#082f49;font-weight:700}.danger{background:#f87171;color:#450a0a}.controls{display:flex;gap:.5rem;flex-wrap:wrap}.inline-form{display:flex;gap:.5rem;flex-wrap:wrap}.inline-form input[name=path]{min-width:min(100%,28rem);flex:1}.info{display:grid;grid-template-columns:10rem 1fr;gap:.4rem 1rem}.info dt{font-weight:700}.info dd{margin:0;overflow-wrap:anywhere}.state{background:#0f172a;border:1px solid #374151;border-radius:999px;padding:.15rem .5rem}
|
||||||
|
|||||||
@@ -0,0 +1,496 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import html
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
SERVICE = "manual-media-import.service"
|
||||||
|
TIMER = "manual-media-import.timer"
|
||||||
|
LOG = Path("/var/log/manual-media-import.log")
|
||||||
|
IMPORTER_STATUS = Path("/run/manual-media-import/status.json")
|
||||||
|
MANUAL_BATCHES = Path("/var/lib/importarr/manual-batches.json")
|
||||||
|
QUEUE_ROOTS = {
|
||||||
|
"manual": Path(os.getenv("IMPORTARR_DOWNLOAD_ROOT", "/data/downloads/manual")),
|
||||||
|
"legacy": Path(os.getenv("IMPORTARR_LEGACY_DOWNLOAD_ROOT", "/data/downloads/legacy")),
|
||||||
|
}
|
||||||
|
VIDEO_EXT = {".mkv", ".mp4", ".m4v", ".avi", ".mov", ".wmv", ".mpg", ".mpeg", ".ts", ".m2ts", ".webm"}
|
||||||
|
SAB_CONFIG = Path(os.getenv("IMPORTARR_SABNZBD_CONFIG", "/config/sabnzbd/sabnzbd.ini"))
|
||||||
|
SAB_API = os.getenv("IMPORTARR_SABNZBD_URL", "http://sabnzbd:8080/api")
|
||||||
|
LONG_RUNTIME_SECONDS = 25 * 60
|
||||||
|
HIGH_MEMORY_BYTES = 8 * 1024**3
|
||||||
|
STALE_QUEUE_SECONDS = 6 * 60 * 60
|
||||||
|
|
||||||
|
|
||||||
|
def run(args: list[str]) -> str:
|
||||||
|
return subprocess.run(args, check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL).stdout
|
||||||
|
|
||||||
|
|
||||||
|
def systemctl_show(unit: str) -> dict[str, str]:
|
||||||
|
data = {}
|
||||||
|
for line in run(["systemctl", "show", unit, "--no-pager"]).splitlines():
|
||||||
|
if "=" in line:
|
||||||
|
key, value = line.split("=", 1)
|
||||||
|
data[key] = value
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def int_value(value: str | None) -> int | None:
|
||||||
|
try:
|
||||||
|
return int(value or "")
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def timestamp_to_iso(usec: str | None) -> str | None:
|
||||||
|
value = int_value(usec)
|
||||||
|
if not value or value <= 0:
|
||||||
|
return None
|
||||||
|
return datetime.fromtimestamp(value / 1_000_000, tz=timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def monotonic_runtime_seconds(service: dict[str, str]) -> int | None:
|
||||||
|
started = int_value(service.get("ActiveEnterTimestampMonotonic"))
|
||||||
|
if not started:
|
||||||
|
main_pid = int_value(service.get("MainPID"))
|
||||||
|
if not main_pid:
|
||||||
|
return None
|
||||||
|
etimes = run(["ps", "-o", "etimes=", "-p", str(main_pid)]).strip()
|
||||||
|
return int_value(etimes)
|
||||||
|
boot_ns = time.clock_gettime_ns(time.CLOCK_BOOTTIME)
|
||||||
|
runtime = int((boot_ns / 1000 - started) / 1_000_000)
|
||||||
|
return max(runtime, 0)
|
||||||
|
|
||||||
|
|
||||||
|
def scan_queue(root: Path) -> dict[str, object]:
|
||||||
|
files = dirs = bytes_total = 0
|
||||||
|
processing_files = processing_dirs = processing_bytes = 0
|
||||||
|
oldest = newest = None
|
||||||
|
top_level: list[dict[str, object]] = []
|
||||||
|
if not root.exists():
|
||||||
|
return {"path": str(root), "exists": False, "files": 0, "dirs": 0, "bytes": 0, "oldest": None, "newest": None, "topLevel": [], "items": [], "processingFiles": 0, "processingDirs": 0, "processingBytes": 0, "processing": [], "processingItems": []}
|
||||||
|
top_level_map: dict[Path, dict[str, object]] = {}
|
||||||
|
processing_map: dict[Path, dict[str, object]] = {}
|
||||||
|
ready_items: list[dict[str, object]] = []
|
||||||
|
processing_items: list[dict[str, object]] = []
|
||||||
|
for dirpath, dirnames, filenames in os.walk(root):
|
||||||
|
for filename in filenames:
|
||||||
|
path = Path(dirpath) / filename
|
||||||
|
if path.suffix.lower() not in VIDEO_EXT or "sample" in filename.lower() or "sample" in str(path.parent).lower():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
st = path.stat()
|
||||||
|
except FileNotFoundError:
|
||||||
|
continue
|
||||||
|
oldest = st.st_mtime if oldest is None else min(oldest, st.st_mtime)
|
||||||
|
newest = st.st_mtime if newest is None else max(newest, st.st_mtime)
|
||||||
|
try:
|
||||||
|
rel = path.relative_to(root)
|
||||||
|
except ValueError:
|
||||||
|
rel = path
|
||||||
|
top = root / rel.parts[0] if rel.parts else path
|
||||||
|
sab = sab_state_for(top.name)
|
||||||
|
transient = top.name.startswith(('_UNPACK_', '__UNPACK__', '_FAILED_', '_ADMIN_'))
|
||||||
|
is_processing = transient or (sab and sab.get("ready") is False and sab.get("status") != "manual")
|
||||||
|
relative_dir = str(Path(*rel.parts[:-1])) if len(rel.parts) > 1 else ""
|
||||||
|
item_label = path.name if not relative_dir else f"{relative_dir} / {path.name}"
|
||||||
|
state = str(sab.get("state") if sab else ("unpacking" if transient else "ready"))
|
||||||
|
file_item = {"name": path.name, "label": item_label, "release": top.name, "relativeDir": relative_dir, "path": str(path), "bytes": st.st_size, "mtime": datetime.fromtimestamp(st.st_mtime, tz=timezone.utc).isoformat(), "state": state}
|
||||||
|
if is_processing:
|
||||||
|
processing_files += 1
|
||||||
|
processing_bytes += st.st_size
|
||||||
|
processing_items.append(file_item)
|
||||||
|
item = processing_map.setdefault(top, {"name": top.name, "type": "dir" if top.is_dir() else "file", "files": 0, "bytes": 0, "mtime": None})
|
||||||
|
item["files"] = int(item["files"]) + 1
|
||||||
|
item["bytes"] = int(item["bytes"]) + st.st_size
|
||||||
|
item["mtime"] = datetime.fromtimestamp(st.st_mtime, tz=timezone.utc).isoformat()
|
||||||
|
continue
|
||||||
|
|
||||||
|
files += 1
|
||||||
|
bytes_total += st.st_size
|
||||||
|
ready_items.append(file_item)
|
||||||
|
item = top_level_map.setdefault(top, {"name": top.name, "type": "dir" if top.is_dir() else "file", "files": 0, "bytes": 0, "mtime": None})
|
||||||
|
item["files"] = int(item["files"]) + 1
|
||||||
|
item["bytes"] = int(item["bytes"]) + st.st_size
|
||||||
|
item["mtime"] = datetime.fromtimestamp(st.st_mtime, tz=timezone.utc).isoformat()
|
||||||
|
dirs = sum(1 for item in top_level_map.values() if item["type"] == "dir")
|
||||||
|
processing_dirs = sum(1 for item in processing_map.values() if item["type"] == "dir")
|
||||||
|
top_level = sorted(top_level_map.values(), key=lambda item: str(item["name"]).lower())
|
||||||
|
processing = sorted(processing_map.values(), key=lambda item: str(item["name"]).lower())
|
||||||
|
return {
|
||||||
|
"path": str(root),
|
||||||
|
"exists": True,
|
||||||
|
"files": files,
|
||||||
|
"dirs": dirs,
|
||||||
|
"bytes": bytes_total,
|
||||||
|
"oldest": datetime.fromtimestamp(oldest, tz=timezone.utc).isoformat() if oldest else None,
|
||||||
|
"newest": datetime.fromtimestamp(newest, tz=timezone.utc).isoformat() if newest else None,
|
||||||
|
"topLevel": top_level[:100],
|
||||||
|
"items": ready_items[:500],
|
||||||
|
"processingFiles": processing_files,
|
||||||
|
"processingDirs": processing_dirs,
|
||||||
|
"processingBytes": processing_bytes,
|
||||||
|
"processing": processing[:100],
|
||||||
|
"processingItems": processing_items[:500],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def read_logs(limit: int = 100) -> list[dict[str, object]]:
|
||||||
|
if not LOG.exists():
|
||||||
|
return []
|
||||||
|
lines = LOG.read_text(errors="replace").splitlines()[-max(1, min(limit, 1000)):]
|
||||||
|
records = []
|
||||||
|
for line in lines:
|
||||||
|
try:
|
||||||
|
records.append(json.loads(line))
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
records.append({"level": "RAW", "msg": line})
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def read_summaries() -> list[dict[str, object]]:
|
||||||
|
if not LOG.exists():
|
||||||
|
return []
|
||||||
|
summaries = []
|
||||||
|
for line in LOG.read_text(errors="replace").splitlines():
|
||||||
|
try:
|
||||||
|
record = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue
|
||||||
|
if record.get("msg") == "summary":
|
||||||
|
summaries.append(record)
|
||||||
|
return summaries
|
||||||
|
|
||||||
|
|
||||||
|
def read_importer_status() -> dict[str, object] | None:
|
||||||
|
if not IMPORTER_STATUS.exists():
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def infer_current_from_logs(logs: list[dict[str, object]]) -> dict[str, object] | None:
|
||||||
|
for record in reversed(logs):
|
||||||
|
if record.get("level") != "MOVE" or record.get("msg") not in {"moving", "moving sidecar"}:
|
||||||
|
continue
|
||||||
|
src = record.get("src")
|
||||||
|
dest = record.get("dest")
|
||||||
|
if not src or not dest:
|
||||||
|
continue
|
||||||
|
src_path = Path(str(src))
|
||||||
|
partial = Path(str(dest) + ".partial")
|
||||||
|
total = None
|
||||||
|
copied = None
|
||||||
|
try:
|
||||||
|
total = src_path.stat().st_size
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
copied = partial.stat().st_size
|
||||||
|
except FileNotFoundError:
|
||||||
|
copied = None
|
||||||
|
percent = round((copied / total * 100), 2) if copied is not None and total else None
|
||||||
|
return {"phase": "copying", "src": str(src), "dest": str(dest), "partial": str(partial), "bytes_copied": copied, "bytes_total": total, "percent": percent, "media_type": record.get("media_type"), "source_tag": record.get("source_tag"), "kind": "inferred"}
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return json.loads(IMPORTER_STATUS.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, json.JSONDecodeError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def queue_status() -> dict[str, object]:
|
||||||
|
global _SAB_HISTORY_CACHE
|
||||||
|
_SAB_HISTORY_CACHE = None
|
||||||
|
roots = {name: scan_queue(path) for name, path in QUEUE_ROOTS.items()}
|
||||||
|
return {
|
||||||
|
"roots": roots,
|
||||||
|
"files": sum(int(r["files"]) for r in roots.values()),
|
||||||
|
"dirs": sum(int(r["dirs"]) for r in roots.values()),
|
||||||
|
"bytes": sum(int(r["bytes"]) for r in roots.values()),
|
||||||
|
"processingFiles": sum(int(r["processingFiles"]) for r in roots.values()),
|
||||||
|
"processingDirs": sum(int(r["processingDirs"]) for r in roots.values()),
|
||||||
|
"processingBytes": sum(int(r["processingBytes"]) for r in roots.values()),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def sab_api_key() -> str | None:
|
||||||
|
try:
|
||||||
|
import re
|
||||||
|
m = re.search(r"^api_key\s*=\s*(\S+)", SAB_CONFIG.read_text(errors="replace"), re.M)
|
||||||
|
return m.group(1) if m else None
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def sab_history() -> list[dict[str, object]]:
|
||||||
|
key = sab_api_key()
|
||||||
|
if not key:
|
||||||
|
return []
|
||||||
|
try:
|
||||||
|
q = urllib.parse.urlencode({"mode": "history", "output": "json", "limit": 200, "apikey": key})
|
||||||
|
data = json.load(urllib.request.urlopen(f"{SAB_API}?{q}", timeout=10))
|
||||||
|
return data.get("history", {}).get("slots", [])
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
_SAB_HISTORY_CACHE: list[dict[str, object]] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def sab_state_for(folder_name: str) -> dict[str, object] | None:
|
||||||
|
global _SAB_HISTORY_CACHE
|
||||||
|
if _SAB_HISTORY_CACHE is None:
|
||||||
|
_SAB_HISTORY_CACHE = sab_history()
|
||||||
|
normalized = folder_name.removeprefix("_UNPACK_").removeprefix("__UNPACK__")
|
||||||
|
for item in _SAB_HISTORY_CACHE:
|
||||||
|
name = str(item.get("name") or "")
|
||||||
|
if name != normalized and name != folder_name:
|
||||||
|
continue
|
||||||
|
status = str(item.get("status") or "")
|
||||||
|
storage = str(item.get("storage") or "")
|
||||||
|
action = str(item.get("action_line") or "")
|
||||||
|
category = str(item.get("category") or item.get("cat") or "")
|
||||||
|
owned = category == "manual"
|
||||||
|
ready = owned and status == "Completed" and bool(storage) and "_UNPACK_" not in storage
|
||||||
|
state = "ready" if ready else ("ignored category " + category if not owned else (status.lower() if status else "sab pending"))
|
||||||
|
if action:
|
||||||
|
state = action
|
||||||
|
return {"ready": ready, "owned": owned, "category": category, "status": status, "storage": storage, "state": state}
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def read_manual_batches() -> list[str]:
|
||||||
|
try:
|
||||||
|
raw = json.loads(MANUAL_BATCHES.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, json.JSONDecodeError):
|
||||||
|
return []
|
||||||
|
return [str(x) for x in raw] if isinstance(raw, list) else []
|
||||||
|
|
||||||
|
|
||||||
|
def write_manual_batches(items: list[str]) -> None:
|
||||||
|
MANUAL_BATCHES.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
MANUAL_BATCHES.write_text(json.dumps(sorted(set(items)), indent=2), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def add_manual_batch(value: str) -> tuple[bool, str]:
|
||||||
|
value = value.strip().strip("/")
|
||||||
|
if not value:
|
||||||
|
return False, "missing folder"
|
||||||
|
base = QUEUE_ROOTS["manual"].resolve()
|
||||||
|
path = (base / value).resolve() if not value.startswith("/srv/") else Path(value).resolve()
|
||||||
|
if not (path == base or path.is_relative_to(base)):
|
||||||
|
return False, "folder must be under manual downloads"
|
||||||
|
if not path.is_dir():
|
||||||
|
return False, "folder does not exist"
|
||||||
|
items = read_manual_batches()
|
||||||
|
items.append(str(path))
|
||||||
|
write_manual_batches(items)
|
||||||
|
return True, str(path)
|
||||||
|
|
||||||
|
|
||||||
|
def status() -> dict[str, object]:
|
||||||
|
service = systemctl_show(SERVICE)
|
||||||
|
timer = systemctl_show(TIMER)
|
||||||
|
logs = read_logs(300)
|
||||||
|
all_logs = read_logs(1000)
|
||||||
|
summaries = read_summaries()
|
||||||
|
last_summary = summaries[-1] if summaries else None
|
||||||
|
cutoff_1h = datetime.now() - timedelta(hours=1)
|
||||||
|
cutoff_24h = datetime.now() - timedelta(hours=24)
|
||||||
|
processed_1h = 0
|
||||||
|
processed_24h = 0
|
||||||
|
processed_total = 0
|
||||||
|
runs_1h = 0
|
||||||
|
runs_24h = 0
|
||||||
|
runs_total = 0
|
||||||
|
for summary in summaries:
|
||||||
|
moved = int(summary.get("moved") or 0)
|
||||||
|
processed_total += moved
|
||||||
|
runs_total += 1
|
||||||
|
try:
|
||||||
|
ts = datetime.fromisoformat(str(summary.get("ts")))
|
||||||
|
except ValueError:
|
||||||
|
ts = None
|
||||||
|
if ts and ts >= cutoff_1h:
|
||||||
|
processed_1h += moved
|
||||||
|
runs_1h += 1
|
||||||
|
if ts and ts >= cutoff_24h:
|
||||||
|
processed_24h += moved
|
||||||
|
runs_24h += 1
|
||||||
|
queue = queue_status()
|
||||||
|
running = service.get("ActiveState") == "activating" or service.get("SubState") in {"start", "running"}
|
||||||
|
current = read_importer_status()
|
||||||
|
if running and (not current or current.get("phase") == "done"):
|
||||||
|
current = infer_current_from_logs(all_logs)
|
||||||
|
runtime = monotonic_runtime_seconds(service) if running else None
|
||||||
|
memory_current = int_value(service.get("MemoryCurrent"))
|
||||||
|
memory_peak = int_value(service.get("MemoryPeak"))
|
||||||
|
warnings = []
|
||||||
|
if runtime and runtime > LONG_RUNTIME_SECONDS:
|
||||||
|
warnings.append("manual-media-import.service has been running longer than 25 minutes")
|
||||||
|
if memory_peak and memory_peak > HIGH_MEMORY_BYTES:
|
||||||
|
warnings.append("manual-media-import.service peak memory is over 8 GiB")
|
||||||
|
if last_summary and int(last_summary.get("errors") or 0) > 0:
|
||||||
|
warnings.append("last importer summary reported errors")
|
||||||
|
now = time.time()
|
||||||
|
for name, root in queue["roots"].items():
|
||||||
|
oldest = root.get("oldest")
|
||||||
|
if oldest:
|
||||||
|
try:
|
||||||
|
age = now - datetime.fromisoformat(str(oldest)).timestamp()
|
||||||
|
if age > STALE_QUEUE_SECONDS:
|
||||||
|
warnings.append(f"{name} queue contains files older than 6 hours")
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
health_state = "warning" if warnings else ("running" if running else service.get("Result", "unknown"))
|
||||||
|
return {
|
||||||
|
"name": "Importarr",
|
||||||
|
"service": {
|
||||||
|
"unit": SERVICE,
|
||||||
|
"activeState": service.get("ActiveState"),
|
||||||
|
"subState": service.get("SubState"),
|
||||||
|
"result": service.get("Result"),
|
||||||
|
"running": running,
|
||||||
|
"mainPid": int_value(service.get("MainPID")),
|
||||||
|
"startedAt": timestamp_to_iso(service.get("ActiveEnterTimestampUSec")),
|
||||||
|
"runtimeSeconds": runtime,
|
||||||
|
"memoryCurrentBytes": memory_current,
|
||||||
|
"memoryPeakBytes": memory_peak,
|
||||||
|
},
|
||||||
|
"timer": {
|
||||||
|
"unit": TIMER,
|
||||||
|
"activeState": timer.get("ActiveState"),
|
||||||
|
"subState": timer.get("SubState"),
|
||||||
|
"lastTrigger": timestamp_to_iso(timer.get("LastTriggerUSec")),
|
||||||
|
"nextElapse": timestamp_to_iso(timer.get("NextElapseUSecRealtime")),
|
||||||
|
},
|
||||||
|
"queue": queue,
|
||||||
|
"lastSummary": last_summary,
|
||||||
|
"processed": {"last1h": processed_1h, "last24h": processed_24h, "total": processed_total, "runsLast1h": runs_1h, "runsLast24h": runs_24h, "runsTotal": runs_total},
|
||||||
|
"current": current,
|
||||||
|
"manualBatches": read_manual_batches(),
|
||||||
|
"health": {"state": health_state, "warnings": warnings},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def fast_health() -> dict[str, object]:
|
||||||
|
service = systemctl_show(SERVICE)
|
||||||
|
timer = systemctl_show(TIMER)
|
||||||
|
running = service.get("ActiveState") == "activating" or service.get("SubState") in {"start", "running"}
|
||||||
|
runtime = monotonic_runtime_seconds(service) if running else None
|
||||||
|
warnings = []
|
||||||
|
if runtime and runtime > LONG_RUNTIME_SECONDS:
|
||||||
|
warnings.append("manual-media-import.service has been running longer than 25 minutes")
|
||||||
|
return {"state": "warning" if warnings else ("running" if running else service.get("Result", "unknown")), "warnings": warnings, "service": service.get("ActiveState"), "timer": timer.get("ActiveState")}
|
||||||
|
|
||||||
|
|
||||||
|
def page() -> bytes:
|
||||||
|
s = status()
|
||||||
|
logs = read_logs(80)
|
||||||
|
def esc(value: object) -> str:
|
||||||
|
return html.escape("—" if value is None else str(value))
|
||||||
|
def gib(value: object) -> str:
|
||||||
|
return "—" if value is None else f"{int(value) / 1024**3:.2f} GiB"
|
||||||
|
def secs(value: object) -> str:
|
||||||
|
if value is None:
|
||||||
|
return "—"
|
||||||
|
seconds = int(value)
|
||||||
|
return f"{seconds // 60}m {seconds % 60}s"
|
||||||
|
def job_name(path: object) -> str:
|
||||||
|
if not path:
|
||||||
|
return ""
|
||||||
|
name = Path(str(path)).name
|
||||||
|
return name[:37] + "..." if len(name) > 40 else name
|
||||||
|
title_icon = "📥"
|
||||||
|
title_runtime = secs(s["service"]["runtimeSeconds"])
|
||||||
|
current = s.get("current") or {}
|
||||||
|
progress = current.get("percent")
|
||||||
|
full_current_name = Path(str(current.get("src", ""))).name if current.get("src") else ""
|
||||||
|
current_src_path = Path(str(current.get("src", ""))) if current.get("src") else None
|
||||||
|
current_top_name = ""
|
||||||
|
current_relative_dir = ""
|
||||||
|
if current_src_path:
|
||||||
|
for root in QUEUE_ROOTS.values():
|
||||||
|
try:
|
||||||
|
rel = current_src_path.relative_to(root)
|
||||||
|
current_top_name = rel.parts[0] if rel.parts else current_src_path.name
|
||||||
|
current_relative_dir = str(Path(*rel.parts[:-1])) if len(rel.parts) > 1 else ""
|
||||||
|
break
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
current_name = job_name(current.get("src"))
|
||||||
|
title = f"{title_icon} {progress}% {title_runtime} - {current_name}" if progress is not None and current_name else f"{title_icon} - idle"
|
||||||
|
warnings = s["health"]["warnings"]
|
||||||
|
warning_html = "".join(f"<p class='warn'>{esc(w)}</p>" for w in warnings) or "<p class='ok'>None</p>"
|
||||||
|
log_text = "\n".join(esc(f"[{r.get('ts','')}] {r.get('level','')} {r.get('msg','')} {json.dumps(r, ensure_ascii=False)}") for r in logs)
|
||||||
|
source_target_html = f"<p class='tiny'><span>Source:</span> {esc(current.get('src'))}</p><p class='tiny'><span>Target:</span> {esc(current.get('dest'))}</p>" if current else ""
|
||||||
|
progress_html = f"<p class='filename'><strong>{esc(full_current_name)}</strong><br><span class='muted'>{esc(current_relative_dir)}</span></p><p>{esc(progress)}% · runtime {title_runtime}</p><progress max='100' value='{esc(progress or 0)}'></progress><p>{esc(current.get('phase'))} · {gib(current.get('bytes_copied'))} / {gib(current.get('bytes_total'))}</p>{source_target_html}" if current else "<p>—</p>"
|
||||||
|
current_row = f"<tr class='active'><td>▶</td><td class='filename'><strong>{esc(full_current_name)}</strong><br><span class='muted'>{esc(current_relative_dir)}</span></td><td>{esc(progress)}%</td><td><progress max='100' value='{esc(progress or 0)}'></progress></td><td>{title_runtime}</td></tr>" if current else ""
|
||||||
|
current_path = str(current_src_path) if current_src_path else ""
|
||||||
|
processing_items = [item for item in s["queue"]["roots"]["manual"]["processingItems"] if item["path"] != current_path]
|
||||||
|
ready_items = [item for item in s["queue"]["roots"]["manual"]["items"] if item["path"] != current_path]
|
||||||
|
processing_rows = "".join(f"<tr class='processing'><td>⏳</td><td class='filename'><strong>{esc(item['name'])}</strong><br><span class='muted'>{esc(item.get('relativeDir') or item['release'])}</span></td><td>unpacking</td><td>{gib(item.get('bytes'))}</td><td>waiting</td></tr>" for item in processing_items[:30])
|
||||||
|
ready_rows = "".join(f"<tr><td>◷</td><td class='filename'><strong>{esc(item['name'])}</strong><br><span class='muted'>{esc(item.get('relativeDir') or item['release'])}</span></td><td>ready</td><td>{gib(item.get('bytes'))}</td><td>queued</td></tr>" for item in ready_items[:30])
|
||||||
|
queue_rows = processing_rows + ready_rows or "<tr><td>✓</td><td colspan='4'>No video files waiting</td></tr>"
|
||||||
|
history = [r for r in reversed(logs) if r.get("level") == "MOVE" and r.get("msg") == "moving"][:12]
|
||||||
|
history_rows = "".join(f"<tr><td>✓</td><td class='filename'>{esc(Path(str(r.get('dest',''))).name)}</td><td>{esc(r.get('media_type',''))}</td><td colspan='2'>{esc(r.get('ts',''))}</td></tr>" for r in history) or "<tr><td>—</td><td colspan='4'>No recent imports</td></tr>"
|
||||||
|
body = f"""<!doctype html><html><head><meta charset='utf-8'><meta name='viewport' content='width=device-width, initial-scale=1'><meta http-equiv='refresh' content='10'><title>{esc(title)}</title><style>body{{font-family:system-ui,sans-serif;margin:0;background:#111;color:#eee}}header{{display:flex;align-items:center;gap:1rem;background:#3b3b3b;padding:.7rem 1.2rem;border-bottom:1px solid #111;flex-wrap:wrap}}header h1{{margin:0;font-size:1.5rem}}.pill{{background:#222;border:1px solid #555;padding:.35rem .7rem}}main{{padding:1rem}}.grid{{display:grid;grid-template-columns:repeat(auto-fit,minmax(260px,1fr));gap:1rem}}.card{{background:#1d1d1d;border:1px solid #333;border-radius:12px;padding:1rem;margin-bottom:1rem;min-width:0;overflow:hidden}}input,button{{padding:.55rem;border:1px solid #555;background:#2b2b2b;color:#eee}}button{{cursor:pointer}}.filename{{overflow-wrap:anywhere;word-break:break-word}}.tiny{{font-size:.78rem;color:#999;line-height:1.25;overflow-wrap:anywhere}}.tiny span{{color:#bbb}}.ok{{color:#60d394}}.warn{{color:#ffd166}}progress{{width:100%;height:1.2rem;accent-color:#7fd37f}}table{{width:100%;border-collapse:collapse;background:#2b2b2b}}th,td{{padding:.65rem;border-bottom:1px solid #111;text-align:left;vertical-align:middle}}th{{background:#444;color:#ddd}}tr:nth-child(even){{background:#333}}tr.active{{background:#3f4a3f}}tr.processing{{background:#4a4232}}pre{{white-space:pre-wrap;overflow-wrap:anywhere;max-height:28rem;overflow:auto}}.muted{{color:#aaa}}a{{color:#8ecae6}}</style></head><body><header><h1>📥 Importarr</h1><span class='pill'>{esc(s['queue']['files'])} videos ready</span><span class='pill'>{esc(s['queue']['processingFiles'])} unpacking</span><span class='pill'>ready {gib(s['queue']['bytes'])}</span><span class='pill'>1h {esc(s['processed']['last1h'])} · 24h {esc(s['processed']['last24h'])} · total {esc(s['processed']['total'])}</span></header><main><div class='grid'><section class='card'><h2>Status</h2><p class='{('warn' if warnings else 'ok')}'>{esc(s['health']['state'])}</p><p>Service: {esc(s['service']['activeState'])}/{esc(s['service']['subState'])}</p><p>Runtime: {secs(s['service']['runtimeSeconds'])}</p><p>Memory current: {gib(s['service']['memoryCurrentBytes'])}</p><p>Memory peak: {gib(s['service']['memoryPeakBytes'])}</p><p class='muted'>Scheduled automatically every 15 minutes. `_UNPACK_` folders are shown as unpacking, not ready.</p></section><section class='card'><h2>Current file</h2>{progress_html}</section><section class='card'><h2>Processed</h2><p>Last 1h: {esc(s['processed']['last1h'])} imported</p><p>Last 24h: {esc(s['processed']['last24h'])} imported</p><p>Total: {esc(s['processed']['total'])} imported</p></section><section class='card'><h2>Warnings</h2>{warning_html}</section></div><section class='card'><h2>Add manual batch</h2><form onsubmit="event.preventDefault();fetch('/api/manual-batches',{{method:'POST',headers:{{'content-type':'application/json'}},body:JSON.stringify({{path:this.path.value}})}}).then(()=>location.reload())"><input name='path' placeholder='folder under manual downloads' size='60'><button>Add folder once</button></form></section><section class='card'><h2>Jobs</h2><table><thead><tr><th></th><th>Name</th><th>State</th><th>Progress / Size</th><th>Runtime</th></tr></thead><tbody>{current_row}{queue_rows}</tbody></table></section><section class='card'><h2>History</h2><table><thead><tr><th></th><th>Name</th><th>Type</th><th colspan='2'>Time</th></tr></thead><tbody>{history_rows}</tbody></table></section><section class='card'><h2>Recent log</h2><pre>{log_text}</pre></section><p><a href='/api/status'>/api/status</a> · <a href='/api/queue'>/api/queue</a> · <a href='/api/logs?limit=100'>/api/logs</a></p></main></body></html>"""
|
||||||
|
return body.encode()
|
||||||
|
|
||||||
|
|
||||||
|
class Handler(BaseHTTPRequestHandler):
|
||||||
|
def send(self, code: int, content_type: str, data: bytes) -> None:
|
||||||
|
self.send_response(code)
|
||||||
|
self.send_header("Content-Type", content_type)
|
||||||
|
self.send_header("Cache-Control", "no-store")
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(data)
|
||||||
|
|
||||||
|
def do_GET(self) -> None:
|
||||||
|
parsed = urllib.parse.urlparse(self.path)
|
||||||
|
if parsed.path == "/":
|
||||||
|
self.send(200, "text/html; charset=utf-8", page())
|
||||||
|
elif parsed.path == "/api/status":
|
||||||
|
self.send(200, "application/json", json.dumps(status()).encode())
|
||||||
|
elif parsed.path == "/api/queue":
|
||||||
|
self.send(200, "application/json", json.dumps(queue_status()).encode())
|
||||||
|
elif parsed.path == "/api/logs":
|
||||||
|
params = urllib.parse.parse_qs(parsed.query)
|
||||||
|
limit = int(params.get("limit", ["100"])[0])
|
||||||
|
self.send(200, "application/json", json.dumps(read_logs(limit)).encode())
|
||||||
|
elif parsed.path == "/health":
|
||||||
|
self.send(200, "application/json", json.dumps(fast_health()).encode())
|
||||||
|
else:
|
||||||
|
self.send(404, "text/plain", b"not found")
|
||||||
|
|
||||||
|
def do_POST(self) -> None:
|
||||||
|
parsed = urllib.parse.urlparse(self.path)
|
||||||
|
if parsed.path != "/api/manual-batches":
|
||||||
|
self.send(404, "text/plain", b"not found")
|
||||||
|
return
|
||||||
|
length = int(self.headers.get("content-length") or 0)
|
||||||
|
try:
|
||||||
|
payload = json.loads(self.rfile.read(length) or b"{}")
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
self.send(400, "application/json", json.dumps({"ok": False, "error": "invalid json"}).encode())
|
||||||
|
return
|
||||||
|
ok, message = add_manual_batch(str(payload.get("path") or ""))
|
||||||
|
self.send(200 if ok else 400, "application/json", json.dumps({"ok": ok, "result": message}).encode())
|
||||||
|
|
||||||
|
def log_message(self, fmt: str, *args: object) -> None:
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
host = os.getenv("IMPORTARR_BIND_HOST", "0.0.0.0")
|
||||||
|
port = int(os.getenv("IMPORTARR_BIND_PORT", "8095"))
|
||||||
|
ThreadingHTTPServer((host, port), Handler).serve_forever()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -32,8 +32,20 @@
|
|||||||
<dt>Movies root</dt><dd>{{ status.movies_root }}</dd>
|
<dt>Movies root</dt><dd>{{ status.movies_root }}</dd>
|
||||||
<dt>TV root</dt><dd>{{ status.tv_root }}</dd>
|
<dt>TV root</dt><dd>{{ status.tv_root }}</dd>
|
||||||
<dt>Write auth</dt><dd>{{ 'enabled' if status.auth_enabled else 'disabled' }}</dd>
|
<dt>Write auth</dt><dd>{{ 'enabled' if status.auth_enabled else 'disabled' }}</dd>
|
||||||
|
<dt>Queue mode</dt><dd id="queue-mode">{{ status.control.queue_mode }}</dd>
|
||||||
|
<dt>Current job</dt><dd id="current-job">{{ status.current or 'idle' }}</dd>
|
||||||
</dl>
|
</dl>
|
||||||
</section>
|
</section>
|
||||||
|
<section class="panel">
|
||||||
|
<h2>Queue controls</h2>
|
||||||
|
<p>Pause and stop prevent new jobs from being added to the queue. They do not interrupt an import already in progress; use cancel current job for that.</p>
|
||||||
|
<div class="controls">
|
||||||
|
<button type="button" data-control="start">Start</button>
|
||||||
|
<button type="button" data-control="pause">Pause</button>
|
||||||
|
<button type="button" data-control="stop">Stop</button>
|
||||||
|
<button type="button" data-control="cancel-current" class="danger">Cancel current job</button>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
<section>
|
<section>
|
||||||
<h2>Manual batches</h2>
|
<h2>Manual batches</h2>
|
||||||
<form id="batch-form" class="inline-form">
|
<form id="batch-form" class="inline-form">
|
||||||
@@ -51,7 +63,8 @@
|
|||||||
</section>
|
</section>
|
||||||
</main>
|
</main>
|
||||||
<script>
|
<script>
|
||||||
async function refresh(){ const r=await fetch('/api/jobs'); const d=await r.json(); document.title=d.jobs.length?`📥 ${d.jobs.length} jobs - Importarr`:'📥 - idle · Importarr'; document.getElementById('jobs').innerHTML='<table><tr><th>Name</th><th>State</th><th>Context</th></tr>'+d.jobs.map(j=>`<tr><td>${j.name||''}</td><td><span class="state">${j.state}</span></td><td><small>${j.relative_path||j.storage||j.reason||''}</small></td></tr>`).join('')+'</table>'; }
|
async function refresh(){ const r=await fetch('/api/jobs'); const d=await r.json(); document.title=d.jobs.length?`📥 ${d.jobs.length} jobs - Importarr`:'📥 - idle · Importarr'; document.getElementById('jobs').innerHTML='<table><tr><th>Name</th><th>State</th><th>Context</th></tr>'+d.jobs.map(j=>`<tr><td>${j.name||''}</td><td><span class="state">${j.state}</span></td><td><small>${j.relative_path||j.storage||j.reason||''}</small></td></tr>`).join('')+'</table>'; if(d.control){ document.getElementById('queue-mode').textContent=d.control.queue_mode; document.getElementById('current-job').textContent=d.control.current||'idle'; } }
|
||||||
|
document.querySelectorAll('[data-control]').forEach(button=>button.addEventListener('click', async()=>{ await fetch(`/api/control/${button.dataset.control}`,{method:'POST'}); await refresh(); }));
|
||||||
document.getElementById('browse-batch').addEventListener('click',()=>document.getElementById('batch-picker').click());
|
document.getElementById('browse-batch').addEventListener('click',()=>document.getElementById('batch-picker').click());
|
||||||
document.getElementById('batch-picker').addEventListener('change',e=>{ const f=e.target.files[0]; if(!f)return; const top=(f.webkitRelativePath||'').split('/')[0]; if(top) document.querySelector('#batch-form [name="path"]').value=top; });
|
document.getElementById('batch-picker').addEventListener('change',e=>{ const f=e.target.files[0]; if(!f)return; const top=(f.webkitRelativePath||'').split('/')[0]; if(top) document.querySelector('#batch-form [name="path"]').value=top; });
|
||||||
document.getElementById('batch-form').addEventListener('submit', async e=>{ e.preventDefault(); await fetch('/api/manual-batches',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({path:e.target.path.value})}); location.reload(); });
|
document.getElementById('batch-form').addEventListener('submit', async e=>{ e.preventDefault(); await fetch('/api/manual-batches',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({path:e.target.path.value})}); location.reload(); });
|
||||||
|
|||||||
+1093
File diff suppressed because it is too large
Load Diff
@@ -22,6 +22,8 @@ test = ["pytest>=8.2", "pytest-asyncio>=0.23"]
|
|||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
importarr = "importarr.main:run"
|
importarr = "importarr.main:run"
|
||||||
|
importarr-status = "importarr.status_ui:main"
|
||||||
|
manual-media-import = "importarr.worker:main"
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
[tool.pytest.ini_options]
|
||||||
testpaths = ["tests"]
|
testpaths = ["tests"]
|
||||||
|
|||||||
@@ -0,0 +1,80 @@
|
|||||||
|
from importarr.config import Settings
|
||||||
|
from importarr.state import State
|
||||||
|
|
||||||
|
|
||||||
|
def configure_main(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("IMPORTARR_STATE_PATH", str(tmp_path / "initial.db"))
|
||||||
|
import importarr.main as main
|
||||||
|
|
||||||
|
download = tmp_path / "downloads"
|
||||||
|
movies = tmp_path / "movies"
|
||||||
|
tv = tmp_path / "tv"
|
||||||
|
monkeypatch.setattr(main, "settings", Settings(download_root=download, movies_root=movies, tv_root=tv, state_path=tmp_path / "state.db"))
|
||||||
|
monkeypatch.setattr(main, "state", State(tmp_path / "state.db"))
|
||||||
|
return main, download, movies, tv
|
||||||
|
|
||||||
|
|
||||||
|
def test_pause_prevents_manual_queue_sync(tmp_path, monkeypatch):
|
||||||
|
main, download, _movies, _tv = configure_main(tmp_path, monkeypatch)
|
||||||
|
batch = download / "Release"
|
||||||
|
batch.mkdir(parents=True)
|
||||||
|
(batch / "Movie.mkv").write_bytes(b"movie")
|
||||||
|
main.state.add_manual_batch(batch)
|
||||||
|
|
||||||
|
main.state.set_app_state("queue_mode", "paused")
|
||||||
|
main.sync_manual_queue()
|
||||||
|
|
||||||
|
assert main.state.list_queue_items() == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_start_reenables_manual_queue_sync(tmp_path, monkeypatch):
|
||||||
|
main, download, _movies, _tv = configure_main(tmp_path, monkeypatch)
|
||||||
|
batch = download / "Release"
|
||||||
|
batch.mkdir(parents=True)
|
||||||
|
(batch / "Movie.mkv").write_bytes(b"movie")
|
||||||
|
main.state.add_manual_batch(batch)
|
||||||
|
|
||||||
|
main.state.set_app_state("queue_mode", "paused")
|
||||||
|
main.sync_manual_queue()
|
||||||
|
main.state.set_app_state("queue_mode", "running")
|
||||||
|
main.sync_manual_queue()
|
||||||
|
|
||||||
|
assert len(main.state.list_queue_items()) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_cancel_current_stops_before_next_manual_item(tmp_path, monkeypatch):
|
||||||
|
main, download, movies, tv = configure_main(tmp_path, monkeypatch)
|
||||||
|
batch = download / "Release"
|
||||||
|
batch.mkdir(parents=True)
|
||||||
|
(batch / "A.mkv").write_bytes(b"a")
|
||||||
|
(batch / "B.mkv").write_bytes(b"b")
|
||||||
|
main.state.add_manual_batch(batch)
|
||||||
|
main.sync_manual_queue()
|
||||||
|
main.state.set_app_state("cancel_requested", "true")
|
||||||
|
|
||||||
|
assert main._import_manual_batches(main.Importer(movies, tv)) == 0
|
||||||
|
assert len(main.state.list_queue_items()) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_cancel_current_stops_active_copy(tmp_path, monkeypatch):
|
||||||
|
main, download, movies, tv = configure_main(tmp_path, monkeypatch)
|
||||||
|
batch = download / "Release"
|
||||||
|
batch.mkdir(parents=True)
|
||||||
|
source = batch / "A.mkv"
|
||||||
|
source.write_bytes(b"a" * (1024 * 1024 + 1))
|
||||||
|
main.state.add_manual_batch(batch)
|
||||||
|
main.sync_manual_queue()
|
||||||
|
|
||||||
|
calls = 0
|
||||||
|
|
||||||
|
def cancel_during_copy() -> bool:
|
||||||
|
nonlocal calls
|
||||||
|
calls += 1
|
||||||
|
return calls > 1
|
||||||
|
|
||||||
|
monkeypatch.setattr(main, "consume_cancel_request", cancel_during_copy)
|
||||||
|
|
||||||
|
assert main._import_manual_batches(main.Importer(movies, tv)) == 0
|
||||||
|
assert source.exists()
|
||||||
|
assert not any(movies.glob("*.partial"))
|
||||||
|
assert main.state.list_queue_items(active_only=False)[0]["state"] == "skipped"
|
||||||
Reference in New Issue
Block a user