Author SHA1 Message Date
daniels 8cb82ee8c4 Add queue override controls
Adds pause, stop, start, and cancel-current controls for #20.
2026-07-29 14:35:28 +02:00
daniels 2f996600c6 Remove private deployment details 2026-07-29 14:13:22 +02:00
daniels 67335a37b9 Document Importarr issue deployment workflow 2026-07-29 13:53:49 +02:00
daniels 259f5e7ee2 Install Importarr services from repo on dgsserver1 2026-07-29 13:41:06 +02:00
21 changed files with 1914 additions and 185 deletions
+1
View File
@@ -5,3 +5,4 @@ __pycache__/
*.pyc *.pyc
*.db *.db
*.partial *.partial
AGENTS.local.md
-80
View File
@@ -1,80 +0,0 @@
# Importarr Agent Instructions
Importarr is owned as a Linux-ops-managed service repository. Treat this checkout as the source of truth for application code, deployment files, and local service installs.
## Repository Source Of Truth
- Work from `/srv/opencode-workspace/importarr` for Importarr code and deploy changes.
- Do not edit host-local legacy scripts as the normal workflow:
- `/usr/local/sbin/importarr-status.py`
- `/usr/local/sbin/manual-media-import.py`
- If an emergency hotfix is made outside the repo, backport it here immediately and reinstall from the repo.
## Seamless Feature Workflow
When asked to implement an Importarr feature, fix, UI change, deployment change, or operational behavior change:
1. Inspect `git status --short --branch` before editing.
2. Implement the smallest correct repo change.
3. Run the narrowest useful verification, normally:
```sh
.venv/bin/pytest -q
```
If shell deploy scripts changed, also run:
```sh
sh -n deploy/systemd-install.sh && sh -n deploy/repo-upgrade.sh
```
4. Inspect `git diff` and ensure no secrets, raw `.env`, tokens, databases, or private material are included.
5. Commit and push completed Importarr changes by default unless the user explicitly asks not to publish or verification is blocked.
6. Install/restart from the repository so the running local service matches the repo:
```sh
make upgrade-local
```
7. Verify the live service:
```sh
make verify-live
```
8. If the live install fails, inspect `systemctl status importarr.service` and `journalctl -u importarr.service`; fix the repo, commit/push the fix, reinstall, and verify again.
## Install Model
- The service virtualenv lives at `/opt/importarr/venv`.
- The systemd unit runs `/opt/importarr/venv/bin/importarr` as the `importarr` system user.
- The package is installed from the repository into the venv using normal wheel/package install, not editable install.
- Do **not** use editable install for the system service: the unprivileged `importarr` user may not be able to read `/srv/opencode-workspace/importarr`, causing `ModuleNotFoundError` at startup.
- `/opt/importarr/repo-upgrade.sh` is the pull-and-reinstall helper for machines that should follow pushed `main`.
## Local Commands
```sh
make test
make install-systemd
make upgrade-local
make repo-upgrade
make verify-live
```
`make repo-upgrade` is for pulling already-pushed changes with `git pull --ff-only`. It refuses to run with uncommitted repo changes.
## Runtime Defaults On This Host
- Local URL: `http://127.0.0.1:8095/`
- Health: `http://127.0.0.1:8095/health`
- Status: `http://127.0.0.1:8095/api/status`
- Systemd service: `importarr.service`
- Env file: `/etc/importarr/importarr.env`
- SQLite state: `/var/lib/importarr/importarr.db`
## Secret Handling
- Prefer `*_FILE` settings for secrets, for example:
- `IMPORTARR_SAB_API_KEY_FILE`
- `IMPORTARR_AUTH_TOKEN_FILE`
- `IMPORTARR_RADARR_API_KEY_FILE`
- `IMPORTARR_SONARR_API_KEY_FILE`
- Never commit real env files, API keys, tokens, private keys, service databases, or backup data.
- Template files may list variable names with placeholder values or commented examples only.
## Linux Ops Follow-Through
For changes that materially alter the live service setup, ports, routes, monitoring, backup coverage, or host ownership, also follow the linux-ops documentation/systems-overview update rules. Do not mix unrelated pre-existing uncommitted changes from `linux-ops-docs` or `systems-overview` into Importarr commits.
+10 -12
View File
@@ -1,9 +1,11 @@
PYTHON ?= .venv/bin/python PYTHON ?= .venv/bin/python
PIP ?= .venv/bin/pip PIP ?= .venv/bin/pip
SERVICE ?= importarr.service SERVICE ?= importarr.service
LIVE_URL ?= http://127.0.0.1:8095 IMPORTARR_PREFIX ?= /opt/importarr
IMPORTARR_REPO_DIR ?= $(CURDIR)
IMPORTARR_URL ?= http://127.0.0.1:8765
.PHONY: test install-systemd install-from-repo upgrade-local repo-upgrade verify-live .PHONY: test install-systemd install-from-repo repo-upgrade verify
test: test:
$(PYTHON) -m pytest $(PYTHON) -m pytest
@@ -12,16 +14,12 @@ install-systemd:
sudo -n sh deploy/systemd-install.sh sudo -n sh deploy/systemd-install.sh
install-from-repo: install-from-repo:
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr sudo -n $(IMPORTARR_PREFIX)/venv/bin/pip install --upgrade $(IMPORTARR_REPO_DIR)
upgrade-local:
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
sudo -n systemctl restart $(SERVICE)
repo-upgrade: repo-upgrade:
sudo -n sh /opt/importarr/repo-upgrade.sh sudo -n IMPORTARR_PREFIX=$(IMPORTARR_PREFIX) IMPORTARR_REPO_DIR=$(IMPORTARR_REPO_DIR) sh deploy/repo-upgrade.sh
verify-live: verify:
curl -fsS $(LIVE_URL)/health curl -fsS $(IMPORTARR_URL)/health
curl -fsS $(LIVE_URL)/api/status curl -fsS $(IMPORTARR_URL)/api/status
curl -fsS $(LIVE_URL)/api/preview curl -fsS $(IMPORTARR_URL)/api/preview
+14 -56
View File
@@ -2,17 +2,17 @@
Importarr is an Arr-style service for importing manually categorized SABnzbd downloads after SAB reports final completion. It owns one SAB category, defaults to `manual`, and refuses to import transient Direct Unpack paths or jobs still in SAB queue/post-processing. Importarr is an Arr-style service for importing manually categorized SABnzbd downloads after SAB reports final completion. It owns one SAB category, defaults to `manual`, and refuses to import transient Direct Unpack paths or jobs still in SAB queue/post-processing.
## New-machine install ## Install
Importarr is intended to feel like a small Arr service: deploy the container or systemd service, edit one env file, point SABnzbd category `manual` at the same completed-download path, then open the web UI. Importarr is intended to feel like a small Arr service: deploy the container or systemd service, edit one env file, point SABnzbd category `manual` at the same completed-download path, then open the web UI.
### Docker Compose, recommended ### Docker Compose, recommended
```sh ```sh
mkdir -p /opt/importarr/config mkdir -p importarr/config
cd /opt/importarr cd importarr
curl -fsSLO https://gitea.delphas.dk/daniels/importarr/raw/branch/main/deploy/docker-compose.example.yml curl -fsSLO https://example.com/importarr/deploy/docker-compose.example.yml
curl -fsSLo importarr.env https://gitea.delphas.dk/daniels/importarr/raw/branch/main/deploy/importarr.env.example curl -fsSLo importarr.env https://example.com/importarr/deploy/importarr.env.example
${EDITOR:-vi} importarr.env ${EDITOR:-vi} importarr.env
docker compose -f docker-compose.example.yml --env-file importarr.env up -d docker compose -f docker-compose.example.yml --env-file importarr.env up -d
``` ```
@@ -26,25 +26,14 @@ docker build -t importarr:local .
### systemd / pip install ### systemd / pip install
```sh ```sh
git clone https://gitea.delphas.dk/daniels/importarr.git git clone https://example.com/importarr.git
cd importarr cd importarr
sudo sh deploy/systemd-install.sh sudo sh deploy/systemd-install.sh
sudo ${EDITOR:-vi} /etc/importarr/importarr.env sudo ${EDITOR:-vi} /etc/importarr/importarr.env
sudo systemctl start importarr.service sudo systemctl start importarr.service
``` ```
The installer creates the `importarr` system user when needed, installs a virtualenv at `/opt/importarr/venv`, and installs the package from the checked-out repository. The repository is therefore the source of truth: pull or edit the repo, reinstall/restart from the repo, and the service runs the package built from that code. The installer creates the `importarr` system user when needed, installs a virtualenv, and installs the package from the checked-out repository. Override install paths with `IMPORTARR_*` variables if the defaults do not fit your environment.
For local upgrades from a checked-out repo on dgsserver1, use the repo workflow instead of editing live scripts:
```sh
cd /srv/opencode-workspace/importarr
.venv/bin/python -m pytest
git status --short --branch
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
sudo -n systemctl restart importarr.service
make verify-live
```
For a machine that should stay current with the repository, use the installed repo-upgrade helper: For a machine that should stay current with the repository, use the installed repo-upgrade helper:
@@ -54,7 +43,7 @@ sudo -n sh /opt/importarr/repo-upgrade.sh
The helper refuses to run when the checkout has uncommitted changes, then performs `git pull --ff-only`, reinstalls the package from the repo, restarts `importarr.service`, and prints service status. Use it after changes have been committed and pushed to `main`. The helper refuses to run when the checkout has uncommitted changes, then performs `git pull --ff-only`, reinstalls the package from the repo, restarts `importarr.service`, and prints service status. Use it after changes have been committed and pushed to `main`.
Release-worthy changes should be committed, tagged with SemVer (`v0.1.1`, `v0.2.0`, ...), pushed with tags, then reinstalled from the tagged checkout or artifact. Do not hand-edit `/usr/local/sbin/importarr-status.py` or `/usr/local/sbin/manual-media-import.py` except for a documented emergency hotfix that is immediately backported here. Release-worthy changes should be committed, tagged with SemVer (`v0.1.1`, `v0.2.0`, ...), pushed with tags, then installed from the tagged checkout or artifact.
### Required setup ### Required setup
@@ -96,46 +85,15 @@ pytest
uvicorn importarr.main:app --reload uvicorn importarr.main:app --reload
``` ```
## Migration notes for dgsserver1 ## Operations
Export the existing script settings into `IMPORTARR_*` env vars, add historical folders as explicit manual batches, run a dry-run/inspection through `/api/preview`, then switch the systemd service or Compose route after the ready set matches expectations. Importarr intentionally does not document private deployment topology, hostnames, reverse proxies, monitoring, backups, or operator workflows in this repository. Keep those details in your own ops runbooks.
On dgsserver1, the packaged service is the only intended active entrypoint after cutover. Keep `manual-media-import.timer` disabled unless a repo-managed worker/timer replaces it later. For a systemd install, prefer a normal package install from the checked-out repo over an editable install so the service user does not need read access to your development checkout.
## Repository-as-install workflow If the service fails, standard systemd diagnostics are usually enough:
Importarr should not drift into host-local scripts. Treat the checked-out repository as the install source:
1. Make changes in `/srv/opencode-workspace/importarr`.
2. Run tests: `make test`.
3. Commit and push the repo change.
4. Install/restart from the same repo: `make upgrade-local` for local changes, or `make repo-upgrade` to pull the latest pushed `main` and restart.
5. Verify the live service: `make verify-live`.
Do not edit `/usr/local/sbin/importarr-status.py`, `/usr/local/sbin/manual-media-import.py`, or files copied out of the repo as the normal workflow. If an emergency live hotfix is unavoidable, backport it to this repository immediately and run the repo install workflow again.
### Local install lessons learned
- The live systemd service runs as the unprivileged `importarr` user.
- Do not install the system service with `pip install --editable /srv/opencode-workspace/importarr`; that can fail at startup if the service user cannot read the workspace checkout.
- The supported local service install is a normal package install from the repo into `/opt/importarr/venv`:
```sh ```sh
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr sudo systemctl --no-pager --full status importarr.service
sudo -n systemctl restart importarr.service sudo journalctl -u importarr.service -n 120 --no-pager
```
- `deploy/systemd-install.sh`, `make upgrade-local`, and `/opt/importarr/repo-upgrade.sh` already use this supported model.
- After every implementation task that should affect the live local service, run:
```sh
make upgrade-local
make verify-live
```
- If `make verify-live` fails, check:
```sh
sudo -n systemctl --no-pager --full status importarr.service
sudo -n journalctl -u importarr.service -n 120 --no-pager
``` ```
+1 -1
View File
@@ -6,7 +6,7 @@ services:
- "8765:8765" - "8765:8765"
volumes: volumes:
- ./config:/config - ./config:/config
- /srv/scrypted/sabnzbd-data/downloads/manual:/data/downloads/manual - /path/to/downloads/manual:/data/downloads/manual
- /path/to/movies:/data/movies - /path/to/movies:/data/movies
- /path/to/tv:/data/tv - /path/to/tv:/data/tv
restart: unless-stopped restart: unless-stopped
+7 -7
View File
@@ -1,16 +1,16 @@
IMPORTARR_SAB_URL=http://127.0.0.1:8080 IMPORTARR_SAB_URL=http://sabnzbd:8080
# Prefer *_FILE for secrets. Plain env vars still work for local/dev installs. # Prefer *_FILE for secrets. Plain env vars still work for local/dev installs.
# IMPORTARR_SAB_API_KEY=change-me # IMPORTARR_SAB_API_KEY=change-me
# IMPORTARR_SAB_API_KEY_FILE=/etc/importarr/sab-api-key # IMPORTARR_SAB_API_KEY_FILE=/etc/importarr/sab-api-key
IMPORTARR_SAB_CATEGORY=manual IMPORTARR_SAB_CATEGORY=manual
IMPORTARR_DOWNLOAD_ROOT=/srv/scrypted/sabnzbd-data/downloads/manual IMPORTARR_DOWNLOAD_ROOT=/data/downloads/manual
IMPORTARR_MOVIES_ROOT=/srv/media/movies IMPORTARR_MOVIES_ROOT=/data/movies
IMPORTARR_TV_ROOT=/srv/media/tv IMPORTARR_TV_ROOT=/data/tv
IMPORTARR_STATE_PATH=/var/lib/importarr/importarr.db IMPORTARR_STATE_PATH=/config/importarr.db
IMPORTARR_LOG_LEVEL=info IMPORTARR_LOG_LEVEL=info
# IMPORTARR_AUTH_TOKEN=change-me # IMPORTARR_AUTH_TOKEN=change-me
# IMPORTARR_AUTH_TOKEN_FILE=/etc/importarr/auth-token # IMPORTARR_AUTH_TOKEN_FILE=/etc/importarr/auth-token
IMPORTARR_BIND_HOST=0.0.0.0 IMPORTARR_BIND_HOST=0.0.0.0
IMPORTARR_BIND_PORT=8095 IMPORTARR_BIND_PORT=8765
IMPORTARR_POLL_SECONDS=60 IMPORTARR_POLL_SECONDS=60
IMPORTARR_REPO_DIR=/srv/opencode-workspace/importarr # IMPORTARR_REPO_DIR=/path/to/importarr
+5 -6
View File
@@ -1,16 +1,15 @@
[Unit] [Unit]
Description=Importarr manual media importer Description=Importarr manual media importer status UI
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
[Service] [Service]
EnvironmentFile=/etc/importarr/importarr.env EnvironmentFile=-/etc/importarr/importarr.env
ExecStart=/opt/importarr/venv/bin/importarr EnvironmentFile=-/opt/importarr/build.env
ExecStart=/opt/importarr/venv/bin/importarr-status
Restart=on-failure Restart=on-failure
RestartSec=5s RestartSec=5s
User=importarr User=root
Group=importarr
StateDirectory=importarr
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target
@@ -0,0 +1,6 @@
[Unit]
Description=Critical alert when manual media importer fails
[Service]
Type=oneshot
ExecStart=/usr/bin/logger -t importarr "manual-media-import.service failed; check journalctl -u manual-media-import.service"
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Import manual SABnzbd media into Jellyfin library roots
Wants=network-online.target
After=network-online.target
OnFailure=manual-media-import-failure.service
[Service]
Type=oneshot
EnvironmentFile=-/etc/importarr/importarr.env
EnvironmentFile=-/opt/importarr/build.env
ExecStart=/opt/importarr/venv/bin/manual-media-import
TimeoutStartSec=30min
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Run manual media importer periodically
[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
AccuracySec=1min
Persistent=true
Unit=manual-media-import.service
[Install]
WantedBy=timers.target
+10 -2
View File
@@ -12,9 +12,10 @@ if [ -f "$ENV_FILE" ]; then
. "$ENV_FILE" . "$ENV_FILE"
fi fi
REPO_DIR=${IMPORTARR_REPO_DIR:-/srv/opencode-workspace/importarr} PREFIX=${IMPORTARR_PREFIX:-/opt/importarr}
REPO_DIR=${IMPORTARR_REPO_DIR:-$(pwd)}
SERVICE=${IMPORTARR_SERVICE:-importarr.service} SERVICE=${IMPORTARR_SERVICE:-importarr.service}
VENV=${IMPORTARR_VENV:-/opt/importarr/venv} VENV=${IMPORTARR_VENV:-$PREFIX/venv}
if [ ! -d "$REPO_DIR/.git" ]; then if [ ! -d "$REPO_DIR/.git" ]; then
echo "Importarr repo not found at $REPO_DIR" >&2 echo "Importarr repo not found at $REPO_DIR" >&2
@@ -31,5 +32,12 @@ fi
git fetch --prune origin git fetch --prune origin
git pull --ff-only git pull --ff-only
"$VENV/bin/pip" install --upgrade "$REPO_DIR" "$VENV/bin/pip" install --upgrade "$REPO_DIR"
GIT_SHA="$(git rev-parse --short=12 HEAD 2>/dev/null || printf development)"
BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
cat > "$PREFIX/build.env" <<EOF
IMPORTARR_GIT_SHA=$GIT_SHA
IMPORTARR_BUILD_DATE=$BUILD_DATE
EOF
systemctl restart "$SERVICE" systemctl restart "$SERVICE"
systemctl restart manual-media-import.timer
systemctl --no-pager --full status "$SERVICE" systemctl --no-pager --full status "$SERVICE"
+14 -6
View File
@@ -6,13 +6,9 @@ if [ "$(id -u)" -ne 0 ]; then
exit 1 exit 1
fi fi
install -d -m 0755 /etc/importarr /var/lib/importarr install -d -m 0755 /etc/importarr /var/lib/importarr /run/manual-media-import
install -d -m 0755 /opt/importarr install -d -m 0755 /opt/importarr
REPO_DIR="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)" REPO_DIR="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)"
if ! id importarr >/dev/null 2>&1; then
useradd --system --home /var/lib/importarr --shell /usr/sbin/nologin importarr
fi
chown importarr:importarr /var/lib/importarr
python3 -m venv /opt/importarr/venv python3 -m venv /opt/importarr/venv
/opt/importarr/venv/bin/pip install --upgrade pip /opt/importarr/venv/bin/pip install --upgrade pip
/opt/importarr/venv/bin/pip install --upgrade "$REPO_DIR" /opt/importarr/venv/bin/pip install --upgrade "$REPO_DIR"
@@ -24,7 +20,19 @@ if ! grep -q '^IMPORTARR_REPO_DIR=' /etc/importarr/importarr.env; then
printf '\nIMPORTARR_REPO_DIR=%s\n' "$REPO_DIR" >> /etc/importarr/importarr.env printf '\nIMPORTARR_REPO_DIR=%s\n' "$REPO_DIR" >> /etc/importarr/importarr.env
fi fi
install -m 0644 "$REPO_DIR/deploy/importarr.service" /etc/systemd/system/importarr.service install -m 0644 "$REPO_DIR/deploy/importarr.service" /etc/systemd/system/importarr.service
install -m 0644 "$REPO_DIR/deploy/manual-media-import.service" /etc/systemd/system/manual-media-import.service
install -m 0644 "$REPO_DIR/deploy/manual-media-import.timer" /etc/systemd/system/manual-media-import.timer
install -m 0644 "$REPO_DIR/deploy/manual-media-import-failure.service" /etc/systemd/system/manual-media-import-failure.service
install -m 0755 "$REPO_DIR/deploy/repo-upgrade.sh" /opt/importarr/repo-upgrade.sh install -m 0755 "$REPO_DIR/deploy/repo-upgrade.sh" /opt/importarr/repo-upgrade.sh
GIT_SHA="$(git -C "$REPO_DIR" rev-parse --short=12 HEAD 2>/dev/null || printf development)"
BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
cat > /opt/importarr/build.env <<EOF
IMPORTARR_GIT_SHA=$GIT_SHA
IMPORTARR_BUILD_DATE=$BUILD_DATE
EOF
systemctl daemon-reload systemctl daemon-reload
systemctl enable importarr.service systemctl enable importarr.service
echo "Edit /etc/importarr/importarr.env, then run: systemctl start importarr.service" systemctl enable manual-media-import.timer
systemctl restart importarr.service
systemctl start manual-media-import.timer
echo "Importarr installed from $REPO_DIR. Future upgrades: sudo -n sh /opt/importarr/repo-upgrade.sh"
+17 -2
View File
@@ -4,6 +4,7 @@ import os
import shutil import shutil
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Callable
@dataclass @dataclass
@@ -13,6 +14,10 @@ class ImportResult:
bytes: int bytes: int
class ImportCancelled(Exception):
"""Raised when an import is cancelled at a safe copy boundary."""
class Importer: class Importer:
def __init__(self, movies_root: Path, tv_root: Path): def __init__(self, movies_root: Path, tv_root: Path):
self.movies_root = movies_root self.movies_root = movies_root
@@ -22,14 +27,24 @@ class Importer:
target_root = self.tv_root if _looks_like_tv(source) else self.movies_root target_root = self.tv_root if _looks_like_tv(source) else self.movies_root
return _unique_path(target_root / source.name) return _unique_path(target_root / source.name)
def import_file(self, source: Path) -> ImportResult: def import_file(self, source: Path, should_cancel: Callable[[], bool] | None = None) -> ImportResult:
target = self.target_for(source) target = self.target_for(source)
target.parent.mkdir(parents=True, exist_ok=True) target.parent.mkdir(parents=True, exist_ok=True)
partial = target.with_name(target.name + ".partial") partial = target.with_name(target.name + ".partial")
try:
with source.open("rb") as src, partial.open("wb") as dst: with source.open("rb") as src, partial.open("wb") as dst:
shutil.copyfileobj(src, dst, length=1024 * 1024) while True:
if should_cancel and should_cancel():
raise ImportCancelled("import cancelled")
chunk = src.read(1024 * 1024)
if not chunk:
break
dst.write(chunk)
dst.flush() dst.flush()
os.fsync(dst.fileno()) os.fsync(dst.fileno())
except ImportCancelled:
partial.unlink(missing_ok=True)
raise
if partial.stat().st_size != source.stat().st_size: if partial.stat().st_size != source.stat().st_size:
raise IOError("partial copy size mismatch") raise IOError("partial copy size mismatch")
partial.rename(target) partial.rename(target)
+102 -5
View File
@@ -12,7 +12,7 @@ from pydantic import BaseModel
from .build_info import build_info from .build_info import build_info
from .config import Settings from .config import Settings
from .importer import Importer from .importer import ImportCancelled, Importer
from .sabnzbd import SabnzbdClient from .sabnzbd import SabnzbdClient
from .readiness import classify_history_item from .readiness import classify_history_item
from .scanner import scan_videos from .scanner import scan_videos
@@ -33,6 +33,10 @@ class RunNowRequest(BaseModel):
force: bool = False force: bool = False
class QueueControlRequest(BaseModel):
mode: str
def require_write_auth(authorization: Annotated[str | None, Header()] = None) -> None: def require_write_auth(authorization: Annotated[str | None, Header()] = None) -> None:
if not settings.auth_token: if not settings.auth_token:
return return
@@ -53,6 +57,7 @@ def index(request: Request) -> HTMLResponse:
@app.get("/api/status") @app.get("/api/status")
def status() -> dict[str, object]: def status() -> dict[str, object]:
history = state.list_history() history = state.list_history()
control = control_status()
return { return {
"app": "Importarr", "app": "Importarr",
"build": build_info(), "build": build_info(),
@@ -66,12 +71,80 @@ def status() -> dict[str, object]:
"manual_batches": len(state.list_manual_batches(active_only=True)), "manual_batches": len(state.list_manual_batches(active_only=True)),
"imported_total": sum(1 for row in history if row["status"] == "imported"), "imported_total": sum(1 for row in history if row["status"] == "imported"),
"failed_total": sum(1 for row in history if row["status"] == "failed"), "failed_total": sum(1 for row in history if row["status"] == "failed"),
"current": None, "current": control["current"],
"control": control,
} }
def control_status() -> dict[str, object]:
mode = state.get_app_state("queue_mode", "running") or "running"
current = state.get_app_state("current_job")
cancel_requested = (state.get_app_state("cancel_requested", "false") or "false") == "true"
return {
"queue_mode": mode,
"queue_accepting_new_jobs": mode == "running",
"cancel_requested": cancel_requested,
"current": current,
}
def queue_accepting_new_jobs() -> bool:
return (state.get_app_state("queue_mode", "running") or "running") == "running"
def cancel_requested() -> bool:
return (state.get_app_state("cancel_requested", "false") or "false") == "true"
def consume_cancel_request() -> bool:
if not cancel_requested():
return False
state.set_app_state("cancel_requested", "false")
return True
def set_current_job(name: str | None) -> None:
state.set_app_state("current_job", name or "")
@app.post("/api/control/queue")
def set_queue_control(payload: QueueControlRequest, _: None = Depends(require_write_auth)) -> dict[str, object]:
if payload.mode not in {"running", "paused", "stopped"}:
raise HTTPException(status_code=400, detail="mode must be running, paused, or stopped")
state.set_app_state("queue_mode", payload.mode)
if payload.mode == "running":
state.set_app_state("cancel_requested", "false")
return control_status()
@app.post("/api/control/start")
def start_queue(_: None = Depends(require_write_auth)) -> dict[str, object]:
state.set_app_state("queue_mode", "running")
state.set_app_state("cancel_requested", "false")
return control_status()
@app.post("/api/control/pause")
def pause_queue(_: None = Depends(require_write_auth)) -> dict[str, object]:
state.set_app_state("queue_mode", "paused")
return control_status()
@app.post("/api/control/stop")
def stop_queue(_: None = Depends(require_write_auth)) -> dict[str, object]:
state.set_app_state("queue_mode", "stopped")
return control_status()
@app.post("/api/control/cancel-current")
def cancel_current(_: None = Depends(require_write_auth)) -> dict[str, object]:
state.set_app_state("cancel_requested", "true")
return control_status()
@app.get("/api/manual-batches") @app.get("/api/manual-batches")
def manual_batches() -> list[dict[str, object]]: def manual_batches() -> list[dict[str, object]]:
if queue_accepting_new_jobs():
sync_manual_queue() sync_manual_queue()
rows = [] rows = []
for batch in state.list_manual_batches(): for batch in state.list_manual_batches():
@@ -107,9 +180,10 @@ async def jobs() -> dict[str, object]:
@app.get("/api/preview") @app.get("/api/preview")
async def preview() -> dict[str, object]: async def preview() -> dict[str, object]:
if queue_accepting_new_jobs():
await sync_queue() await sync_queue()
jobs = queue_jobs() jobs = queue_jobs()
return {"sab_status": "ok", "jobs": jobs, "would_import": sum(1 for row in jobs if row["state"] in {"ready", "manual_batch"})} return {"sab_status": "ok", "jobs": jobs, "would_import": sum(1 for row in jobs if row["state"] in {"ready", "manual_batch"}), "control": control_status()}
async def sync_queue() -> None: async def sync_queue() -> None:
@@ -156,6 +230,8 @@ def manual_batch_jobs() -> list[dict[str, object]]:
def sync_manual_queue() -> None: def sync_manual_queue() -> None:
if not queue_accepting_new_jobs():
return
root = settings.download_root.resolve() root = settings.download_root.resolve()
for batch in state.list_manual_batches(active_only=True): for batch in state.list_manual_batches(active_only=True):
seen: set[str] = set() seen: set[str] = set()
@@ -185,37 +261,58 @@ async def _import_ready_sab_jobs(importer: Importer, force: bool = False) -> int
return 0 return 0
imported = 0 imported = 0
for item in data.get("history", {}).get("slots", []): for item in data.get("history", {}).get("slots", []):
if consume_cancel_request():
break
readiness = classify_history_item(item, active, settings.sab_category, settings.download_root, force_status=force) readiness = classify_history_item(item, active, settings.sab_category, settings.download_root, force_status=force)
if readiness.storage is None or (not readiness.ready and not force): if readiness.storage is None or (not readiness.ready and not force):
continue continue
for video in scan_videos(readiness.storage): for video in scan_videos(readiness.storage):
if consume_cancel_request():
return imported
set_current_job(str(video.path))
try: try:
result = importer.import_file(video.path) result = importer.import_file(video.path, should_cancel=consume_cancel_request)
state.add_history(result.source, result.target, "imported", result.bytes) state.add_history(result.source, result.target, "imported", result.bytes)
state.mark_queue_item("sab", str(video.path), "imported") state.mark_queue_item("sab", str(video.path), "imported")
imported += 1 imported += 1
except ImportCancelled:
state.add_history(video.path, video.path, "cancelled", 0, "cancelled")
state.mark_queue_item("sab", str(video.path), "skipped", "cancelled")
return imported
except Exception as exc: except Exception as exc:
state.add_history(video.path, video.path, "failed", 0, exc.__class__.__name__) state.add_history(video.path, video.path, "failed", 0, exc.__class__.__name__)
state.mark_queue_item("sab", str(video.path), "failed", exc.__class__.__name__) state.mark_queue_item("sab", str(video.path), "failed", exc.__class__.__name__)
finally:
set_current_job(None)
return imported return imported
def _import_manual_batches(importer: Importer) -> int: def _import_manual_batches(importer: Importer) -> int:
if queue_accepting_new_jobs():
sync_manual_queue() sync_manual_queue()
imported = 0 imported = 0
for batch in state.list_manual_batches(active_only=True): for batch in state.list_manual_batches(active_only=True):
path = Path(batch["path"]) path = Path(batch["path"])
items = [item for item in state.list_queue_items() if item["source_type"] == "manual" and item["batch_id"] == batch["id"]] items = [item for item in state.list_queue_items() if item["source_type"] == "manual" and item["batch_id"] == batch["id"]]
for item in items: for item in items:
if consume_cancel_request():
return imported
source = Path(item["source_path"]) source = Path(item["source_path"])
set_current_job(str(source))
try: try:
result = importer.import_file(source) result = importer.import_file(source, should_cancel=consume_cancel_request)
state.add_history(result.source, result.target, "imported", result.bytes) state.add_history(result.source, result.target, "imported", result.bytes)
state.mark_queue_item("manual", item["source_id"], "imported") state.mark_queue_item("manual", item["source_id"], "imported")
imported += 1 imported += 1
except ImportCancelled:
state.add_history(source, source, "cancelled", 0, "cancelled")
state.mark_queue_item("manual", item["source_id"], "skipped", "cancelled")
return imported
except Exception as exc: except Exception as exc:
state.add_history(source, source, "failed", 0, exc.__class__.__name__) state.add_history(source, source, "failed", 0, exc.__class__.__name__)
state.mark_queue_item("manual", item["source_id"], "failed", exc.__class__.__name__) state.mark_queue_item("manual", item["source_id"], "failed", exc.__class__.__name__)
finally:
set_current_job(None)
if not scan_videos(path): if not scan_videos(path):
state.complete_manual_batch(batch["id"]) state.complete_manual_batch(batch["id"])
return imported return imported
+11
View File
@@ -55,6 +55,17 @@ class State:
) )
self.conn.commit() self.conn.commit()
def get_app_state(self, key: str, default: str | None = None) -> str | None:
row = self.conn.execute("select value from app_state where key = ?", (key,)).fetchone()
return row["value"] if row else default
def set_app_state(self, key: str, value: str) -> None:
self.conn.execute(
"insert into app_state(key, value) values (?, ?) on conflict(key) do update set value=excluded.value",
(key, value),
)
self.conn.commit()
def add_manual_batch(self, path: Path) -> dict[str, Any]: def add_manual_batch(self, path: Path) -> dict[str, Any]:
self.conn.execute("insert or ignore into manual_batches(path) values (?)", (str(path),)) self.conn.execute("insert or ignore into manual_batches(path) values (?)", (str(path),))
self.conn.commit() self.conn.commit()
+1 -1
View File
@@ -1 +1 @@
body{font-family:system-ui,sans-serif;margin:0;background:#111827;color:#e5e7eb}header,main{max-width:1100px;margin:auto;padding:1rem}.topbar{display:flex;justify-content:space-between;gap:1rem;align-items:center;background:#0f172a}.build{text-align:right}.build strong{font-size:1.2rem}.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(12rem,1fr));gap:1rem}.cards article,.panel{background:#1f2937;border-radius:.75rem;padding:1rem;margin-top:1rem}strong{display:block;font-size:2rem}span,small,dd{color:#9ca3af}table{width:100%;border-collapse:collapse;background:#1f2937;margin-top:1rem}th,td{padding:.6rem;border-bottom:1px solid #374151;text-align:left}input,button{padding:.6rem;border-radius:.4rem;border:1px solid #374151}button{background:#38bdf8;color:#082f49;font-weight:700}.inline-form{display:flex;gap:.5rem;flex-wrap:wrap}.inline-form input[name=path]{min-width:min(100%,28rem);flex:1}.info{display:grid;grid-template-columns:10rem 1fr;gap:.4rem 1rem}.info dt{font-weight:700}.info dd{margin:0;overflow-wrap:anywhere}.state{background:#0f172a;border:1px solid #374151;border-radius:999px;padding:.15rem .5rem} body{font-family:system-ui,sans-serif;margin:0;background:#111827;color:#e5e7eb}header,main{max-width:1100px;margin:auto;padding:1rem}.topbar{display:flex;justify-content:space-between;gap:1rem;align-items:center;background:#0f172a}.build{text-align:right}.build strong{font-size:1.2rem}.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(12rem,1fr));gap:1rem}.cards article,.panel{background:#1f2937;border-radius:.75rem;padding:1rem;margin-top:1rem}strong{display:block;font-size:2rem}span,small,dd{color:#9ca3af}table{width:100%;border-collapse:collapse;background:#1f2937;margin-top:1rem}th,td{padding:.6rem;border-bottom:1px solid #374151;text-align:left}input,button{padding:.6rem;border-radius:.4rem;border:1px solid #374151}button{background:#38bdf8;color:#082f49;font-weight:700}.danger{background:#f87171;color:#450a0a}.controls{display:flex;gap:.5rem;flex-wrap:wrap}.inline-form{display:flex;gap:.5rem;flex-wrap:wrap}.inline-form input[name=path]{min-width:min(100%,28rem);flex:1}.info{display:grid;grid-template-columns:10rem 1fr;gap:.4rem 1rem}.info dt{font-weight:700}.info dd{margin:0;overflow-wrap:anywhere}.state{background:#0f172a;border:1px solid #374151;border-radius:999px;padding:.15rem .5rem}
+496
View File
@@ -0,0 +1,496 @@
#!/usr/bin/env python3
from __future__ import annotations
import html
import json
import os
import subprocess
import time
import urllib.parse
import urllib.request
from datetime import datetime, timedelta, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
SERVICE = "manual-media-import.service"
TIMER = "manual-media-import.timer"
LOG = Path("/var/log/manual-media-import.log")
IMPORTER_STATUS = Path("/run/manual-media-import/status.json")
MANUAL_BATCHES = Path("/var/lib/importarr/manual-batches.json")
QUEUE_ROOTS = {
"manual": Path(os.getenv("IMPORTARR_DOWNLOAD_ROOT", "/data/downloads/manual")),
"legacy": Path(os.getenv("IMPORTARR_LEGACY_DOWNLOAD_ROOT", "/data/downloads/legacy")),
}
VIDEO_EXT = {".mkv", ".mp4", ".m4v", ".avi", ".mov", ".wmv", ".mpg", ".mpeg", ".ts", ".m2ts", ".webm"}
SAB_CONFIG = Path(os.getenv("IMPORTARR_SABNZBD_CONFIG", "/config/sabnzbd/sabnzbd.ini"))
SAB_API = os.getenv("IMPORTARR_SABNZBD_URL", "http://sabnzbd:8080/api")
LONG_RUNTIME_SECONDS = 25 * 60
HIGH_MEMORY_BYTES = 8 * 1024**3
STALE_QUEUE_SECONDS = 6 * 60 * 60
def run(args: list[str]) -> str:
return subprocess.run(args, check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL).stdout
def systemctl_show(unit: str) -> dict[str, str]:
data = {}
for line in run(["systemctl", "show", unit, "--no-pager"]).splitlines():
if "=" in line:
key, value = line.split("=", 1)
data[key] = value
return data
def int_value(value: str | None) -> int | None:
try:
return int(value or "")
except ValueError:
return None
def timestamp_to_iso(usec: str | None) -> str | None:
value = int_value(usec)
if not value or value <= 0:
return None
return datetime.fromtimestamp(value / 1_000_000, tz=timezone.utc).isoformat()
def monotonic_runtime_seconds(service: dict[str, str]) -> int | None:
started = int_value(service.get("ActiveEnterTimestampMonotonic"))
if not started:
main_pid = int_value(service.get("MainPID"))
if not main_pid:
return None
etimes = run(["ps", "-o", "etimes=", "-p", str(main_pid)]).strip()
return int_value(etimes)
boot_ns = time.clock_gettime_ns(time.CLOCK_BOOTTIME)
runtime = int((boot_ns / 1000 - started) / 1_000_000)
return max(runtime, 0)
def scan_queue(root: Path) -> dict[str, object]:
files = dirs = bytes_total = 0
processing_files = processing_dirs = processing_bytes = 0
oldest = newest = None
top_level: list[dict[str, object]] = []
if not root.exists():
return {"path": str(root), "exists": False, "files": 0, "dirs": 0, "bytes": 0, "oldest": None, "newest": None, "topLevel": [], "items": [], "processingFiles": 0, "processingDirs": 0, "processingBytes": 0, "processing": [], "processingItems": []}
top_level_map: dict[Path, dict[str, object]] = {}
processing_map: dict[Path, dict[str, object]] = {}
ready_items: list[dict[str, object]] = []
processing_items: list[dict[str, object]] = []
for dirpath, dirnames, filenames in os.walk(root):
for filename in filenames:
path = Path(dirpath) / filename
if path.suffix.lower() not in VIDEO_EXT or "sample" in filename.lower() or "sample" in str(path.parent).lower():
continue
try:
st = path.stat()
except FileNotFoundError:
continue
oldest = st.st_mtime if oldest is None else min(oldest, st.st_mtime)
newest = st.st_mtime if newest is None else max(newest, st.st_mtime)
try:
rel = path.relative_to(root)
except ValueError:
rel = path
top = root / rel.parts[0] if rel.parts else path
sab = sab_state_for(top.name)
transient = top.name.startswith(('_UNPACK_', '__UNPACK__', '_FAILED_', '_ADMIN_'))
is_processing = transient or (sab and sab.get("ready") is False and sab.get("status") != "manual")
relative_dir = str(Path(*rel.parts[:-1])) if len(rel.parts) > 1 else ""
item_label = path.name if not relative_dir else f"{relative_dir} / {path.name}"
state = str(sab.get("state") if sab else ("unpacking" if transient else "ready"))
file_item = {"name": path.name, "label": item_label, "release": top.name, "relativeDir": relative_dir, "path": str(path), "bytes": st.st_size, "mtime": datetime.fromtimestamp(st.st_mtime, tz=timezone.utc).isoformat(), "state": state}
if is_processing:
processing_files += 1
processing_bytes += st.st_size
processing_items.append(file_item)
item = processing_map.setdefault(top, {"name": top.name, "type": "dir" if top.is_dir() else "file", "files": 0, "bytes": 0, "mtime": None})
item["files"] = int(item["files"]) + 1
item["bytes"] = int(item["bytes"]) + st.st_size
item["mtime"] = datetime.fromtimestamp(st.st_mtime, tz=timezone.utc).isoformat()
continue
files += 1
bytes_total += st.st_size
ready_items.append(file_item)
item = top_level_map.setdefault(top, {"name": top.name, "type": "dir" if top.is_dir() else "file", "files": 0, "bytes": 0, "mtime": None})
item["files"] = int(item["files"]) + 1
item["bytes"] = int(item["bytes"]) + st.st_size
item["mtime"] = datetime.fromtimestamp(st.st_mtime, tz=timezone.utc).isoformat()
dirs = sum(1 for item in top_level_map.values() if item["type"] == "dir")
processing_dirs = sum(1 for item in processing_map.values() if item["type"] == "dir")
top_level = sorted(top_level_map.values(), key=lambda item: str(item["name"]).lower())
processing = sorted(processing_map.values(), key=lambda item: str(item["name"]).lower())
return {
"path": str(root),
"exists": True,
"files": files,
"dirs": dirs,
"bytes": bytes_total,
"oldest": datetime.fromtimestamp(oldest, tz=timezone.utc).isoformat() if oldest else None,
"newest": datetime.fromtimestamp(newest, tz=timezone.utc).isoformat() if newest else None,
"topLevel": top_level[:100],
"items": ready_items[:500],
"processingFiles": processing_files,
"processingDirs": processing_dirs,
"processingBytes": processing_bytes,
"processing": processing[:100],
"processingItems": processing_items[:500],
}
def read_logs(limit: int = 100) -> list[dict[str, object]]:
if not LOG.exists():
return []
lines = LOG.read_text(errors="replace").splitlines()[-max(1, min(limit, 1000)):]
records = []
for line in lines:
try:
records.append(json.loads(line))
except json.JSONDecodeError:
records.append({"level": "RAW", "msg": line})
return records
def read_summaries() -> list[dict[str, object]]:
if not LOG.exists():
return []
summaries = []
for line in LOG.read_text(errors="replace").splitlines():
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
if record.get("msg") == "summary":
summaries.append(record)
return summaries
def read_importer_status() -> dict[str, object] | None:
if not IMPORTER_STATUS.exists():
return None
def infer_current_from_logs(logs: list[dict[str, object]]) -> dict[str, object] | None:
for record in reversed(logs):
if record.get("level") != "MOVE" or record.get("msg") not in {"moving", "moving sidecar"}:
continue
src = record.get("src")
dest = record.get("dest")
if not src or not dest:
continue
src_path = Path(str(src))
partial = Path(str(dest) + ".partial")
total = None
copied = None
try:
total = src_path.stat().st_size
except FileNotFoundError:
pass
try:
copied = partial.stat().st_size
except FileNotFoundError:
copied = None
percent = round((copied / total * 100), 2) if copied is not None and total else None
return {"phase": "copying", "src": str(src), "dest": str(dest), "partial": str(partial), "bytes_copied": copied, "bytes_total": total, "percent": percent, "media_type": record.get("media_type"), "source_tag": record.get("source_tag"), "kind": "inferred"}
return None
try:
return json.loads(IMPORTER_STATUS.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return None
def queue_status() -> dict[str, object]:
global _SAB_HISTORY_CACHE
_SAB_HISTORY_CACHE = None
roots = {name: scan_queue(path) for name, path in QUEUE_ROOTS.items()}
return {
"roots": roots,
"files": sum(int(r["files"]) for r in roots.values()),
"dirs": sum(int(r["dirs"]) for r in roots.values()),
"bytes": sum(int(r["bytes"]) for r in roots.values()),
"processingFiles": sum(int(r["processingFiles"]) for r in roots.values()),
"processingDirs": sum(int(r["processingDirs"]) for r in roots.values()),
"processingBytes": sum(int(r["processingBytes"]) for r in roots.values()),
}
def sab_api_key() -> str | None:
try:
import re
m = re.search(r"^api_key\s*=\s*(\S+)", SAB_CONFIG.read_text(errors="replace"), re.M)
return m.group(1) if m else None
except OSError:
return None
def sab_history() -> list[dict[str, object]]:
key = sab_api_key()
if not key:
return []
try:
q = urllib.parse.urlencode({"mode": "history", "output": "json", "limit": 200, "apikey": key})
data = json.load(urllib.request.urlopen(f"{SAB_API}?{q}", timeout=10))
return data.get("history", {}).get("slots", [])
except Exception:
return []
_SAB_HISTORY_CACHE: list[dict[str, object]] | None = None
def sab_state_for(folder_name: str) -> dict[str, object] | None:
global _SAB_HISTORY_CACHE
if _SAB_HISTORY_CACHE is None:
_SAB_HISTORY_CACHE = sab_history()
normalized = folder_name.removeprefix("_UNPACK_").removeprefix("__UNPACK__")
for item in _SAB_HISTORY_CACHE:
name = str(item.get("name") or "")
if name != normalized and name != folder_name:
continue
status = str(item.get("status") or "")
storage = str(item.get("storage") or "")
action = str(item.get("action_line") or "")
category = str(item.get("category") or item.get("cat") or "")
owned = category == "manual"
ready = owned and status == "Completed" and bool(storage) and "_UNPACK_" not in storage
state = "ready" if ready else ("ignored category " + category if not owned else (status.lower() if status else "sab pending"))
if action:
state = action
return {"ready": ready, "owned": owned, "category": category, "status": status, "storage": storage, "state": state}
return None
def read_manual_batches() -> list[str]:
try:
raw = json.loads(MANUAL_BATCHES.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return []
return [str(x) for x in raw] if isinstance(raw, list) else []
def write_manual_batches(items: list[str]) -> None:
MANUAL_BATCHES.parent.mkdir(parents=True, exist_ok=True)
MANUAL_BATCHES.write_text(json.dumps(sorted(set(items)), indent=2), encoding="utf-8")
def add_manual_batch(value: str) -> tuple[bool, str]:
value = value.strip().strip("/")
if not value:
return False, "missing folder"
base = QUEUE_ROOTS["manual"].resolve()
path = (base / value).resolve() if not value.startswith("/srv/") else Path(value).resolve()
if not (path == base or path.is_relative_to(base)):
return False, "folder must be under manual downloads"
if not path.is_dir():
return False, "folder does not exist"
items = read_manual_batches()
items.append(str(path))
write_manual_batches(items)
return True, str(path)
def status() -> dict[str, object]:
service = systemctl_show(SERVICE)
timer = systemctl_show(TIMER)
logs = read_logs(300)
all_logs = read_logs(1000)
summaries = read_summaries()
last_summary = summaries[-1] if summaries else None
cutoff_1h = datetime.now() - timedelta(hours=1)
cutoff_24h = datetime.now() - timedelta(hours=24)
processed_1h = 0
processed_24h = 0
processed_total = 0
runs_1h = 0
runs_24h = 0
runs_total = 0
for summary in summaries:
moved = int(summary.get("moved") or 0)
processed_total += moved
runs_total += 1
try:
ts = datetime.fromisoformat(str(summary.get("ts")))
except ValueError:
ts = None
if ts and ts >= cutoff_1h:
processed_1h += moved
runs_1h += 1
if ts and ts >= cutoff_24h:
processed_24h += moved
runs_24h += 1
queue = queue_status()
running = service.get("ActiveState") == "activating" or service.get("SubState") in {"start", "running"}
current = read_importer_status()
if running and (not current or current.get("phase") == "done"):
current = infer_current_from_logs(all_logs)
runtime = monotonic_runtime_seconds(service) if running else None
memory_current = int_value(service.get("MemoryCurrent"))
memory_peak = int_value(service.get("MemoryPeak"))
warnings = []
if runtime and runtime > LONG_RUNTIME_SECONDS:
warnings.append("manual-media-import.service has been running longer than 25 minutes")
if memory_peak and memory_peak > HIGH_MEMORY_BYTES:
warnings.append("manual-media-import.service peak memory is over 8 GiB")
if last_summary and int(last_summary.get("errors") or 0) > 0:
warnings.append("last importer summary reported errors")
now = time.time()
for name, root in queue["roots"].items():
oldest = root.get("oldest")
if oldest:
try:
age = now - datetime.fromisoformat(str(oldest)).timestamp()
if age > STALE_QUEUE_SECONDS:
warnings.append(f"{name} queue contains files older than 6 hours")
except ValueError:
pass
health_state = "warning" if warnings else ("running" if running else service.get("Result", "unknown"))
return {
"name": "Importarr",
"service": {
"unit": SERVICE,
"activeState": service.get("ActiveState"),
"subState": service.get("SubState"),
"result": service.get("Result"),
"running": running,
"mainPid": int_value(service.get("MainPID")),
"startedAt": timestamp_to_iso(service.get("ActiveEnterTimestampUSec")),
"runtimeSeconds": runtime,
"memoryCurrentBytes": memory_current,
"memoryPeakBytes": memory_peak,
},
"timer": {
"unit": TIMER,
"activeState": timer.get("ActiveState"),
"subState": timer.get("SubState"),
"lastTrigger": timestamp_to_iso(timer.get("LastTriggerUSec")),
"nextElapse": timestamp_to_iso(timer.get("NextElapseUSecRealtime")),
},
"queue": queue,
"lastSummary": last_summary,
"processed": {"last1h": processed_1h, "last24h": processed_24h, "total": processed_total, "runsLast1h": runs_1h, "runsLast24h": runs_24h, "runsTotal": runs_total},
"current": current,
"manualBatches": read_manual_batches(),
"health": {"state": health_state, "warnings": warnings},
}
def fast_health() -> dict[str, object]:
service = systemctl_show(SERVICE)
timer = systemctl_show(TIMER)
running = service.get("ActiveState") == "activating" or service.get("SubState") in {"start", "running"}
runtime = monotonic_runtime_seconds(service) if running else None
warnings = []
if runtime and runtime > LONG_RUNTIME_SECONDS:
warnings.append("manual-media-import.service has been running longer than 25 minutes")
return {"state": "warning" if warnings else ("running" if running else service.get("Result", "unknown")), "warnings": warnings, "service": service.get("ActiveState"), "timer": timer.get("ActiveState")}
def page() -> bytes:
s = status()
logs = read_logs(80)
def esc(value: object) -> str:
return html.escape("" if value is None else str(value))
def gib(value: object) -> str:
return "" if value is None else f"{int(value) / 1024**3:.2f} GiB"
def secs(value: object) -> str:
if value is None:
return ""
seconds = int(value)
return f"{seconds // 60}m {seconds % 60}s"
def job_name(path: object) -> str:
if not path:
return ""
name = Path(str(path)).name
return name[:37] + "..." if len(name) > 40 else name
title_icon = "📥"
title_runtime = secs(s["service"]["runtimeSeconds"])
current = s.get("current") or {}
progress = current.get("percent")
full_current_name = Path(str(current.get("src", ""))).name if current.get("src") else ""
current_src_path = Path(str(current.get("src", ""))) if current.get("src") else None
current_top_name = ""
current_relative_dir = ""
if current_src_path:
for root in QUEUE_ROOTS.values():
try:
rel = current_src_path.relative_to(root)
current_top_name = rel.parts[0] if rel.parts else current_src_path.name
current_relative_dir = str(Path(*rel.parts[:-1])) if len(rel.parts) > 1 else ""
break
except ValueError:
continue
current_name = job_name(current.get("src"))
title = f"{title_icon} {progress}% {title_runtime} - {current_name}" if progress is not None and current_name else f"{title_icon} - idle"
warnings = s["health"]["warnings"]
warning_html = "".join(f"<p class='warn'>{esc(w)}</p>" for w in warnings) or "<p class='ok'>None</p>"
log_text = "\n".join(esc(f"[{r.get('ts','')}] {r.get('level','')} {r.get('msg','')} {json.dumps(r, ensure_ascii=False)}") for r in logs)
source_target_html = f"<p class='tiny'><span>Source:</span> {esc(current.get('src'))}</p><p class='tiny'><span>Target:</span> {esc(current.get('dest'))}</p>" if current else ""
progress_html = f"<p class='filename'><strong>{esc(full_current_name)}</strong><br><span class='muted'>{esc(current_relative_dir)}</span></p><p>{esc(progress)}% · runtime {title_runtime}</p><progress max='100' value='{esc(progress or 0)}'></progress><p>{esc(current.get('phase'))} · {gib(current.get('bytes_copied'))} / {gib(current.get('bytes_total'))}</p>{source_target_html}" if current else "<p>—</p>"
current_row = f"<tr class='active'><td>▶</td><td class='filename'><strong>{esc(full_current_name)}</strong><br><span class='muted'>{esc(current_relative_dir)}</span></td><td>{esc(progress)}%</td><td><progress max='100' value='{esc(progress or 0)}'></progress></td><td>{title_runtime}</td></tr>" if current else ""
current_path = str(current_src_path) if current_src_path else ""
processing_items = [item for item in s["queue"]["roots"]["manual"]["processingItems"] if item["path"] != current_path]
ready_items = [item for item in s["queue"]["roots"]["manual"]["items"] if item["path"] != current_path]
processing_rows = "".join(f"<tr class='processing'><td>⏳</td><td class='filename'><strong>{esc(item['name'])}</strong><br><span class='muted'>{esc(item.get('relativeDir') or item['release'])}</span></td><td>unpacking</td><td>{gib(item.get('bytes'))}</td><td>waiting</td></tr>" for item in processing_items[:30])
ready_rows = "".join(f"<tr><td>◷</td><td class='filename'><strong>{esc(item['name'])}</strong><br><span class='muted'>{esc(item.get('relativeDir') or item['release'])}</span></td><td>ready</td><td>{gib(item.get('bytes'))}</td><td>queued</td></tr>" for item in ready_items[:30])
queue_rows = processing_rows + ready_rows or "<tr><td>✓</td><td colspan='4'>No video files waiting</td></tr>"
history = [r for r in reversed(logs) if r.get("level") == "MOVE" and r.get("msg") == "moving"][:12]
history_rows = "".join(f"<tr><td>✓</td><td class='filename'>{esc(Path(str(r.get('dest',''))).name)}</td><td>{esc(r.get('media_type',''))}</td><td colspan='2'>{esc(r.get('ts',''))}</td></tr>" for r in history) or "<tr><td>—</td><td colspan='4'>No recent imports</td></tr>"
body = f"""<!doctype html><html><head><meta charset='utf-8'><meta name='viewport' content='width=device-width, initial-scale=1'><meta http-equiv='refresh' content='10'><title>{esc(title)}</title><style>body{{font-family:system-ui,sans-serif;margin:0;background:#111;color:#eee}}header{{display:flex;align-items:center;gap:1rem;background:#3b3b3b;padding:.7rem 1.2rem;border-bottom:1px solid #111;flex-wrap:wrap}}header h1{{margin:0;font-size:1.5rem}}.pill{{background:#222;border:1px solid #555;padding:.35rem .7rem}}main{{padding:1rem}}.grid{{display:grid;grid-template-columns:repeat(auto-fit,minmax(260px,1fr));gap:1rem}}.card{{background:#1d1d1d;border:1px solid #333;border-radius:12px;padding:1rem;margin-bottom:1rem;min-width:0;overflow:hidden}}input,button{{padding:.55rem;border:1px solid #555;background:#2b2b2b;color:#eee}}button{{cursor:pointer}}.filename{{overflow-wrap:anywhere;word-break:break-word}}.tiny{{font-size:.78rem;color:#999;line-height:1.25;overflow-wrap:anywhere}}.tiny span{{color:#bbb}}.ok{{color:#60d394}}.warn{{color:#ffd166}}progress{{width:100%;height:1.2rem;accent-color:#7fd37f}}table{{width:100%;border-collapse:collapse;background:#2b2b2b}}th,td{{padding:.65rem;border-bottom:1px solid #111;text-align:left;vertical-align:middle}}th{{background:#444;color:#ddd}}tr:nth-child(even){{background:#333}}tr.active{{background:#3f4a3f}}tr.processing{{background:#4a4232}}pre{{white-space:pre-wrap;overflow-wrap:anywhere;max-height:28rem;overflow:auto}}.muted{{color:#aaa}}a{{color:#8ecae6}}</style></head><body><header><h1>📥 Importarr</h1><span class='pill'>{esc(s['queue']['files'])} videos ready</span><span class='pill'>{esc(s['queue']['processingFiles'])} unpacking</span><span class='pill'>ready {gib(s['queue']['bytes'])}</span><span class='pill'>1h {esc(s['processed']['last1h'])} · 24h {esc(s['processed']['last24h'])} · total {esc(s['processed']['total'])}</span></header><main><div class='grid'><section class='card'><h2>Status</h2><p class='{('warn' if warnings else 'ok')}'>{esc(s['health']['state'])}</p><p>Service: {esc(s['service']['activeState'])}/{esc(s['service']['subState'])}</p><p>Runtime: {secs(s['service']['runtimeSeconds'])}</p><p>Memory current: {gib(s['service']['memoryCurrentBytes'])}</p><p>Memory peak: {gib(s['service']['memoryPeakBytes'])}</p><p class='muted'>Scheduled automatically every 15 minutes. `_UNPACK_` folders are shown as unpacking, not ready.</p></section><section class='card'><h2>Current file</h2>{progress_html}</section><section class='card'><h2>Processed</h2><p>Last 1h: {esc(s['processed']['last1h'])} imported</p><p>Last 24h: {esc(s['processed']['last24h'])} imported</p><p>Total: {esc(s['processed']['total'])} imported</p></section><section class='card'><h2>Warnings</h2>{warning_html}</section></div><section class='card'><h2>Add manual batch</h2><form onsubmit="event.preventDefault();fetch('/api/manual-batches',{{method:'POST',headers:{{'content-type':'application/json'}},body:JSON.stringify({{path:this.path.value}})}}).then(()=>location.reload())"><input name='path' placeholder='folder under manual downloads' size='60'><button>Add folder once</button></form></section><section class='card'><h2>Jobs</h2><table><thead><tr><th></th><th>Name</th><th>State</th><th>Progress / Size</th><th>Runtime</th></tr></thead><tbody>{current_row}{queue_rows}</tbody></table></section><section class='card'><h2>History</h2><table><thead><tr><th></th><th>Name</th><th>Type</th><th colspan='2'>Time</th></tr></thead><tbody>{history_rows}</tbody></table></section><section class='card'><h2>Recent log</h2><pre>{log_text}</pre></section><p><a href='/api/status'>/api/status</a> · <a href='/api/queue'>/api/queue</a> · <a href='/api/logs?limit=100'>/api/logs</a></p></main></body></html>"""
return body.encode()
class Handler(BaseHTTPRequestHandler):
def send(self, code: int, content_type: str, data: bytes) -> None:
self.send_response(code)
self.send_header("Content-Type", content_type)
self.send_header("Cache-Control", "no-store")
self.end_headers()
self.wfile.write(data)
def do_GET(self) -> None:
parsed = urllib.parse.urlparse(self.path)
if parsed.path == "/":
self.send(200, "text/html; charset=utf-8", page())
elif parsed.path == "/api/status":
self.send(200, "application/json", json.dumps(status()).encode())
elif parsed.path == "/api/queue":
self.send(200, "application/json", json.dumps(queue_status()).encode())
elif parsed.path == "/api/logs":
params = urllib.parse.parse_qs(parsed.query)
limit = int(params.get("limit", ["100"])[0])
self.send(200, "application/json", json.dumps(read_logs(limit)).encode())
elif parsed.path == "/health":
self.send(200, "application/json", json.dumps(fast_health()).encode())
else:
self.send(404, "text/plain", b"not found")
def do_POST(self) -> None:
parsed = urllib.parse.urlparse(self.path)
if parsed.path != "/api/manual-batches":
self.send(404, "text/plain", b"not found")
return
length = int(self.headers.get("content-length") or 0)
try:
payload = json.loads(self.rfile.read(length) or b"{}")
except json.JSONDecodeError:
self.send(400, "application/json", json.dumps({"ok": False, "error": "invalid json"}).encode())
return
ok, message = add_manual_batch(str(payload.get("path") or ""))
self.send(200 if ok else 400, "application/json", json.dumps({"ok": ok, "result": message}).encode())
def log_message(self, fmt: str, *args: object) -> None:
return
def main() -> None:
host = os.getenv("IMPORTARR_BIND_HOST", "0.0.0.0")
port = int(os.getenv("IMPORTARR_BIND_PORT", "8095"))
ThreadingHTTPServer((host, port), Handler).serve_forever()
if __name__ == "__main__":
main()
+14 -1
View File
@@ -32,8 +32,20 @@
<dt>Movies root</dt><dd>{{ status.movies_root }}</dd> <dt>Movies root</dt><dd>{{ status.movies_root }}</dd>
<dt>TV root</dt><dd>{{ status.tv_root }}</dd> <dt>TV root</dt><dd>{{ status.tv_root }}</dd>
<dt>Write auth</dt><dd>{{ 'enabled' if status.auth_enabled else 'disabled' }}</dd> <dt>Write auth</dt><dd>{{ 'enabled' if status.auth_enabled else 'disabled' }}</dd>
<dt>Queue mode</dt><dd id="queue-mode">{{ status.control.queue_mode }}</dd>
<dt>Current job</dt><dd id="current-job">{{ status.current or 'idle' }}</dd>
</dl> </dl>
</section> </section>
<section class="panel">
<h2>Queue controls</h2>
<p>Pause and stop prevent new jobs from being added to the queue. They do not interrupt an import already in progress; use cancel current job for that.</p>
<div class="controls">
<button type="button" data-control="start">Start</button>
<button type="button" data-control="pause">Pause</button>
<button type="button" data-control="stop">Stop</button>
<button type="button" data-control="cancel-current" class="danger">Cancel current job</button>
</div>
</section>
<section> <section>
<h2>Manual batches</h2> <h2>Manual batches</h2>
<form id="batch-form" class="inline-form"> <form id="batch-form" class="inline-form">
@@ -51,7 +63,8 @@
</section> </section>
</main> </main>
<script> <script>
async function refresh(){ const r=await fetch('/api/jobs'); const d=await r.json(); document.title=d.jobs.length?`📥 ${d.jobs.length} jobs - Importarr`:'📥 - idle · Importarr'; document.getElementById('jobs').innerHTML='<table><tr><th>Name</th><th>State</th><th>Context</th></tr>'+d.jobs.map(j=>`<tr><td>${j.name||''}</td><td><span class="state">${j.state}</span></td><td><small>${j.relative_path||j.storage||j.reason||''}</small></td></tr>`).join('')+'</table>'; } async function refresh(){ const r=await fetch('/api/jobs'); const d=await r.json(); document.title=d.jobs.length?`📥 ${d.jobs.length} jobs - Importarr`:'📥 - idle · Importarr'; document.getElementById('jobs').innerHTML='<table><tr><th>Name</th><th>State</th><th>Context</th></tr>'+d.jobs.map(j=>`<tr><td>${j.name||''}</td><td><span class="state">${j.state}</span></td><td><small>${j.relative_path||j.storage||j.reason||''}</small></td></tr>`).join('')+'</table>'; if(d.control){ document.getElementById('queue-mode').textContent=d.control.queue_mode; document.getElementById('current-job').textContent=d.control.current||'idle'; } }
document.querySelectorAll('[data-control]').forEach(button=>button.addEventListener('click', async()=>{ await fetch(`/api/control/${button.dataset.control}`,{method:'POST'}); await refresh(); }));
document.getElementById('browse-batch').addEventListener('click',()=>document.getElementById('batch-picker').click()); document.getElementById('browse-batch').addEventListener('click',()=>document.getElementById('batch-picker').click());
document.getElementById('batch-picker').addEventListener('change',e=>{ const f=e.target.files[0]; if(!f)return; const top=(f.webkitRelativePath||'').split('/')[0]; if(top) document.querySelector('#batch-form [name="path"]').value=top; }); document.getElementById('batch-picker').addEventListener('change',e=>{ const f=e.target.files[0]; if(!f)return; const top=(f.webkitRelativePath||'').split('/')[0]; if(top) document.querySelector('#batch-form [name="path"]').value=top; });
document.getElementById('batch-form').addEventListener('submit', async e=>{ e.preventDefault(); await fetch('/api/manual-batches',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({path:e.target.path.value})}); location.reload(); }); document.getElementById('batch-form').addEventListener('submit', async e=>{ e.preventDefault(); await fetch('/api/manual-batches',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({path:e.target.path.value})}); location.reload(); });
+1093
View File
File diff suppressed because it is too large Load Diff
+2
View File
@@ -22,6 +22,8 @@ test = ["pytest>=8.2", "pytest-asyncio>=0.23"]
[project.scripts] [project.scripts]
importarr = "importarr.main:run" importarr = "importarr.main:run"
importarr-status = "importarr.status_ui:main"
manual-media-import = "importarr.worker:main"
[tool.pytest.ini_options] [tool.pytest.ini_options]
testpaths = ["tests"] testpaths = ["tests"]
+80
View File
@@ -0,0 +1,80 @@
from importarr.config import Settings
from importarr.state import State
def configure_main(tmp_path, monkeypatch):
monkeypatch.setenv("IMPORTARR_STATE_PATH", str(tmp_path / "initial.db"))
import importarr.main as main
download = tmp_path / "downloads"
movies = tmp_path / "movies"
tv = tmp_path / "tv"
monkeypatch.setattr(main, "settings", Settings(download_root=download, movies_root=movies, tv_root=tv, state_path=tmp_path / "state.db"))
monkeypatch.setattr(main, "state", State(tmp_path / "state.db"))
return main, download, movies, tv
def test_pause_prevents_manual_queue_sync(tmp_path, monkeypatch):
main, download, _movies, _tv = configure_main(tmp_path, monkeypatch)
batch = download / "Release"
batch.mkdir(parents=True)
(batch / "Movie.mkv").write_bytes(b"movie")
main.state.add_manual_batch(batch)
main.state.set_app_state("queue_mode", "paused")
main.sync_manual_queue()
assert main.state.list_queue_items() == []
def test_start_reenables_manual_queue_sync(tmp_path, monkeypatch):
main, download, _movies, _tv = configure_main(tmp_path, monkeypatch)
batch = download / "Release"
batch.mkdir(parents=True)
(batch / "Movie.mkv").write_bytes(b"movie")
main.state.add_manual_batch(batch)
main.state.set_app_state("queue_mode", "paused")
main.sync_manual_queue()
main.state.set_app_state("queue_mode", "running")
main.sync_manual_queue()
assert len(main.state.list_queue_items()) == 1
def test_cancel_current_stops_before_next_manual_item(tmp_path, monkeypatch):
main, download, movies, tv = configure_main(tmp_path, monkeypatch)
batch = download / "Release"
batch.mkdir(parents=True)
(batch / "A.mkv").write_bytes(b"a")
(batch / "B.mkv").write_bytes(b"b")
main.state.add_manual_batch(batch)
main.sync_manual_queue()
main.state.set_app_state("cancel_requested", "true")
assert main._import_manual_batches(main.Importer(movies, tv)) == 0
assert len(main.state.list_queue_items()) == 2
def test_cancel_current_stops_active_copy(tmp_path, monkeypatch):
main, download, movies, tv = configure_main(tmp_path, monkeypatch)
batch = download / "Release"
batch.mkdir(parents=True)
source = batch / "A.mkv"
source.write_bytes(b"a" * (1024 * 1024 + 1))
main.state.add_manual_batch(batch)
main.sync_manual_queue()
calls = 0
def cancel_during_copy() -> bool:
nonlocal calls
calls += 1
return calls > 1
monkeypatch.setattr(main, "consume_cancel_request", cancel_during_copy)
assert main._import_manual_batches(main.Importer(movies, tv)) == 0
assert source.exists()
assert not any(movies.glob("*.partial"))
assert main.state.list_queue_items(active_only=False)[0]["state"] == "skipped"