daniels/importarr
1
0
Fork 0
Code Issues 21 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Remove private deployment details

Browse Source
This commit is contained in:
daniels
2026-07-29 14:13:22 +02:00
parent 67335a37b9
commit 2f996600c6
10 changed files with 63 additions and 212 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -5,3 +5,4 @@ __pycache__/
*.pyc
*.db
*.partial
AGENTS.local.md
AGENTS.md
-104
View File
@@ -1,104 +0,0 @@
# Importarr Agent Instructions
Importarr is owned as a Linux-ops-managed service repository. Treat this checkout as the source of truth for application code, deployment files, and local service installs.
## Repository Source Of Truth
- Work from `/srv/opencode-workspace/importarr` for Importarr code and deploy changes.
- The live service is installed on `dgsserver1` from a repo checkout at `/opt/importarr/repo`; gmk1 installs are not the public Importarr service.
- Do not edit host-local legacy scripts as the normal workflow:
- `/usr/local/sbin/importarr-status.py`
- `/usr/local/sbin/manual-media-import.py`
- If an emergency hotfix is made outside the repo, backport it here immediately and reinstall from the repo.
## Seamless Feature Workflow
When asked to implement an Importarr feature, fix, UI change, deployment change, or operational behavior change:
1. Inspect `git status --short --branch` before editing.
2. Implement the smallest correct repo change.
3. Run the narrowest useful verification, normally:
```sh
.venv/bin/pytest -q
```
If shell deploy scripts changed, also run:
```sh
sh -n deploy/systemd-install.sh && sh -n deploy/repo-upgrade.sh
```
4. Inspect `git diff` and ensure no secrets, raw `.env`, tokens, databases, or private material are included.
5. Commit and push completed Importarr changes by default unless the user explicitly asks not to publish or verification is blocked.
6. Install/restart on `dgsserver1` from the repository so the public service matches the repo:
```sh
ssh -p 2222 opencode@dgsserver1 'sudo -n sh /opt/importarr/repo-upgrade.sh'
```
7. Verify the live service:
```sh
make verify-live
```
8. If the live install fails, inspect `systemctl status importarr.service` and `journalctl -u importarr.service`; fix the repo, commit/push the fix, reinstall, and verify again.
## Gitea Issue Completion Workflow
When the user points an agent at an Importarr Gitea issue and asks to solve it, the expected end-to-end flow is mandatory unless the user explicitly says not to deploy:
1. Read the issue first, including comments and acceptance criteria.
2. Implement the smallest correct repo change on a dedicated issue branch when appropriate.
3. Run the narrowest useful tests/checks.
4. Commit and push the completed change to `main` or merge/push the issue branch as instructed.
5. Update the Gitea issue with what changed and the verification that ran, then close it when solved.
6. Update the `dgsserver1` install from the pushed repo state:
```sh
ssh -p 2222 opencode@dgsserver1 'sudo -n sh /opt/importarr/repo-upgrade.sh'
```
7. Restart/verify the `dgsserver1` service from the repo-managed install and confirm the public URL:
```sh
ssh -p 2222 opencode@dgsserver1 'sudo -n systemctl status importarr.service --no-pager -l'
curl -fsS https://importarr.delphas.dk/health
```
If the worker timer is intentionally paused, do not re-enable it unless the issue explicitly includes worker scheduling or the user approves.
## Install Model
- The service virtualenv lives at `/opt/importarr/venv` on `dgsserver1`.
- The status UI systemd unit runs `/opt/importarr/venv/bin/importarr-status` from the packaged repo install.
- The manual worker systemd unit runs `/opt/importarr/venv/bin/manual-media-import` from the packaged repo install.
- The package is installed from the repository into the venv using normal wheel/package install, not editable install.
- Do **not** use editable install for the system service: the unprivileged `importarr` user may not be able to read `/srv/opencode-workspace/importarr`, causing `ModuleNotFoundError` at startup.
- `/opt/importarr/repo-upgrade.sh` is the pull-and-reinstall helper for machines that should follow pushed `main`.
## Local Commands
```sh
make test
make install-systemd
make upgrade-local
make repo-upgrade
make verify-live
```
`make repo-upgrade` is for pulling already-pushed changes with `git pull --ff-only`. It refuses to run with uncommitted repo changes.
## Runtime Defaults On This Host
- Public URL: `https://importarr.delphas.dk/`
- dgsserver1 local URL: `http://127.0.0.1:8095/`
- Health: `http://127.0.0.1:8095/health`
- Status: `http://127.0.0.1:8095/api/status`
- Systemd service: `importarr.service`
- Env file: `/etc/importarr/importarr.env`
- SQLite state: `/var/lib/importarr/importarr.db`
## Secret Handling
- Prefer `*_FILE` settings for secrets, for example:
- `IMPORTARR_SAB_API_KEY_FILE`
- `IMPORTARR_AUTH_TOKEN_FILE`
- `IMPORTARR_RADARR_API_KEY_FILE`
- `IMPORTARR_SONARR_API_KEY_FILE`
- Never commit real env files, API keys, tokens, private keys, service databases, or backup data.
- Template files may list variable names with placeholder values or commented examples only.
## Linux Ops Follow-Through
For changes that materially alter the live service setup, ports, routes, monitoring, backup coverage, or host ownership, also follow the linux-ops documentation/systems-overview update rules. Do not mix unrelated pre-existing uncommitted changes from `linux-ops-docs` or `systems-overview` into Importarr commits.
Makefile
+10 -12
View File
@@ -1,9 +1,11 @@
PYTHON ?= .venv/bin/python
PIP ?= .venv/bin/pip
SERVICE ?= importarr.service
LIVE_URL ?= http://127.0.0.1:8095
IMPORTARR_PREFIX ?= /opt/importarr
IMPORTARR_REPO_DIR ?= $(CURDIR)
IMPORTARR_URL ?= http://127.0.0.1:8765
.PHONY: test install-systemd install-from-repo upgrade-local repo-upgrade verify-live
.PHONY: test install-systemd install-from-repo repo-upgrade verify
test:
$(PYTHON) -m pytest
@@ -12,16 +14,12 @@ install-systemd:
sudo -n sh deploy/systemd-install.sh
install-from-repo:
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
upgrade-local:
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
sudo -n systemctl restart $(SERVICE)
sudo -n $(IMPORTARR_PREFIX)/venv/bin/pip install --upgrade $(IMPORTARR_REPO_DIR)
repo-upgrade:
sudo -n sh /opt/importarr/repo-upgrade.sh
sudo -n IMPORTARR_PREFIX=$(IMPORTARR_PREFIX) IMPORTARR_REPO_DIR=$(IMPORTARR_REPO_DIR) sh deploy/repo-upgrade.sh
verify-live:
curl -fsS $(LIVE_URL)/health
curl -fsS $(LIVE_URL)/api/status
curl -fsS $(LIVE_URL)/api/preview
verify:
curl -fsS $(IMPORTARR_URL)/health
curl -fsS $(IMPORTARR_URL)/api/status
curl -fsS $(IMPORTARR_URL)/api/preview
README.md
+14 -56
View File
@@ -2,17 +2,17 @@
Importarr is an Arr-style service for importing manually categorized SABnzbd downloads after SAB reports final completion. It owns one SAB category, defaults to `manual`, and refuses to import transient Direct Unpack paths or jobs still in SAB queue/post-processing.
## New-machine install
## Install
Importarr is intended to feel like a small Arr service: deploy the container or systemd service, edit one env file, point SABnzbd category `manual` at the same completed-download path, then open the web UI.
### Docker Compose, recommended
```sh
mkdir -p /opt/importarr/config
cd /opt/importarr
curl -fsSLO https://gitea.delphas.dk/daniels/importarr/raw/branch/main/deploy/docker-compose.example.yml
curl -fsSLo importarr.env https://gitea.delphas.dk/daniels/importarr/raw/branch/main/deploy/importarr.env.example
mkdir -p importarr/config
cd importarr
curl -fsSLO https://example.com/importarr/deploy/docker-compose.example.yml
curl -fsSLo importarr.env https://example.com/importarr/deploy/importarr.env.example
${EDITOR:-vi} importarr.env
docker compose -f docker-compose.example.yml --env-file importarr.env up -d
```
@@ -26,25 +26,14 @@ docker build -t importarr:local .
### systemd / pip install
```sh
git clone https://gitea.delphas.dk/daniels/importarr.git
git clone https://example.com/importarr.git
cd importarr
sudo sh deploy/systemd-install.sh
sudo ${EDITOR:-vi} /etc/importarr/importarr.env
sudo systemctl start importarr.service
```
The installer creates the `importarr` system user when needed, installs a virtualenv at `/opt/importarr/venv`, and installs the package from the checked-out repository. The repository is therefore the source of truth: pull or edit the repo, reinstall/restart from the repo, and the service runs the package built from that code.
For local upgrades from a checked-out repo on dgsserver1, use the repo workflow instead of editing live scripts:
```sh
cd /srv/opencode-workspace/importarr
.venv/bin/python -m pytest
git status --short --branch
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
sudo -n systemctl restart importarr.service
make verify-live
```
The installer creates the `importarr` system user when needed, installs a virtualenv, and installs the package from the checked-out repository. Override install paths with `IMPORTARR_*` variables if the defaults do not fit your environment.
For a machine that should stay current with the repository, use the installed repo-upgrade helper:
@@ -54,7 +43,7 @@ sudo -n sh /opt/importarr/repo-upgrade.sh
The helper refuses to run when the checkout has uncommitted changes, then performs `git pull --ff-only`, reinstalls the package from the repo, restarts `importarr.service`, and prints service status. Use it after changes have been committed and pushed to `main`.
Release-worthy changes should be committed, tagged with SemVer (`v0.1.1`, `v0.2.0`, ...), pushed with tags, then reinstalled from the tagged checkout or artifact. Do not hand-edit `/usr/local/sbin/importarr-status.py` or `/usr/local/sbin/manual-media-import.py` except for a documented emergency hotfix that is immediately backported here.
Release-worthy changes should be committed, tagged with SemVer (`v0.1.1`, `v0.2.0`, ...), pushed with tags, then installed from the tagged checkout or artifact.
### Required setup
@@ -96,46 +85,15 @@ pytest
uvicorn importarr.main:app --reload
```
## Migration notes for dgsserver1
## Operations
Export the existing script settings into `IMPORTARR_*` env vars, add historical folders as explicit manual batches, run a dry-run/inspection through `/api/preview`, then switch the systemd service or Compose route after the ready set matches expectations.
Importarr intentionally does not document private deployment topology, hostnames, reverse proxies, monitoring, backups, or operator workflows in this repository. Keep those details in your own ops runbooks.
On dgsserver1, the packaged service is the only intended active entrypoint after cutover. Keep `manual-media-import.timer` disabled unless a repo-managed worker/timer replaces it later.
For a systemd install, prefer a normal package install from the checked-out repo over an editable install so the service user does not need read access to your development checkout.
## Repository-as-install workflow
Importarr should not drift into host-local scripts. Treat the checked-out repository as the install source:
1. Make changes in `/srv/opencode-workspace/importarr`.
2. Run tests: `make test`.
3. Commit and push the repo change.
4. Install/restart from the same repo: `make upgrade-local` for local changes, or `make repo-upgrade` to pull the latest pushed `main` and restart.
5. Verify the live service: `make verify-live`.
Do not edit `/usr/local/sbin/importarr-status.py`, `/usr/local/sbin/manual-media-import.py`, or files copied out of the repo as the normal workflow. If an emergency live hotfix is unavoidable, backport it to this repository immediately and run the repo install workflow again.
### Local install lessons learned
- The live systemd service runs as the unprivileged `importarr` user.
- Do not install the system service with `pip install --editable /srv/opencode-workspace/importarr`; that can fail at startup if the service user cannot read the workspace checkout.
- The supported local service install is a normal package install from the repo into `/opt/importarr/venv`:
If the service fails, standard systemd diagnostics are usually enough:
```sh
sudo -n /opt/importarr/venv/bin/pip install --upgrade /srv/opencode-workspace/importarr
sudo -n systemctl restart importarr.service
```
- `deploy/systemd-install.sh`, `make upgrade-local`, and `/opt/importarr/repo-upgrade.sh` already use this supported model.
- After every implementation task that should affect the live local service, run:
```sh
make upgrade-local
make verify-live
```
- If `make verify-live` fails, check:
```sh
sudo -n systemctl --no-pager --full status importarr.service
sudo -n journalctl -u importarr.service -n 120 --no-pager
sudo systemctl --no-pager --full status importarr.service
sudo journalctl -u importarr.service -n 120 --no-pager
```
deploy/docker-compose.example.yml
+1 -1
View File
@@ -6,7 +6,7 @@ services:
- "8765:8765"
volumes:
- ./config:/config
- /srv/scrypted/sabnzbd-data/downloads/manual:/data/downloads/manual
- /path/to/downloads/manual:/data/downloads/manual
- /path/to/movies:/data/movies
- /path/to/tv:/data/tv
restart: unless-stopped
deploy/importarr.env.example
+7 -7
View File
@@ -1,16 +1,16 @@
IMPORTARR_SAB_URL=http://127.0.0.1:8080
IMPORTARR_SAB_URL=http://sabnzbd:8080
# Prefer *_FILE for secrets. Plain env vars still work for local/dev installs.
# IMPORTARR_SAB_API_KEY=change-me
# IMPORTARR_SAB_API_KEY_FILE=/etc/importarr/sab-api-key
IMPORTARR_SAB_CATEGORY=manual
IMPORTARR_DOWNLOAD_ROOT=/srv/scrypted/sabnzbd-data/downloads/manual
IMPORTARR_MOVIES_ROOT=/srv/media/movies
IMPORTARR_TV_ROOT=/srv/media/tv
IMPORTARR_STATE_PATH=/var/lib/importarr/importarr.db
IMPORTARR_DOWNLOAD_ROOT=/data/downloads/manual
IMPORTARR_MOVIES_ROOT=/data/movies
IMPORTARR_TV_ROOT=/data/tv
IMPORTARR_STATE_PATH=/config/importarr.db
IMPORTARR_LOG_LEVEL=info
# IMPORTARR_AUTH_TOKEN=change-me
# IMPORTARR_AUTH_TOKEN_FILE=/etc/importarr/auth-token
IMPORTARR_BIND_HOST=0.0.0.0
IMPORTARR_BIND_PORT=8095
IMPORTARR_BIND_PORT=8765
IMPORTARR_POLL_SECONDS=60
IMPORTARR_REPO_DIR=/srv/opencode-workspace/importarr
# IMPORTARR_REPO_DIR=/path/to/importarr
deploy/manual-media-import-failure.service
+1 -1
View File
@@ -3,4 +3,4 @@ Description=Critical alert when manual media importer fails
[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /usr/local/sbin/ha-critical-notify.py service-recovery "manual-media-import.service failed" --host dgsserver1 --resource manual-media-import --details "Check journalctl -u manual-media-import.service and /var/log/manual-media-import.log"
ExecStart=/usr/bin/logger -t importarr "manual-media-import.service failed; check journalctl -u manual-media-import.service"
deploy/repo-upgrade.sh
+4 -3
View File
@@ -12,9 +12,10 @@ if [ -f "$ENV_FILE" ]; then
. "$ENV_FILE"
fi
REPO_DIR=${IMPORTARR_REPO_DIR:-/srv/opencode-workspace/importarr}
PREFIX=${IMPORTARR_PREFIX:-/opt/importarr}
REPO_DIR=${IMPORTARR_REPO_DIR:-$(pwd)}
SERVICE=${IMPORTARR_SERVICE:-importarr.service}
VENV=${IMPORTARR_VENV:-/opt/importarr/venv}
VENV=${IMPORTARR_VENV:-$PREFIX/venv}
if [ ! -d "$REPO_DIR/.git" ]; then
echo "Importarr repo not found at $REPO_DIR" >&2
@@ -33,7 +34,7 @@ git pull --ff-only
"$VENV/bin/pip" install --upgrade "$REPO_DIR"
GIT_SHA="$(git rev-parse --short=12 HEAD 2>/dev/null || printf development)"
BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
cat > /opt/importarr/build.env <<EOF
cat > "$PREFIX/build.env" <<EOF
IMPORTARR_GIT_SHA=$GIT_SHA
IMPORTARR_BUILD_DATE=$BUILD_DATE
EOF
importarr/status_ui.py
+4 -4
View File
@@ -18,12 +18,12 @@ LOG = Path("/var/log/manual-media-import.log")
IMPORTER_STATUS = Path("/run/manual-media-import/status.json")
MANUAL_BATCHES = Path("/var/lib/importarr/manual-batches.json")
QUEUE_ROOTS = {
"manual": Path("/srv/scrypted/sabnzbd-data/downloads/manual"),
"danish_legacy": Path("/srv/scrypted/sabnzbd-data/downloads/danish"),
"manual": Path(os.getenv("IMPORTARR_DOWNLOAD_ROOT", "/data/downloads/manual")),
"legacy": Path(os.getenv("IMPORTARR_LEGACY_DOWNLOAD_ROOT", "/data/downloads/legacy")),
}
VIDEO_EXT = {".mkv", ".mp4", ".m4v", ".avi", ".mov", ".wmv", ".mpg", ".mpeg", ".ts", ".m2ts", ".webm"}
SAB_CONFIG = Path("/opt/stacks/media-transform/sabnzbd/config/sabnzbd.ini")
SAB_API = "http://127.0.0.1:8080/api"
SAB_CONFIG = Path(os.getenv("IMPORTARR_SABNZBD_CONFIG", "/config/sabnzbd/sabnzbd.ini"))
SAB_API = os.getenv("IMPORTARR_SABNZBD_URL", "http://sabnzbd:8080/api")
LONG_RUNTIME_SECONDS = 25 * 60
HIGH_MEMORY_BYTES = 8 * 1024**3
STALE_QUEUE_SECONDS = 6 * 60 * 60
importarr/worker.py
+21 -24
View File
@@ -1,10 +1,5 @@
#!/usr/bin/env python3
"""Import manual SABnzbd downloads into Jellyfin movie/TV library roots.
Run on dgsserver1. Intended live install path:
`/usr/local/sbin/manual-media-import.py`, managed by
`manual-media-import.timer`.
"""
"""Import manual SABnzbd downloads into Jellyfin movie/TV library roots."""
from __future__ import annotations
@@ -25,38 +20,38 @@ from collections import Counter
from datetime import datetime
from pathlib import Path
MANUAL_DOWNLOADS = Path("/srv/scrypted/sabnzbd-data/downloads/manual")
LEGACY_DANISH_DOWNLOADS = Path("/srv/scrypted/sabnzbd-data/downloads/danish")
MANUAL_DOWNLOADS = Path(os.getenv("IMPORTARR_DOWNLOAD_ROOT", "/data/downloads/manual"))
LEGACY_DANISH_DOWNLOADS = Path(os.getenv("IMPORTARR_LEGACY_DOWNLOAD_ROOT", "/data/downloads/legacy"))
DOWNLOAD_ROOTS = [MANUAL_DOWNLOADS, LEGACY_DANISH_DOWNLOADS]
SAB_TRANSIENT_PREFIXES = ("_UNPACK_", "__UNPACK__", "_FAILED_", "_ADMIN_")
HOST_MEDIA_PREFIX = "/srv/media"
MOVIES_ROOT = Path("/srv/media/movies")
TV_ROOT = Path("/srv/media/tv")
HOST_MEDIA_PREFIX = os.getenv("IMPORTARR_HOST_MEDIA_PREFIX", "/data")
MOVIES_ROOT = Path(os.getenv("IMPORTARR_MOVIES_ROOT", "/data/movies"))
TV_ROOT = Path(os.getenv("IMPORTARR_TV_ROOT", "/data/tv"))
LOG = Path("/var/log/manual-media-import.log")
LOCK = Path("/run/manual-media-import.lock")
STATUS = Path("/run/manual-media-import/status.json")
MANUAL_BATCHES = Path("/var/lib/importarr/manual-batches.json")
HOME_ASSISTANT_WEBHOOK = "https://hass.delphas.dk/api/webhook/jellyfin_event"
HOME_ASSISTANT_WEBHOOK = os.getenv("IMPORTARR_HOME_ASSISTANT_WEBHOOK", "")
RADARR_CONFIG = Path("/opt/stacks/media-transform/radarr/config/config.xml")
RADARR_URL = "http://127.0.0.1:7878"
RADARR_DB = Path("/opt/stacks/media-transform/radarr/config/radarr.db")
SONARR_CONFIG = Path("/opt/stacks/media-transform/sonarr/config/config.xml")
SONARR_URL = "http://127.0.0.1:8989"
SONARR_DB = Path("/opt/stacks/media-transform/sonarr/config/sonarr.db")
SABNZBD_CONFIG = Path("/opt/stacks/media-transform/sabnzbd/config/sabnzbd.ini")
SABNZBD_URL = "http://127.0.0.1:8080/api"
RADARR_CONFIG = Path(os.getenv("IMPORTARR_RADARR_CONFIG", "/config/radarr/config.xml"))
RADARR_URL = os.getenv("IMPORTARR_RADARR_URL", "http://radarr:7878")
RADARR_DB = Path(os.getenv("IMPORTARR_RADARR_DB", "/config/radarr/radarr.db"))
SONARR_CONFIG = Path(os.getenv("IMPORTARR_SONARR_CONFIG", "/config/sonarr/config.xml"))
SONARR_URL = os.getenv("IMPORTARR_SONARR_URL", "http://sonarr:8989")
SONARR_DB = Path(os.getenv("IMPORTARR_SONARR_DB", "/config/sonarr/sonarr.db"))
SABNZBD_CONFIG = Path(os.getenv("IMPORTARR_SABNZBD_CONFIG", "/config/sabnzbd/sabnzbd.ini"))
SABNZBD_URL = os.getenv("IMPORTARR_SABNZBD_URL", "http://sabnzbd:8080/api")
ARR_API_TIMEOUT = 10
VIDEO_EXT = {".mkv", ".mp4", ".m4v", ".avi", ".mov", ".wmv", ".mpg", ".mpeg", ".ts", ".m2ts", ".webm"}
SIDECAR_EXT = {".srt", ".ass", ".sub", ".idx", ".nfo"}
MIN_SIZE = 100 * 1024 * 1024
IMDB_DATASET_CANDIDATES = [
Path("/srv/media/imdb"),
Path("/srv/media/imdb-datasets"),
Path("/srv/media/.cache/imdb"),
Path("/data/imdb"),
Path("/data/imdb-datasets"),
Path("/data/.cache/imdb"),
Path("/var/lib/imdb"),
Path("/opt/imdb"),
]
@@ -869,6 +864,8 @@ def copy_then_remove(src: Path, dest: Path, *, media_type: str | None = None, so
def notify_home_assistant(item: dict[str, object], dest: Path, source_tag: str) -> None:
if not HOME_ASSISTANT_WEBHOOK:
return
item_type = str(item.get("item_type") or "Movie")
name = item.get("title") or dest.parent.name
payload_body: dict[str, object] = {
@@ -906,7 +903,7 @@ def source_tag_for(path: Path) -> str:
for root in DOWNLOAD_ROOTS:
try:
path.relative_to(root)
return "manual" if root == MANUAL_DOWNLOADS else "danish-legacy"
return "manual" if root == MANUAL_DOWNLOADS else "legacy"
except ValueError:
continue
return "manual"