{% extends 'workflows/base_shell.html' %}
{% load static %}

{% block title %}Developer Handbook{% endblock %}



{% block extra_css %}
<link rel="stylesheet" href="{% static 'workflows/css/docs_pages.css' %}" />
{% endblock %}

{% block shell_body %}
      {% include 'workflows/includes/app_header.html' with header_show_home=1 header_inside_shell=1 %}
    <div class="page-stack">
    <div class="top">
      <h1>Developer Handbook</h1>
      <div style="display:flex; gap:8px; flex-wrap:wrap;">
        <a class="btn btn-secondary" href="/admin-tools/wiki/">Project Wiki</a>
      </div>
    </div>
    <p class="sub">Engineering runbook for development, deployment, maintenance, and extension of the current company portal deployment.</p>

    <div class="toc">
      <a href="#overview">Overview</a>
      <a href="#structure">Structure</a>
      <a href="#local">Local Dev</a>
      <a href="#docker">Docker</a>
      <a href="#db">Database</a>
      <a href="#translations">Translations</a>
      <a href="#pdf">PDF Pipeline</a>
      <a href="#email">Email Pipeline</a>
      <a href="#nextcloud">Nextcloud</a>
      <a href="#builders">Builders</a>
      <a href="#testing">Testing</a>
      <a href="#backup">Backup</a>
      <a href="#cicd">CI/CD</a>
      <a href="#deploy">Deployment</a>
      <a href="#troubleshooting">Troubleshooting</a>
      <a href="#security">Security</a>
    </div>

    <h2 id="overview">1) Overview</h2>
    <div class="box">
      <p>This handbook is for developers and maintainers. It documents the actual engineering workflow of the standalone product repository.</p>
      <ul>
        <li>Repository: <code>workdock-platform</code> (current local path; legacy compose project name may still be retained for runtime continuity)</li>
        <li>Main stack: Django + Celery + PostgreSQL + Redis + MailHog</li>
        <li>Runtime mode: Docker Compose for local development and staging-style operation</li>
      </ul>
    </div>

    <h2 id="structure">2) Repository Structure</h2>
    <ul>
      <li><code>/backend/config/</code>: Django settings, WSGI, URL config</li>
      <li><code>/backend/workflows/</code>: application logic, views, models, tasks, templates, static assets</li>
      <li><code>/backend/workflows/templates/workflows/base_shell.html</code>: standard page shell for new staff-facing pages</li>
      <li><code>/backend/workflows/roles.py</code>: centralized role names, capability matrix, and template permission helpers</li>
      <li>Rule: all interactive app pages should extend <code>base_shell.html</code>; do not rebuild topbar/frame logic in page-local templates.</li>
      <li><code>/backend/media/templates/</code>: PDF HTML templates and letterhead source files</li>
      <li><code>/backend/media/pdfs/</code>: generated PDF outputs on host volume</li>
      <li><code>/backend/locale/</code>: translation catalogs</li>
      <li><code>/docker-compose.yml</code>: local runtime orchestration</li>
      <li><code>/Makefile</code>: repeatable translation commands</li>
      <li><code>/.github/workflows/i18n.yml</code>: translation compile validation in CI</li>
    </ul>

    <h2 id="local">3) Local Development Workflow</h2>
    <h3>Start</h3>
    <pre><code>cd /path/to/workdock-platform
docker compose up -d --build</code></pre>
    <h3>Main URLs</h3>
    <ul>
      <li>App: <code>http://127.0.0.1:8088/</code></li>
      <li>MailHog: <code>http://127.0.0.1:8025/</code></li>
      <li>Health check: <code>http://127.0.0.1:8088/healthz/</code></li>
    </ul>
    <h3>Bootstrap users</h3>
    <ul>
      <li>Admin: <code>admin_test</code> / <code>admin12345</code></li>
      <li>User: <code>user_test</code> / <code>user12345</code></li>
    </ul>

    <h2 id="docker">4) Docker Operations</h2>
    <pre><code>docker compose up -d --build
docker compose restart web
docker compose restart worker
docker compose logs --no-color --tail=120 web
docker compose logs --no-color --tail=120 worker
docker compose down
docker compose down -v</code></pre>
    <div class="note">
      The source code is bind-mounted into the container. Most template/view/static changes only require a web restart, not a full rebuild. Image changes such as system packages require <code>docker compose up -d --build</code>.
    </div>

    <h2 id="db">5) Database and Migrations</h2>
    <pre><code>docker compose exec -T web python manage.py makemigrations
docker compose exec -T web python manage.py migrate
docker compose exec -T web python manage.py check</code></pre>
    <ul>
      <li>Never edit or remove historical migrations casually.</li>
      <li>When adding fields to builder-driven models, preserve fallback behavior for existing rows.</li>
      <li>Fresh boot sequence runs migrations automatically in <code>entrypoint-web.sh</code>.</li>
    </ul>

    <h3>Role and Permission Model</h3>
    <ul>
      <li>Stable Django group names: <code>Platform Owner</code>, <code>Super Admin</code>, <code>Admin</code>, <code>IT Staff</code>, <code>Staff</code>.</li>
      <li>Groups are created automatically through a <code>post_migrate</code> hook in <code>workflows.signals</code>.</li>
      <li>Capability checks are centralized in <code>workflows.roles.CAPABILITIES</code>.</li>
      <li>Use <code>_require_capability(...)</code> in views instead of flat <code>is_staff</code> checks.</li>
      <li>Templates receive permission flags from <code>workflows.context_processors.role_context</code>.</li>
      <li><code>Platform Owner</code> is the product-level role. Company roles remain <code>Super Admin</code>, <code>Admin</code>, <code>IT Staff</code>, and <code>Staff</code>.</li>
      <li>User management lives at <code>/admin-tools/users/</code> and is the preferred path for normal role assignment, account activation, invitation mail dispatch, password-reset mail dispatch, and controlled user deletion.</li>
      <li>Backward-compatibility rule: authenticated legacy users with <code>is_staff=True</code> but no explicit role group currently fall back to the <code>Admin</code> capability set.</li>
      <li><code>superuser</code> accounts resolve to <code>Super Admin</code>.</li>
      <li>When adding a new operational page or action, define the capability in <code>roles.py</code>, gate the view, and hide the UI affordance when the capability is absent.</li>
    </ul>

    <h2 id="translations">6) Translation Workflow</h2>
    <h3>Standard Django i18n path</h3>
    <pre><code>make i18n-update-en
make i18n-compile</code></pre>
    <p>Equivalent raw commands:</p>
    <pre><code>docker compose exec -T web django-admin makemessages -l en
docker compose exec -T web django-admin compilemessages</code></pre>
    <ul>
      <li><code>gettext</code> is installed in the Docker image.</li>
      <li>Do not use custom ad hoc <code>.mo</code> compilation anymore.</li>
      <li>Phase 1 bilingual support covers fixed UI, including the main admin-tool labels/buttons.</li>
      <li>Phase 2 covers builder-driven labels and checklist items with explicit German and English fields.</li>
      <li>Email phase covers <code>NotificationTemplate</code>, <code>NotificationRule</code>, and the welcome-email settings UI with explicit DE/EN subject/body fields.</li>
      <li>Request records now persist <code>preferred_language</code> so workflow emails can render in the submitter's active UI language with German fallback.</li>
      <li><code>preferred_language</code> is normalized in model <code>save()</code> and also has a DB default of <code>de</code>, so alternate creation paths cannot insert null values.</li>
    </ul>

    <h2 id="pdf">7) PDF Pipeline</h2>
    <ul>
      <li>PDF generation is HTML-to-PDF using <code>xhtml2pdf</code>.</li>
      <li>Letterhead overlay defaults to <code>templates.pdf</code>, but can now be replaced from Admin Apps → <code>Branding</code>.</li>
      <li>Main logic lives in <code>backend/workflows/tasks.py</code>.</li>
      <li>Fixed PDF labels/headings are rendered from task-level DE/EN text dictionaries, not hard-coded directly in request processing logic.</li>
      <li>PDF language follows the normalized request <code>preferred_language</code>, with German fallback.</li>
      <li>Key templates:
        <ul>
          <li><code>onboarding_template.html</code></li>
          <li><code>offboarding_template.html</code></li>
          <li><code>onboarding_intro_template.html</code></li>
          <li><code>onboarding_intro_session_pdf.html</code></li>
        </ul>
      </li>
    </ul>
    <div class="note">
      xhtml2pdf is sensitive to layout complexity. Keep print templates conservative and verify every structural change with a real generated PDF.
    </div>

    <h2 id="email">8) Email Pipeline</h2>
    <ul>
      <li>Notification defaults are defined in <code>DEFAULT_NOTIFICATION_TEMPLATES</code> in <code>tasks.py</code>.</li>
      <li>Admin-configured overrides live in <code>NotificationTemplate</code> and <code>NotificationRule</code>.</li>
      <li>Both models now support explicit DE/EN content fields. Prefer filling both languages in the frontend integrations UI instead of embedding mixed-language copy into a single template.</li>
      <li>Welcome-email configuration under Admin Apps has the same DE/EN split and follows the same runtime language selection.</li>
      <li>The request objects passed into email tasks always carry a normalized <code>preferred_language</code> value.</li>
      <li>Mail sending uses Celery tasks and supports test mode redirection.</li>
      <li>MailHog is the local verification path when test mode is active.</li>
    </ul>

    <h2 id="nextcloud">9) Nextcloud Integration</h2>
    <ul>
      <li>Configured from Admin Apps → Integrations.</li>
      <li>Upload logic lives in <code>backend/workflows/services.py</code>.</li>
      <li>Feature can be globally toggled without changing environment variables.</li>
      <li>Failures should degrade gracefully and not block request persistence.</li>
      <li>Remote backup currently reuses the configured Nextcloud connection, but writes into a separate backup directory configured under <code>Integrationen → Backup-Ziel</code>.</li>
      <li>Do not point remote backup at the same Nextcloud directory used for normal onboarding/offboarding document uploads.</li>
    </ul>

    <h2 id="branding">10) Branding</h2>
    <ul>
      <li>Portal-level branding is stored in the singleton model <code>PortalBranding</code>.</li>
      <li>Configured from Admin Apps → <code>Branding</code>.</li>
      <li>Current scope: portal title, company name, company domain, support email, sender display name, login subtitle, footer/legal text, logo, favicon, PDF letterhead, and primary/secondary colors.</li>
      <li>Shared header/logo rendering now uses the branding context processor instead of hardcoded customer-specific asset references.</li>
      <li>The company domain now drives onboarding/offboarding email autofill and domain validation, so new customer deployments no longer require <code>@tub.co</code> code changes.</li>
      <li>Outgoing system mail sender names are now branded through the same layer.</li>
      <li>User invitation emails and welcome-template fallbacks also use the configured branding defaults.</li>
    </ul>

    <h2 id="app-registry">10b) App Registry</h2>
    <ul>
      <li>Registry definitions live in <code>workflows/app_registry.py</code>.</li>
      <li>DB overrides live in <code>PortalAppConfig</code>.</li>
      <li>The landing page now renders from registry data instead of hardcoded cards.</li>
      <li>Security remains code-based: app visibility/order is configurable, but access still depends on role capabilities in <code>roles.py</code>.</li>
      <li>Management UI: <code>/admin-tools/apps/</code> for <code>Platform Owner</code>.</li>
    </ul>

    <h2 id="trial">10c) Trial Lifecycle</h2>
    <ul>
      <li>Deployment-level trial settings are stored in the singleton model <code>PortalTrialConfig</code>.</li>
      <li>Management UI: <code>/admin-tools/trial/</code> for <code>Platform Owner</code>.</li>
      <li>Current scope: trial enable/disable, start/end timestamps, bilingual shell banner text, production-integration restriction, and cleanup readiness.</li>
      <li>Enforcement lives in <code>workflows.middleware.TrialModeMiddleware</code>, so expiry is handled centrally instead of per-view.</li>
      <li>While trial restriction is active, service-level integration checks force Nextcloud off and E-Mail into test mode.</li>
      <li>Expired-trial cleanup is intentionally CLI-only:
        <pre><code>docker compose exec -T web python manage.py cleanup_expired_trial_workspace --yes-delete</code></pre>
      </li>
    </ul>

    <h2 id="builders">11) Builder Architecture</h2>
    <h3>Form Builder</h3>
    <ul>
      <li>Model: <code>FormFieldConfig</code> + <code>FormOption</code></li>
      <li>Controls field order, visibility, required flags, option sets, and bilingual label/help-text overrides</li>
    </ul>
    <h3>Intro Builder</h3>
    <ul>
      <li>Model: <code>IntroChecklistItem</code></li>
      <li>Controls additional checklist items used by intro PDF and live intro checklist</li>
      <li>Now supports bilingual DE/EN labels</li>
    </ul>
    <div class="note">
      Dynamic content should use explicit DE/EN fields with German fallback, not machine translation at runtime.
    </div>
    <h3>Audit Trail</h3>
    <ul>
      <li>Model: <code>AdminAuditLog</code></li>
      <li>Purpose: record staff-side mutations that affect operations or configuration.</li>
      <li>Current hooks include builder edits, PDF generation, welcome-email actions, integration changes, mode toggles, tests, and request deletions.</li>
      <li>Staff UI page: <code>/admin-tools/audit-log/</code></li>
      <li>The current UI supports filtering by action, user, and date range. Keep filters server-side to avoid loading unbounded audit rows into the browser.</li>
      <li>Backup UI page: <code>/admin-tools/backups/</code> for create, verify, and delete actions. Keep real restore CLI-only.</li>
    </ul>

    <h2 id="testing">11) Testing and Validation</h2>
    <pre><code>docker compose exec -T web python manage.py check
docker compose exec -T web python manage.py test
docker compose exec -T web python manage.py run_staging_e2e_check</code></pre>
    <ul>
      <li>Use <code>check</code> after model/view/template changes.</li>
      <li>Use targeted shell checks for render validation when changing templates or routes.</li>
      <li>New pages should extend <code>base_shell.html</code> and keep header/frame logic out of page-local templates.</li>
      <li>Use real PDF generation tests when changing PDF templates or intro/offboarding document logic.</li>
      <li>Use the dedicated Release Checklist page as the final go/no-go runbook before shipping changes.</li>
      <li>The automated bilingual smoke tests now cover DE/EN request language capture and English email-template rendering.</li>
      <li>Onboarding and offboarding request objects now expose explicit processing state and last-error fields. Async tasks are responsible for transitioning <code>submitted → processing → completed/failed</code>.</li>
      <li>The Requests Dashboard includes a retry action for failed requests. Retries reset the error text, set the request back to <code>submitted</code>, and enqueue the appropriate Celery task again.</li>
    </ul>

    <h2 id="backup">12) Backup and Restore</h2>
    <pre><code>make backup-create
make backup-verify BACKUP_DIR=backups/backup_YYYYmmdd_HHMMSS</code></pre>
    <ul>
      <li>Backups are stored under <code>backend/backups/</code> and ignored by git.</li>
      <li>Each backup contains a PostgreSQL custom-format dump, a compressed <code>media/</code> archive, metadata, and SHA256 checksums.</li>
      <li>Backup metadata records both local availability and remote backup status.</li>
      <li>Remote backup target configuration is managed in <code>Integrationen → Backup-Ziel</code>.</li>
      <li>Current remote target support: <code>nextcloud</code> implemented, <code>s3</code> and <code>nfs</code> config-ready but not yet implemented.</li>
      <li>Verification is non-destructive: it restores into a temporary verification database and extracts media into a temporary directory.</li>
      <li>For scheduled operational hygiene, verify the newest bundle directly from the app container:
        <pre><code>docker compose exec -T web python manage.py verify_latest_backup --create-if-missing</code></pre>
      </li>
      <li>The Backup &amp; Recovery page now shows whether the latest verification is current, stale, missing, or still unverified.</li>
      <li>Real restore is explicit and destructive by design:
        <pre><code>./scripts/backup_restore.sh --yes-restore backend/backups/backup_YYYYmmdd_HHMMSS</code></pre>
      </li>
      <li>Do not run the restore script casually against a live working dataset.</li>
      <li>The staff UI uses the shared action-progress overlay for backup creation and verification so long-running actions present one standard app behavior.</li>
    </ul>

    <h2 id="cicd">13) CI/CD</h2>
    <ul>
      <li>Repository model: one private GitHub repository, not separate dev/prod repositories.</li>
      <li>Branch model: <code>develop</code> for the test deployment, <code>main</code> reserved for production.</li>
      <li>Current test workflow file: <code>.github/workflows/deploy-test.yml</code>.</li>
      <li>Current production workflow file: <code>.github/workflows/deploy-prod.yml</code>.</li>
      <li>GitHub Actions uploads the working tree to the server over SSH. The server does not clone from GitHub.</li>
      <li>This is intentional for a private repository: it removes the need to store GitHub deploy keys on the target server.</li>
    </ul>
    <h3>GitHub Environments</h3>
    <ul>
      <li>Create GitHub environments named <code>development</code> and <code>production</code>.</li>
      <li><strong>Development</strong> secrets:
        <ul>
          <li><code>TEST_DEPLOY_HOST</code></li>
          <li><code>TEST_DEPLOY_USER</code></li>
          <li><code>TEST_DEPLOY_PORT</code></li>
          <li><code>TEST_DEPLOY_PATH</code></li>
          <li><code>TEST_DEPLOY_SSH_KEY</code></li>
        </ul>
      </li>
      <li><strong>Production</strong> secrets:
        <ul>
          <li><code>PROD_DEPLOY_HOST</code></li>
          <li><code>PROD_DEPLOY_USER</code></li>
          <li><code>PROD_DEPLOY_PORT</code></li>
          <li><code>PROD_DEPLOY_PATH</code></li>
          <li><code>PROD_DEPLOY_SSH_KEY</code></li>
        </ul>
      </li>
    </ul>
    <h3>Current test deployment values</h3>
    <ul>
      <li>Host: <code>192.168.2.55</code></li>
      <li>User: <code>root</code></li>
      <li>Path: <code>/opt/workdock</code></li>
      <li>URL: <code>http://192.168.2.55:8088</code></li>
    </ul>
    <div class="note">
      The current LAN test deployment intentionally uses <code>DJANGO_DEBUG=1</code> in <code>.env.test</code> because the security checks correctly reject insecure cookie settings when <code>DEBUG=0</code> and the deployment is still plain HTTP. This is acceptable for the internal test box only. Production must run with HTTPS and <code>DEBUG=0</code>.
    </div>

    <h2 id="deploy">14) Deployment</h2>
    <h3>Test server stack</h3>
    <ul>
      <li>Stack file: <code>docker-compose.prod.yml</code></li>
      <li>Entrypoints:
        <ul>
          <li><code>backend/entrypoint-web-prod.sh</code></li>
          <li><code>backend/entrypoint-worker-prod.sh</code></li>
        </ul>
      </li>
      <li>Reverse proxy/static/media: <code>deploy/Caddyfile</code></li>
      <li>Main deploy script: <code>scripts/deploy_stack.sh</code></li>
    </ul>
    <h3>What the deploy script does</h3>
    <ol>
      <li>Validate env file presence</li>
      <li>Build <code>web</code>, <code>worker</code>, and <code>caddy</code></li>
      <li>Start <code>db</code> and <code>redis</code></li>
      <li>Initialize writable volume ownership for media/static/backups</li>
      <li>Run migrations</li>
      <li>Run <code>bootstrap_initial_users</code></li>
      <li>Run <code>collectstatic</code></li>
      <li>Optionally run <code>manage.py check</code></li>
      <li>Start <code>web</code>, <code>worker</code>, and <code>caddy</code></li>
      <li>Wait until <code>/healthz/</code> becomes healthy</li>
    </ol>
    <h3>Manual deploy</h3>
    <pre><code>cd /opt/workdock
RUN_DJANGO_CHECK=0 DEPLOY_HEALTH_URL="http://127.0.0.1:8088/healthz/" ./scripts/deploy_stack.sh .env.test docker-compose.prod.yml</code></pre>
    <h3>Validation after deploy</h3>
    <pre><code>curl -I http://192.168.2.55:8088/healthz/
ssh root@192.168.2.55 "cd /opt/workdock && docker compose --env-file .env.test -f docker-compose.prod.yml ps"</code></pre>
    <h3>Proxmox / LXC requirement</h3>
    <p>The current server is an Ubuntu CT on Proxmox running Docker inside the container. The CT required Proxmox-side configuration before Docker containers could start correctly.</p>
    <pre><code>features: nesting=1,keyctl=1
lxc.apparmor.profile: unconfined
lxc.mount.entry: /dev/null sys/module/apparmor/parameters/enabled none bind 0 0</code></pre>
    <p>Those lines belong in <code>/etc/pve/lxc/&lt;CTID&gt;.conf</code> on the Proxmox host, followed by <code>pct restart &lt;CTID&gt;</code>.</p>
    <h3>Production expectations</h3>
    <ul>
      <li>Use a real hostname</li>
      <li>Use HTTPS</li>
      <li>Use <code>DJANGO_DEBUG=0</code></li>
      <li>Use secure cookies and SSL redirect</li>
      <li>Run deployment with <code>RUN_DJANGO_CHECK=1</code></li>
    </ul>
    <h3>Release checklist</h3>
    <ol>
      <li>Run <code>manage.py check</code></li>
      <li>Run tests or targeted verification</li>
      <li>Run translation compile step</li>
      <li>Rebuild containers if Python dependencies changed, then verify <code>python -c "import requests"</code> does not emit a compatibility warning</li>
      <li>Generate at least one onboarding/offboarding PDF if PDF templates changed</li>
      <li>Verify MailHog or SMTP path if email behavior changed</li>
      <li>Verify Nextcloud upload if integration behavior changed</li>
      <li>Update Project Wiki and Developer Handbook if architecture or operational workflow changed</li>
      <li>Take a snapshot commit before major next-phase work</li>
    </ol>

    <h2 id="troubleshooting">15) Troubleshooting</h2>
    <ul>
      <li><strong>Page looks stale:</strong> restart <code>web</code> and hard-refresh browser</li>
      <li><strong>Second request hangs:</strong> inspect web logs and verify health endpoint</li>
      <li><strong>PDF looks unchanged:</strong> regenerate the PDF; browser may cache old file names unless the path changes</li>
      <li><strong>Language switch not visible:</strong> verify translation catalog compiled and restart web</li>
      <li><strong>Mail not visible:</strong> check MailHog on port <code>8025</code> and test/production mode toggle</li>
      <li><strong>Nextcloud sync unclear:</strong> verify config in Integrations page and inspect service logs</li>
      <li><strong>Requests dependency warning appears:</strong> verify <code>chardet==5.2.0</code> is installed in the rebuilt image and restart <code>web</code>/<code>worker</code></li>
    </ul>

    <h2 id="security">16) Security and Maintenance Notes</h2>
    <ul>
      <li>Containers run as non-root <code>app</code> user.</li>
      <li>Keep secrets in <code>.env</code>, not in tracked files.</li>
      <li>Avoid destructive git operations in a dirty repo.</li>
      <li>Prefer standard framework workflows over custom one-off maintenance scripts.</li>
      <li>When adding new features, document them in both the Project Wiki and this handbook if they change engineering workflow.</li>
    </ul>
    </div>
  
{% endblock %}