docs: expand engineering handbook and contributor guide

CONTRIBUTING.md

@@ -0,0 +1,119 @@
+# Contributing Guide
+
+## Purpose
+This repository is the standalone productized Workdock platform.
+
+Use this file as the quick-start guide for future coders. It complements:
+- `DEPLOYMENT.md`
+- `backend/workflows/templates/workflows/project_wiki.html`
+- `backend/workflows/templates/workflows/developer_handbook.html`
+
+If you change architecture, deployment flow, or operational behavior, update the relevant documentation in the same branch.
+
+## Branch Policy
+- `develop`: active integration branch
+- `main`: stable branch for production promotion
+- short-lived feature branches: branch from `develop`, merge back into `develop`
+
+Do not use `main` as the default development branch.
+
+## Current Delivery Model
+- GitHub Actions is used for CI
+- the current test server is local/LAN-only
+- deployment to the test server is manual from a LAN-connected machine
+
+Standard test deployment command:
+
+```bash
+./scripts/deploy_test_from_mac.sh
+```
+
+Why:
+- GitHub-hosted runners cannot reliably reach the private LAN target at `192.168.2.55`
+
+## Architecture Summary
+Main stack:
+- Django
+- Celery
+- PostgreSQL
+- Redis
+- Caddy
+
+Important repository areas:
+- `backend/config/`: Django configuration
+- `backend/workflows/`: application code
+- `backend/workflows/templates/workflows/`: templates and in-app documentation
+- `backend/workflows/static/workflows/`: CSS and JS
+- `backend/media/templates/`: PDF HTML templates
+
+Current backend modularization:
+- `views.py`: thin route wrapper layer
+- split view modules:
+  - `request_views.py`
+  - `account_views.py`
+  - `admin_config_views.py`
+  - `form_builder_views.py`
+  - `observability_views.py`
+  - `integration_admin_views.py`
+  - `welcome_email_views.py`
+  - `intro_builder_views.py`
+- `tasks.py`: async task entrypoints/orchestration
+- `pdf_rendering.py` and `pdf_sections.py`: PDF rendering
+- `email_workflows.py` and `notification_dispatch.py`: mail/notification orchestration
+- `models.py`: stable import facade over split model modules
+
+## Engineering Rules
+1. Preserve runtime behavior while refactoring.
+2. Prefer shared UI/system primitives over page-local one-off patterns.
+3. Do not silently overwrite environment-specific configuration data.
+4. Treat database-backed admin configuration as runtime state, not seed data.
+5. Update docs when workflow or architecture changes.
+
+## Code vs Data
+Not all visible behavior comes from code.
+
+Code-driven:
+- page layout
+- permissions
+- routing
+- task behavior
+- deploy scripts
+
+Database-driven:
+- app registry order and visibility
+- branding and company config
+- builder settings
+- intro checklist items
+- notification templates and rules
+
+If local UI and server UI differ, check the database-backed config before assuming a code drift.
+
+## Testing Rules
+Minimum validation after non-trivial changes:
+
+```bash
+docker compose exec -T web python manage.py check
+docker compose exec -T web python manage.py test
+```
+
+Also verify specifically when changing:
+- PDFs
+- deployment/config
+- templates/UI
+
+## Documentation Rules
+Keep these current:
+- `DEPLOYMENT.md`
+- `PRODUCTIZATION_ROADMAP.md`
+- `backend/workflows/templates/workflows/project_wiki.html`
+- `backend/workflows/templates/workflows/developer_handbook.html`
+
+## Recommended Change Flow
+1. Start from `develop`
+2. Implement the change
+3. Validate locally
+4. Update docs if needed
+5. Push to GitHub
+6. Let CI run
+7. Deploy from the Mac if runtime verification is needed
+8. Promote `develop` to `main` when stable

backend/workflows/templates/workflows/developer_handbook.html

@@ -23,9 +23,11 @@
     <div class="toc">
       <a href="#overview">Overview</a>
       <a href="#structure">Structure</a>
+      <a href="#workflow">Workflow</a>
       <a href="#local">Local Dev</a>
       <a href="#docker">Docker</a>
       <a href="#db">Database</a>
+      <a href="#guidelines">Guidelines</a>
       <a href="#translations">Translations</a>
       <a href="#pdf">PDF Pipeline</a>
       <a href="#email">Email Pipeline</a>
@@ -47,6 +49,7 @@
         <li>Repository: <code>workdock-platform</code> (current local path; legacy compose project name may still be retained for runtime continuity)</li>
         <li>Main stack: Django + Celery + PostgreSQL + Redis + MailHog</li>
         <li>Runtime mode: Docker Compose for local development and staging-style operation</li>
+        <li>Repository-level quick-start guide for future coders: <code>CONTRIBUTING.md</code></li>
       </ul>
     </div>
 
@@ -54,6 +57,9 @@
     <ul>
       <li><code>/backend/config/</code>: Django settings, WSGI, URL config</li>
       <li><code>/backend/workflows/</code>: application logic, views, models, tasks, templates, static assets</li>
+      <li><code>/backend/workflows/views.py</code>: thin route wrapper layer; most view logic now lives in split modules by domain</li>
+      <li><code>/backend/workflows/models.py</code>: stable model import surface over split model modules</li>
+      <li><code>/backend/workflows/tasks.py</code>: async task entrypoints; PDF and notification logic were moved into dedicated modules</li>
       <li><code>/backend/workflows/templates/workflows/base_shell.html</code>: standard page shell for new staff-facing pages</li>
       <li><code>/backend/workflows/roles.py</code>: centralized role names, capability matrix, and template permission helpers</li>
       <li>Rule: all interactive app pages should extend <code>base_shell.html</code>; do not rebuild topbar/frame logic in page-local templates.</li>
@@ -65,7 +71,29 @@
       <li><code>/.github/workflows/i18n.yml</code>: translation compile validation in CI</li>
     </ul>
 
-    <h2 id="local">3) Local Development Workflow</h2>
+    <h2 id="workflow">3) Working Model</h2>
+    <div class="box">
+      <h3>Branch strategy</h3>
+      <ul>
+        <li><code>develop</code> is the active integration branch.</li>
+        <li><code>main</code> is the stable branch intended for production promotion.</li>
+        <li>Feature work should start from <code>develop</code> and merge back into <code>develop</code>.</li>
+      </ul>
+    </div>
+    <div class="box">
+      <h3>Normal change flow</h3>
+      <ol>
+        <li>start from <code>develop</code></li>
+        <li>implement the change</li>
+        <li>run validation</li>
+        <li>update docs if architecture or workflow changed</li>
+        <li>push to GitHub and let CI run</li>
+        <li>deploy from the Mac if test-server verification is needed</li>
+        <li>promote <code>develop</code> into <code>main</code> when stable</li>
+      </ol>
+    </div>
+
+    <h2 id="local">4) Local Development Workflow</h2>
     <h3>Start</h3>
     <pre><code>cd /path/to/workdock-platform
 docker compose up -d --build</code></pre>
@@ -81,7 +109,7 @@ docker compose up -d --build</code></pre>
       <li>User: <code>user_test</code> / <code>user12345</code></li>
     </ul>
 
-    <h2 id="docker">4) Docker Operations</h2>
+    <h2 id="docker">5) Docker Operations</h2>
     <pre><code>docker compose up -d --build
 docker compose restart web
 docker compose restart worker
@@ -93,7 +121,7 @@ docker compose down -v</code></pre>
       The source code is bind-mounted into the container. Most template/view/static changes only require a web restart, not a full rebuild. Image changes such as system packages require <code>docker compose up -d --build</code>.
     </div>
 
-    <h2 id="db">5) Database and Migrations</h2>
+    <h2 id="db">6) Database and Migrations</h2>
     <pre><code>docker compose exec -T web python manage.py makemigrations
 docker compose exec -T web python manage.py migrate
 docker compose exec -T web python manage.py check</code></pre>
@@ -101,6 +129,7 @@ docker compose exec -T web python manage.py check</code></pre>
       <li>Never edit or remove historical migrations casually.</li>
       <li>When adding fields to builder-driven models, preserve fallback behavior for existing rows.</li>
       <li>Fresh boot sequence runs migrations automatically in <code>entrypoint-web.sh</code>.</li>
+      <li>Do not assume database-backed admin config matches between local and deployed environments.</li>
     </ul>
 
     <h3>Role and Permission Model</h3>
@@ -117,7 +146,27 @@ docker compose exec -T web python manage.py check</code></pre>
       <li>When adding a new operational page or action, define the capability in <code>roles.py</code>, gate the view, and hide the UI affordance when the capability is absent.</li>
     </ul>
 
-    <h2 id="translations">6) Translation Workflow</h2>
+    <h2 id="guidelines">7) Engineering Guidelines</h2>
+    <div class="box">
+      <h3>Core rules</h3>
+      <ol>
+        <li>Preserve behavior while refactoring.</li>
+        <li>Prefer shared components over page-local special cases.</li>
+        <li>Do not overwrite environment-specific runtime config as a side effect of code deploys.</li>
+        <li>Keep code-driven behavior and data-driven behavior mentally separate.</li>
+        <li>Update documentation in the same branch when operational workflow changes.</li>
+      </ol>
+    </div>
+    <div class="box">
+      <h3>Code vs data</h3>
+      <ul>
+        <li><strong>Code-driven:</strong> routing, permissions, deployment scripts, rendering, tasks, layout.</li>
+        <li><strong>Data-driven:</strong> app registry order, branding, company config, builder config, intro checklist items, notification templates.</li>
+        <li>If local UI and server UI differ, inspect runtime configuration before assuming code drift.</li>
+      </ul>
+    </div>
+
+    <h2 id="translations">8) Translation Workflow</h2>
     <h3>Standard Django i18n path</h3>
     <pre><code>make i18n-update-en
 make i18n-compile</code></pre>
@@ -134,11 +183,12 @@ docker compose exec -T web django-admin compilemessages</code></pre>
       <li><code>preferred_language</code> is normalized in model <code>save()</code> and also has a DB default of <code>de</code>, so alternate creation paths cannot insert null values.</li>
     </ul>
 
-    <h2 id="pdf">7) PDF Pipeline</h2>
+    <h2 id="pdf">9) PDF Pipeline</h2>
     <ul>
       <li>PDF generation is HTML-to-PDF using <code>xhtml2pdf</code>.</li>
       <li>Letterhead overlay defaults to <code>templates.pdf</code>, but can now be replaced from Admin Apps → <code>Branding</code>.</li>
-      <li>Main logic lives in <code>backend/workflows/tasks.py</code>.</li>
+      <li>Task entrypoints live in <code>backend/workflows/tasks.py</code>.</li>
+      <li>Rendering and section-building now live in <code>pdf_rendering.py</code> and <code>pdf_sections.py</code>.</li>
       <li>Fixed PDF labels/headings are rendered from task-level DE/EN text dictionaries, not hard-coded directly in request processing logic.</li>
       <li>PDF language follows the normalized request <code>preferred_language</code>, with German fallback.</li>
       <li>Key templates:
@@ -154,18 +204,19 @@ docker compose exec -T web django-admin compilemessages</code></pre>
       xhtml2pdf is sensitive to layout complexity. Keep print templates conservative and verify every structural change with a real generated PDF.
     </div>
 
-    <h2 id="email">8) Email Pipeline</h2>
+    <h2 id="email">10) Email Pipeline</h2>
     <ul>
-      <li>Notification defaults are defined in <code>DEFAULT_NOTIFICATION_TEMPLATES</code> in <code>tasks.py</code>.</li>
+      <li>Notification defaults are defined in the workflow/email orchestration layer.</li>
       <li>Admin-configured overrides live in <code>NotificationTemplate</code> and <code>NotificationRule</code>.</li>
       <li>Both models now support explicit DE/EN content fields. Prefer filling both languages in the frontend integrations UI instead of embedding mixed-language copy into a single template.</li>
       <li>Welcome-email configuration under Admin Apps has the same DE/EN split and follows the same runtime language selection.</li>
       <li>The request objects passed into email tasks always carry a normalized <code>preferred_language</code> value.</li>
       <li>Mail sending uses Celery tasks and supports test mode redirection.</li>
+      <li>Related helper modules include <code>email_workflows.py</code> and <code>notification_dispatch.py</code>.</li>
       <li>MailHog is the local verification path when test mode is active.</li>
     </ul>
 
-    <h2 id="nextcloud">9) Nextcloud Integration</h2>
+    <h2 id="nextcloud">11) Nextcloud Integration</h2>
     <ul>
       <li>Configured from Admin Apps → Integrations.</li>
       <li>Upload logic lives in <code>backend/workflows/services.py</code>.</li>
@@ -175,7 +226,7 @@ docker compose exec -T web django-admin compilemessages</code></pre>
       <li>Do not point remote backup at the same Nextcloud directory used for normal onboarding/offboarding document uploads.</li>
     </ul>
 
-    <h2 id="branding">10) Branding</h2>
+    <h2 id="branding">12) Branding</h2>
     <ul>
       <li>Portal-level branding is stored in the singleton model <code>PortalBranding</code>.</li>
       <li>Configured from Admin Apps → <code>Branding</code>.</li>
@@ -186,7 +237,7 @@ docker compose exec -T web django-admin compilemessages</code></pre>
       <li>User invitation emails and welcome-template fallbacks also use the configured branding defaults.</li>
     </ul>
 
-    <h2 id="app-registry">10b) App Registry</h2>
+    <h2 id="app-registry">12b) App Registry</h2>
     <ul>
       <li>Registry definitions live in <code>workflows/app_registry.py</code>.</li>
       <li>DB overrides live in <code>PortalAppConfig</code>.</li>
@@ -195,7 +246,7 @@ docker compose exec -T web django-admin compilemessages</code></pre>
       <li>Management UI: <code>/admin-tools/apps/</code> for <code>Platform Owner</code>.</li>
     </ul>
 
-    <h2 id="trial">10c) Trial Lifecycle</h2>
+    <h2 id="trial">12c) Trial Lifecycle</h2>
     <ul>
       <li>Deployment-level trial settings are stored in the singleton model <code>PortalTrialConfig</code>.</li>
       <li>Management UI: <code>/admin-tools/trial/</code> for <code>Platform Owner</code>.</li>
@@ -207,11 +258,14 @@ docker compose exec -T web django-admin compilemessages</code></pre>
       </li>
     </ul>
 
-    <h2 id="builders">11) Builder Architecture</h2>
+    <h2 id="builders">13) Builder Architecture</h2>
     <h3>Form Builder</h3>
     <ul>
       <li>Model: <code>FormFieldConfig</code> + <code>FormOption</code></li>
       <li>Controls field order, visibility, required flags, option sets, and bilingual label/help-text overrides</li>
+      <li>Static definitions live in <code>form_builder_config.py</code>.</li>
+      <li>Runtime behavior lives in <code>form_builder_runtime.py</code>.</li>
+      <li><code>form_builder.py</code> is the stable facade import path.</li>
     </ul>
     <h3>Intro Builder</h3>
     <ul>
@@ -232,7 +286,7 @@ docker compose exec -T web django-admin compilemessages</code></pre>
       <li>Backup UI page: <code>/admin-tools/backups/</code> for create, verify, and delete actions. Keep real restore CLI-only.</li>
     </ul>
 
-    <h2 id="testing">11) Testing and Validation</h2>
+    <h2 id="testing">14) Testing and Validation</h2>
     <pre><code>docker compose exec -T web python manage.py check
 docker compose exec -T web python manage.py test
 docker compose exec -T web python manage.py run_staging_e2e_check</code></pre>
@@ -247,7 +301,7 @@ docker compose exec -T web python manage.py run_staging_e2e_check</code></pre>
       <li>The Requests Dashboard includes a retry action for failed requests. Retries reset the error text, set the request back to <code>submitted</code>, and enqueue the appropriate Celery task again.</li>
     </ul>
 
-    <h2 id="backup">12) Backup and Restore</h2>
+    <h2 id="backup">15) Backup and Restore</h2>
     <pre><code>make backup-create
 make backup-verify BACKUP_DIR=backups/backup_YYYYmmdd_HHMMSS</code></pre>
     <ul>
@@ -268,7 +322,7 @@ make backup-verify BACKUP_DIR=backups/backup_YYYYmmdd_HHMMSS</code></pre>
       <li>The staff UI uses the shared action-progress overlay for backup creation and verification so long-running actions present one standard app behavior.</li>
     </ul>
 
-    <h2 id="hosts">13) Host and Domain Configuration</h2>
+    <h2 id="hosts">16) Host and Domain Configuration</h2>
     <ul>
       <li>Primary env variables:
         <ul>
@@ -286,7 +340,7 @@ make backup-verify BACKUP_DIR=backups/backup_YYYYmmdd_HHMMSS</code></pre>
       <li>An <code>Invalid HTTP_HOST header</code> failure happens before normal page routing, so a broken hostname cannot render a custom error page on that same broken host. Use a working host or IP to access the runbook and fix the env file.</li>
     </ul>
 
-    <h2 id="cicd">14) CI/CD</h2>
+    <h2 id="cicd">17) CI/CD</h2>
     <div class="box">
       <h3>Current operating model</h3>
       <ul>
@@ -381,7 +435,7 @@ make backup-verify BACKUP_DIR=backups/backup_YYYYmmdd_HHMMSS</code></pre>
       The current LAN test deployment intentionally uses <code>DJANGO_DEBUG=1</code> in <code>.env.test</code> because the security checks correctly reject insecure cookie settings when <code>DEBUG=0</code> and the deployment is still plain HTTP behind a local test topology. This is acceptable for the test box only. Production must run with HTTPS and <code>DEBUG=0</code>.
     </div>
 
-    <h2 id="deploy">15) Deployment</h2>
+    <h2 id="deploy">18) Deployment</h2>
     <h3>Test server stack</h3>
     <ul>
       <li>Stack file: <code>docker-compose.prod.yml</code></li>
@@ -445,7 +499,7 @@ lxc.mount.entry: /dev/null sys/module/apparmor/parameters/enabled none bind 0 0<
       <li>Take a snapshot commit before major next-phase work</li>
     </ol>
 
-    <h2 id="troubleshooting">16) Troubleshooting</h2>
+    <h2 id="troubleshooting">19) Troubleshooting</h2>
     <ul>
       <li><strong>Page looks stale:</strong> restart <code>web</code> and hard-refresh browser</li>
       <li><strong>Second request hangs:</strong> inspect web logs and verify health endpoint</li>
@@ -456,7 +510,7 @@ lxc.mount.entry: /dev/null sys/module/apparmor/parameters/enabled none bind 0 0<
       <li><strong>Requests dependency warning appears:</strong> verify <code>chardet==5.2.0</code> is installed in the rebuilt image and restart <code>web</code>/<code>worker</code></li>
     </ul>
 
-    <h2 id="security">17) Security and Maintenance Notes</h2>
+    <h2 id="security">20) Security and Maintenance Notes</h2>
     <ul>
       <li>Containers run as non-root <code>app</code> user.</li>
       <li>Keep secrets in <code>.env</code>, not in tracked files.</li>
@@ -464,6 +518,15 @@ lxc.mount.entry: /dev/null sys/module/apparmor/parameters/enabled none bind 0 0<
       <li>Prefer standard framework workflows over custom one-off maintenance scripts.</li>
       <li>When adding new features, document them in both the Project Wiki and this handbook if they change engineering workflow.</li>
     </ul>
+    <div class="box">
+      <h3>For the next coder</h3>
+      <ul>
+        <li>Read this page first, then the Project Wiki, then the release checklist.</li>
+        <li>Assume some visible behavior is runtime configuration, not only template or view code.</li>
+        <li>Keep stable import surfaces intact where possible when modularizing.</li>
+        <li>Prefer explicit, documented behavior over hidden automation.</li>
+      </ul>
+    </div>
     </div>
   
 {% endblock %}

backend/workflows/templates/workflows/handbook.html

@@ -36,9 +36,10 @@
       <section class="card">
         <div class="eyebrow">{% trans "Engineering" %}</div>
         <h2>{% trans "Developer Handbook" %}</h2>
-        <p>{% trans "Engineering documentation for architecture, local setup, Docker, migrations, translations, deployment, testing, and long-term maintenance." %}</p>
+        <p>{% trans "Engineering runbook for architecture, branch workflow, deployment, CI/CD, code guidelines, and long-term maintenance." %}</p>
         <ul>
           <li>{% trans "repository and service structure" %}</li>
+          <li>{% trans "branch workflow and coding guidelines" %}</li>
           <li>{% trans "Docker and migration workflow" %}</li>
           <li>{% trans "translation and builder architecture" %}</li>
           <li>{% trans "CI/CD, deployment, security, and maintenance notes" %}</li>