Merge pull request #1 from KleineEnte/main

German Translations

README.md

@@ -0,0 +1,188 @@
+
+# onboarding_system
+
+This project automates the synchronization of local folders (`attachments` and `onboarded_person`) with corresponding directories in a Nextcloud instance. It checks for new or modified files in local folders and uploads them to the designated Nextcloud directories, using a persistent session for efficient authentication. Additionally, it includes features for email notification and PDF generation from HTML templates and CSV files.
+
+## Features
+- Generate pdf from the dynamic CSV file.
+- Send email to to the recipient when it is necessary.
+- Automatically syncs files between local folders and Nextcloud.
+- Uploads only new or modified files to Nextcloud.
+- Processes CSV attachments from emails to generate PDFs.
+- Sends email notifications based on specific CSV content.
+- Configurable synchronization interval (default is 60 seconds).
+- Uses a persistent HTTP session for efficient communication with Nextcloud.
+- Tracks files locally to prevent re-uploading of unchanged files.
+
+## Requirements
+
+- Python 3.8+
+- Required Python libraries:
+  - `requests`
+  - `python-dotenv`
+  - `HTTPBasicAuth` (from `requests`)
+  - `pandas`
+  - `jinja2`
+  - `pdfkit`
+  - `PyPDF2`
+  - `imaplib`
+  - `email`
+  - `smtplib`
+
+## Setup
+
+### Step 1: Clone the Repository
+
+```bash
+git clone https://github.com/Bostame/onboarding_system.git
+cd onboarding_system
+```
+
+### Step 2: Install Dependencies
+
+Make sure you have Python 3.8+ installed. Use the following command to install dependencies:
+
+```bash
+pip install -r requirements.yml
+```
+
+### Step 3: Configure Environment Variables
+
+Create an `.env` file inside the `env/` directory with the following content:
+
+```env
+# Nextcloud credentials and URLs
+NEXTCLOUD_BASE_URL=https://your-nextcloud-url/remote.php/dav/files/your-user
+NEXTCLOUD_USERNAME=your-username
+NEXTCLOUD_PASSWORD=your-password
+NEXTCLOUD_DIRECTORY=Group-on-off-boarding
+
+# Sync interval in seconds (default: 60 seconds)
+SYNC_INTERVAL=60
+
+# Local folder paths
+TEMPLATES_DIR=templates
+ATTACHMENTS_DIR=attachments
+ONBOARDED_DIR=onboarded_person
+TEMP_PDF_DIR=temp_pdf
+EMAIL_TEXT_DIR=email_text
+
+# Email details (if applicable)
+IMAP_SERVER=imap.example.com
+SMTP_SERVER=smtp.example.com
+EMAIL_PORT=465
+EMAIL_ACCOUNT=your-email@example.com
+PASSWORD=your-email-password
+MINUTH_EMAIL=minuth@example.com
+DRITICH_EMAIL=dritich@example.com
+```
+
+### Step 4: Create Local Folders
+
+Ensure the following local directories exist:
+
+- `attachments/`
+- `onboarded_person/`
+- `templates/`
+- `temp_pdf/`
+- `email_text/`
+
+You can modify these folder paths via the `.env` file as needed.
+
+### Step 5: Run the Script
+
+To manually run the synchronization and CSV processing:
+
+```bash
+python sync_script.py
+python main.py
+```
+
+### Creating Systemd Service Units
+
+To continuously run the synchronization and processing in the background, create systemd service units.
+
+#### Step 1: Create the `onboarding-system.service` and `nextcloud-sync.service` Unit
+
+1. Create a service file:
+
+```bash
+sudo nano /etc/systemd/system/onboarding-system.service
+sudo nano /etc/systemd/system/nextcloud-sync.service
+```
+
+2. Add the following content to the file:
+
+```ini
+[Unit]
+Description=Nextcloud Folder Sync Service
+After=network.target
+
+[Service]
+User=root
+Environment="PATH=/root/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
+WorkingDirectory=/path/to/your/project
+ExecStart=/bin/bash -lc 'source /root/miniconda3/etc/profile.d/conda.sh && conda activate csv_pdf && /root/miniconda3/envs/csv_pdf/bin/python main.py'
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+```
+and
+
+```ini
+[Unit]
+Description=Nextcloud Folder Sync Service
+After=network.target
+
+[Service]
+User=root
+Environment="PATH=/root/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
+WorkingDirectory=/path/to/your/project
+ExecStart=/bin/bash -lc 'source /root/miniconda3/etc/profile.d/conda.sh && conda activate csv_pdf && /root/miniconda3/envs/csv_pdf/bin/python sync_script.py'
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+```
+
+3. Save the file and close it.
+
+#### Step 2: Reload Systemd Daemon
+
+```bash
+sudo systemctl daemon-reload
+```
+
+#### Step 3: Enable the Service
+
+```bash
+sudo systemctl enable onboarding-system.service
+sudo systemctl enable nextcloud-sync.service
+```
+
+#### Step 4: Start the Service
+
+```bash
+sudo systemctl start onboarding-system.service
+sudo systemctl start nextcloud-sync.service
+```
+
+#### Step 5: Check the Service Status
+
+```bash
+sudo systemctl status onboarding-system.service
+sudo systemctl status nextcloud-sync.service
+```
+
+### Logging
+
+To view logs for the `nextcloud-sync` service, use the following command:
+
+```bash
+journalctl -u nextcloud-sync.service
+```
+
+## License
+
+This project is licensed under the MIT License. See the `LICENSE` file for details.

email_text/dittrich_email.txt

@@ -0,0 +1,16 @@
+Subject: HR Works Zugriff für neue/n Mitarbeiter/in
+
+Hello Stefanie,
+
+Es ist wieder soweit. Zuwachs!
+
+Könntest du deshalb bitte ein HR Works Konto mit den folgenden Daten erstellen:
+
+Name: {{VORNAME}} {{NACHNAME}}
+Abteilung: {{ABTEILUNG}}
+Vertragsbeginn: {{VERTRAGSBEGINN}}
+
+Falls du noch irgendwelche anderen Informationen benötigen solltest, kannst du dich bei der it-service@tubs.de melden!
+
+Vielen Dank und schöne Grüße,  
+Die IT

email_text/dritich_email.txt

@@ -1,16 +0,0 @@
-Subject: HR Works Access Request for New Employee
-
-Hello Dritich,
-
-A new employee requires access to the HR system (HR Works) for their onboarding process.
-
-Please proceed with creating the HR Works account for the following employee:
-
-Employee Name: {{VORNAME}} {{NACHNAME}}
-Department: {{ABTEILUNG}}
-Start Date: {{VERTRAGSBEGINN}}
-
-If you require any additional information, please let us know.
-
-Best regards,  
-The Onboarding Team

email_text/minuth_email.txt

@@ -1,16 +1,16 @@
-Subject: New Key Request for Onboarding Employee
+Subject: Neuer Schlüssel für Mitarbeiter/in
 
-Hello Minuth,
+Hello Carola,
 
-We have a new employee who requires a key (Schlüssel) as part of their onboarding process.
+Es ist wieder soweit. Zuwachs!
 
-Please proceed with creating and providing the necessary key for the following employee:
+Könntest du dich bitte um die Beantragung eines neuen Schlüssels mit den folgenden Daten kümmern:
 
-Employee Name: {{VORNAME}} {{NACHNAME}}
+Name: {{VORNAME}} {{NACHNAME}}
-Department: {{ABTEILUNG}}
+Abteilung: {{ABTEILUNG}}
-Start Date: {{VERTRAGSBEGINN}}
+Vertragsbeginn: {{VERTRAGSBEGINN}}
 
-If you have any questions or need further details, feel free to reach out.
+Falls du noch irgendwelche anderen Informationen benötigen solltest, kannst du dich bei der it-service@tubs.de melden!
 
-Best regards,  
+Vielen Dank und schöne Grüße,  
-The Onboarding Team
+Die IT

env/.env

@@ -17,3 +17,10 @@ ONBOARDED_DIR=onboarded_person
 TEMP_PDF_DIR=temp_pdf
 EMAIL_TEXT_DIR=email_text
 
+#Nextcloud Information
+NEXTCLOUD_BASE_URL=
+NEXTCLOUD_USERNAME=
+NEXTCLOUD_PASSWORD=
+NEXTCLOUD_DIRECTORY=
+SYNC_INTERVAL=30  # Check every 60 seconds
+

environment.yml

@@ -0,0 +1,54 @@
+name: csv_pdf
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - bzip2=1.0.8
+  - ca-certificates=2024.7.4
+  - libffi=3.4.2
+  - libsqlite=3.46.0
+  - libzlib=1.3.1
+  - openssl=3.3.1
+  - pip=24.2
+  - python=3.10.14
+  - setuptools=72.1.0
+  - tk=8.6.13
+  - ucrt=10.0.22621.0
+  - vc=14.3
+  - vc14_runtime=14.40.33810
+  - vs2015_runtime=14.40.33810
+  - wheel=0.44.0
+  - xz=5.2.6
+  - pip:
+      - backports-unittest-mock==1.5.1
+      - certifi==2024.8.30
+      - chardet==5.2.0
+      - charset-normalizer==3.3.2
+      - colorama==0.4.6
+      - docx2pdf==0.1.8
+      - idna==3.10
+      - imaplib2==3.6
+      - jinja2==3.1.4
+      - lxml==5.3.0
+      - markdown==3.6
+      - markupsafe==2.1.5
+      - mock==5.1.0
+      - numpy==2.0.1
+      - pandas==2.2.2
+      - pdfkit==1.0.0
+      - pdfrw==0.4
+      - pillow==10.4.0
+      - pypdf2==3.0.1
+      - python-dateutil==2.9.0.post0
+      - python-docx==1.1.2
+      - python-dotenv==1.0.1
+      - pytz==2024.1
+      - pywin32==306
+      - reportlab==4.2.2
+      - requests==2.32.3
+      - six==1.16.0
+      - tqdm==4.66.5
+      - typing-extensions==4.12.2
+      - tzdata==2024.1
+      - urllib3==2.2.3
+

main.py

@@ -19,6 +19,10 @@ import re
 # Load environment variables from .env file
 load_dotenv(dotenv_path=Path(__file__).parent / 'env' / '.env')
 
+# Load the sync interval from the environment or set a default (in seconds)
+SYNC_INTERVAL = int(os.getenv('SYNC_INTERVAL', 60))  # Default is 60 seconds
+
+
 # Retrieve secret credentials and directories from environment variables
 IMAP_SERVER = os.getenv('IMAP_SERVER')
 SMTP_SERVER = os.getenv('SMTP_SERVER')
@@ -273,4 +277,4 @@ def countdown_timer(seconds):
 # Continuously check for new emails every 30 seconds
 while True:
     check_email_for_csv()
-    countdown_timer(30)
+    countdown_timer(SYNC_INTERVAL)

syn_nextcloud.py

@@ -1,19 +1,21 @@
 import os
 import requests
 import time
+import json
 from requests.auth import HTTPBasicAuth
 from dotenv import load_dotenv
 from pathlib import Path
+from urllib.parse import unquote
 
 # Load environment variables from .env file securely
 env_path = Path(__file__).parent / 'env' / '.env'
 load_dotenv(dotenv_path=env_path)
 
 # Load Nextcloud credentials and base information from the .env file
-NEXTCLOUD_BASE_URL = os.getenv('NEXTCLOUD_BASE_URL').rstrip('/')  # Remove trailing slash if any
+NEXTCLOUD_BASE_URL = os.getenv('NEXTCLOUD_BASE_URL').rstrip('/')
 NEXTCLOUD_USERNAME = os.getenv('NEXTCLOUD_USERNAME')
 NEXTCLOUD_PASSWORD = os.getenv('NEXTCLOUD_PASSWORD')
-NEXTCLOUD_DIRECTORY = os.getenv('NEXTCLOUD_DIRECTORY').strip('/')  # Ensure no leading/trailing slashes
+NEXTCLOUD_DIRECTORY = os.getenv('NEXTCLOUD_DIRECTORY').strip('/')
 
 # Load the sync interval from the environment or set a default (in seconds)
 SYNC_INTERVAL = int(os.getenv('SYNC_INTERVAL', 60))  # Default is 60 seconds
@@ -24,11 +26,27 @@ local_folders = {
     'onboarded_person': 'onboarded_person'
 }
 
-def get_nextcloud_files(folder_name):
+# Local file tracking for uploads
+TRACKING_FILE = 'file_tracking.json'
+
+def load_tracking_data():
+    """Load the file tracking data from the tracking file."""
+    if os.path.exists(TRACKING_FILE):
+        with open(TRACKING_FILE, 'r') as f:
+            return json.load(f)
+    return {}
+
+def save_tracking_data(data):
+    """Save the file tracking data to the tracking file."""
+    with open(TRACKING_FILE, 'w') as f:
+        json.dump(data, f, indent=4)
+
+def get_nextcloud_files(session, folder_name):
     """
-    Get a list of files from a specific Nextcloud folder (e.g., 'attachments' or 'onboarded_person').
+    Get a list of files from a specific Nextcloud folder using a persistent session.
 
     Args:
+        session (requests.Session): The persistent session object.
         folder_name (str): The subfolder name under the group folder on Nextcloud.
         
     Returns:
@@ -39,22 +57,20 @@ def get_nextcloud_files(folder_name):
         print(f"Connecting to Nextcloud folder: {nextcloud_url}")
 
         # Make a PROPFIND request to list files in the Nextcloud directory
-        headers = {
+        headers = {'Depth': '1'}
-            'Depth': '1',  # Depth header to get all files in the directory
+        response = session.request("PROPFIND", nextcloud_url, headers=headers)
-        }
-        response = requests.request("PROPFIND", nextcloud_url, auth=HTTPBasicAuth(NEXTCLOUD_USERNAME, NEXTCLOUD_PASSWORD), headers=headers)
 
         if response.status_code != 207:
             raise Exception(f"Failed to list Nextcloud files in {folder_name}. Status code: {response.status_code}")
 
         print(f"Successfully retrieved file list from Nextcloud folder: {folder_name}.")
 
-        # Extract filenames from the XML response
         files = []
         for line in response.text.split('\n'):
             if '<d:href>' in line:
                 filename = line.split('<d:href>')[1].split('</d:href>')[0].split('/')[-1]
-                if filename:  # Avoid empty strings
+                filename = unquote(filename).strip()
+                if filename:
                     files.append(filename)
 
         print(f"Nextcloud '{folder_name}' contains {len(files)} files: {files}")
@@ -64,11 +80,12 @@ def get_nextcloud_files(folder_name):
         print(f"Error retrieving files from Nextcloud folder {folder_name}: {e}")
         return []
 
-def upload_file_to_nextcloud(file_path, filename, folder_name):
+def upload_file_to_nextcloud(session, file_path, filename, folder_name):
     """
-    Upload a file to a specific Nextcloud folder (e.g., 'attachments' or 'onboarded_person').
+    Upload a file to a specific Nextcloud folder using a persistent session.
     
     Args:
+        session (requests.Session): The persistent session object.
         file_path (str): The full path to the file to upload.
         filename (str): The name of the file being uploaded.
         folder_name (str): The Nextcloud subfolder where the file should be uploaded.
@@ -78,43 +95,57 @@ def upload_file_to_nextcloud(file_path, filename, folder_name):
         print(f"Uploading file: {filename} to Nextcloud folder: {folder_name}...")
 
         with open(file_path, 'rb') as f:
-            response = requests.put(nextcloud_url, data=f, auth=HTTPBasicAuth(NEXTCLOUD_USERNAME, NEXTCLOUD_PASSWORD))
+            response = session.put(nextcloud_url, data=f)
 
         if response.status_code not in [200, 201, 204]:  # 204 is a valid status code for success
             raise Exception(f"Failed to upload {filename}. Status code: {response.status_code}")
         else:
             print(f"Successfully uploaded {filename} to Nextcloud folder: {folder_name}.")
+            return True
 
     except Exception as e:
         print(f"Error uploading file {filename} to Nextcloud folder {folder_name}: {e}")
+    return False
 
-def check_for_new_files(local_folder, nextcloud_files, nextcloud_folder):
+def check_for_new_files(session, local_folder, nextcloud_files, nextcloud_folder, tracking_data):
     """
-    Check for new files in the local folder and upload them to Nextcloud.
+    Check for new files in the local folder and upload them to Nextcloud using a persistent session.
     
     Args:
+        session (requests.Session): The persistent session object.
         local_folder (str): The local folder to check for new files.
         nextcloud_files (list): The list of files already present in the Nextcloud folder.
         nextcloud_folder (str): The corresponding Nextcloud folder to upload new files to.
+        tracking_data (dict): The dictionary containing tracked files and their modification times.
     """
     try:
         local_files = os.listdir(local_folder)
         print(f"Files found in local folder '{local_folder}': {local_files}")
 
         for file in local_files:
-            if file not in nextcloud_files:
             file_path = os.path.join(local_folder, file)
-                print(f"New file detected: {file} (from {local_folder})")
+            file_mtime = os.path.getmtime(file_path)
-                upload_file_to_nextcloud(file_path, file, nextcloud_folder)
+
+            # Check if the file is new or modified
+            if file not in nextcloud_files and (file not in tracking_data or tracking_data[file] != file_mtime):
+                print(f"New or modified file detected: {file} (from {local_folder})")
+                if upload_file_to_nextcloud(session, file_path, file, nextcloud_folder):
+                    # Update tracking data after successful upload
+                    tracking_data[file] = file_mtime
             else:
-                print(f"File {file} already exists in Nextcloud folder {nextcloud_folder}. Skipping...")
+                print(f"File {file} already exists in Nextcloud folder {nextcloud_folder} or has not been modified. Skipping...")
     except Exception as e:
         print(f"Error checking for new files in {local_folder}: {e}")
 
-def sync_folders():
+def sync_folders(session):
     """
-    Check local folders for new files and upload them to the corresponding Nextcloud folders.
+    Check local folders for new files and upload them to the corresponding Nextcloud folders using a persistent session.
+    
+    Args:
+        session (requests.Session): The persistent session object.
     """
+    tracking_data = load_tracking_data()
+
     try:
         for local_folder, nextcloud_folder in local_folders.items():
             print(f"Checking local folder: {local_folder} for new files...")
@@ -126,10 +157,13 @@ def sync_folders():
                 continue
 
             # Get a list of files in the corresponding Nextcloud folder
-            nextcloud_files = get_nextcloud_files(nextcloud_folder)
+            nextcloud_files = get_nextcloud_files(session, nextcloud_folder)
 
             # Check for new files in the local folder and upload them
-            check_for_new_files(local_folder, nextcloud_files, nextcloud_folder)
+            check_for_new_files(session, local_folder, nextcloud_files, nextcloud_folder, tracking_data)
+
+        # Save updated tracking data
+        save_tracking_data(tracking_data)
 
     except Exception as e:
         print(f"Error during folder synchronization: {e}")
@@ -151,12 +185,17 @@ def start_periodic_sync():
     Start the periodic synchronization of local folders to Nextcloud.
     The synchronization interval is specified in the .env file or defaults to 60 seconds.
     """
+    # Use a persistent session for Nextcloud requests
+    with requests.Session() as session:
+        # Set up authentication for the session
+        session.auth = HTTPBasicAuth(NEXTCLOUD_USERNAME, NEXTCLOUD_PASSWORD)
+
         while True:
             print("Starting folder synchronization...")
-        sync_folders()
+            sync_folders(session)
             print("Synchronization complete.")
             countdown_timer(SYNC_INTERVAL)  # Wait for the specified sync interval
 
 if __name__ == "__main__":
-    print("Starting periodic folder synchronization service...")
+    print("Starting periodic folder synchronization service with a persistent session...")
     start_periodic_sync()