[Feature Request] Lightweight Document Deduplication (SHA-256 Hashing with Error Isolation) for Data Recipe Ingest Nodes

[TIHTCM]

Is your feature request related to a problem?

Yes. When accumulating large volumes of raw documents (PDFs, text files) over time for long-term fine-tuning projects—such as foreign language tutoring or technical domain tracking—it is highly common to accidentally leave an old document in the input folder or re-ingest a previously processed file.

Because Unsloth Studio doesn't track ingest history across separate pipeline sessions, duplicates are processed repeatedly. This injects redundant tokens into the final dataset, causing severe model overfitting (where the model memorizes exact duplicate phrases instead of learning general rules) and wasting local GPU/API compute resources during synthetic data generation.

Proposed Solution

I propose a lightweight, zero-configuration solution: add a hidden state manifest file (e.g., .unsloth_ingest_manifest.json) inside the user's designated workspace or input directory.

When a Data Recipe ingest node scans a folder, it calculates a rapid block-hash of each file, compares it against the local manifest, and seamlessly skips duplicates. I also suggest a simple UI checkbox/toggle in the ingest node settings ([x] Skip duplicate files) so users retain full agency to bypass this behavior if desired.

Developer Integration & API Documentation

Code Component Map

1. get_file_hash(file_path): An isolated utility function that streams a file to compute a SHA-256 hash string. It wraps file access in defensive try/except blocks to isolate OS-level read locks.
2. scan_and_filter_folder(input_folder, skip_duplicates): Main hook designed to run inside the folder scanning lifecycle of a Data Recipe Ingest Node. It updates internal memory structures and accepts a boolean flag directly linked to the frontend UI toggle component.

Expected Manifest Schema (.unsloth_ingest_manifest.json)

The manifest utilizes a flat key-value dictionary structure where the unique file content hash serves as the primary key. This layout allows for O(1) lookups while natively storing secondary metadata keys for future UI features (e.g., a "skipped files" overview page).

{
  "8f430b201a...8e5b4c": {
    "filename": "french_lesson_1.pdf",
    "size": 4194304
  },
  "2a18c9402f...1b7f3d": {
    "filename": "blueprint_spec_v2.pdf",
    "size": 18256102
  }
}

Suggested Implementation (Minimal, Drop-In Code)

Developers can implement this directly inside the directory-scanning logic of the ingest node with minimal changes to existing file-handling patterns:

import hashlib
import json
import os
from pathlib import Path

def get_file_hash(file_path: Path) -> str:
    """
    Computes a SHA-256 block hash to identify unique files by content.
    
    Args:
        file_path (Path): Path to the target file.
        
    Returns:
        str: Hexadecimal string hash, or an empty string if unreadable.
    """
    hasher = hashlib.sha256()
    try:
        with open(file_path, 'rb') as f:
            # Read in 64kb chunks for maximum execution speed and zero RAM spikes
            while chunk := f.read(65536):
                hasher.update(chunk)
        return hasher.hexdigest()
    except (OSError, IOError) as e:
        # Prevent app crash if a file is locked by an external PDF reader/OS or corrupted
        print(f"[Unsloth Ingest Error] Failed to read {file_path.name}: {e}")
        return ""

def scan_and_filter_folder(input_folder: str, skip_duplicates: bool = True) -> list[Path]:
    """
    Scans a folder, verifies contents against a local manifest, and yields clean files.
    
    Args:
        input_folder (str): Target system directory path passed by the Ingest Node.
        skip_duplicates (bool): User-controlled UI toggle value.
        
    Returns:
        list[Path]: Filtered collection of file targets safe for downstream processing.
    """
    folder_path = Path(input_folder)
    manifest_path = folder_path / ".unsloth_ingest_manifest.json"
    
    # 1. Load existing manifest or initialize an empty structure gracefully
    if manifest_path.exists():
        try:
            with open(manifest_path, 'r', encoding='utf-8') as f:
                processed_manifest = json.load(f)
        except json.JSONDecodeError:
            # Handle corrupted files or manual end-user profile edits without breaking the app
            print(f"[Unsloth Ingest Warning] Manifest file corrupted. Resetting index.")
            processed_manifest = {}
    else:
        processed_manifest = {} 

    files_to_process = []
    new_manifest_entries = {}

    # 2. Iterative folder evaluation
    for file_item in folder_path.iterdir():
        # Ignore sub-directories, hidden system configuration files, and the manifest itself
        if file_item.is_file() and not file_item.name.startswith('.'):
            file_size = file_item.stat().st_size
            
            # Generate unique fingerprint identifier
            file_hash = get_file_hash(file_item)
            if not file_hash: 
                continue # Skip processing line if file permissions or execution locks block access
            
            # Match fingerprint against the local history manifest
            if file_hash in processed_manifest:
                if skip_duplicates:
                    # Echo log out to terminal context or UI node terminal windows
                    print(f"[Unsloth Ingest] Skipped duplicate: '{file_item.name}' (Matches processed file: '{processed_manifest[file_hash]['filename']}')")
                    continue
            
            # Collect file for execution and track temporary index state
            files_to_process.append(file_item)
            new_manifest_entries[file_hash] = {
                "filename": file_item.name,
                "size": file_size
            }

    # 3. Save new files to the local manifest file system
    # NOTE FOR DEVS: To ensure safety against mid-pipeline crashes, call this update block 
    # after the dataset extraction/recipe generation step successfully completes.
    if new_manifest_entries:
        processed_manifest.update(new_manifest_entries)
        try:
            with open(manifest_path, 'w', encoding='utf-8') as f:
                json.dump(processed_manifest, f, indent=2)
        except (OSError, IOError) as e:
            print(f"[Unsloth Ingest Error] Could not write manifest to disk: {e}")

    return files_to_process

Best Practices Incorporated for Developers

• Zero External Database Overhead: Utilizes a simple, flat JSON manifest hidden right inside the user's data directory. This completely avoids managing SQLite schemas or introducing global application state headaches.
• Content-Based Protection: Checking file contents ensures that if a user renames a file (e.g., changing textbook.pdf to textbook_copy.pdf), Unsloth still accurately flags it as a duplicate.
• Streamed Chunk Reading: The get_file_hash function streams data in 64KB blocks, ensuring zero RAM spikes even when users process multi-gigabyte files.
• Error Isolation: Outfitted with robust try-except wrappers to protect Unsloth Studio from crashing if an incoming file is corrupted or actively locked by an external PDF reader.

User Documentation & Workflow Guide

How it Works for the User

1. Toggle Switch Activation: Inside the settings panel of your Ingest Node, you will see a new toggle checkbox labeled Skip Duplicate Files.
2. First Run Ingest: When you run your Data Recipe on a directory, Unsloth automatically creates an invisible tracker file (.unsloth_ingest_manifest.json) inside that folder. It logs a digital fingerprint of all your current files.
3. Subsequent Additions: As you work on your long-term project and add new files over weeks, you can leave old files in the folder. Unsloth will read the folder, automatically cross-reference the fingerprints, skip files you have already processed, and instantly begin working only on the newly added files.
4. Renaming Protection: If you accidentally drop the same document into the folder under a different name (e.g., lesson1_backup.pdf), Unsloth analyzes the file contents rather than the name, matching the old fingerprint and saving you from duplicating data.

Benefits to the Community and Users

1. Flawless Dataset Quality: Guarantees data integrity by eliminating duplicate rows out of the box. This prevents severe model degradation and ensures your fine-tuned model generalizes cleanly instead of rote-memorizing repetitive text.
2. Substantial Compute & Cost Savings: Many users use local or external LLM nodes inside their Data Recipes to extract and parse raw text into clean training structures. Eliminating duplicate file processing saves vast amounts of local GPU time and costly API tokens.
3. Streamlined UI UX Workflow: Provides a true "set-and-forget" workspace experience. Users can continuously drop fresh documents into their folders over the course of weeks without needing to rigorously handle tedious manual file systems, staging directories, or archives.