Rclone md5 per file - Revision history

Busk at 16:25, 3 July 2025

2025-07-03T16:25:18Z

Busk at 15:56, 3 July 2025

2025-07-03T15:56:34Z

Busk: Created page with "# MD5 for blobs Adds MD5 hashsum per file on blob that doesn't support the backend command ## if backend ``` rclone backend set-md5 --all "azdst:test/foo" ``` ## If not ```..."

2025-07-03T15:54:27Z

Created page with "# MD5 for blobs Adds MD5 hashsum per file on blob that doesn't support the backend command ## if backend ``` rclone backend set-md5 --all "azdst:test/foo" ``` ## If not ```..."

New page

# MD5 for blobs
Adds MD5 hashsum per file on blob that doesn't support the backend command

## if backend
```
rclone backend set-md5 --all "azdst:test/foo"
```

## If not

```
#!/bin/bash
set -euxo pipefail

REMOTE="$1"
HASHSUM="MD5" # Or another hash like SHA1

# Create a temporary directory to stage the checksum files
TMP_DIR=$(mktemp -d)

# Ensure the temporary directory is removed when the script exits
trap 'rm -rf -- "$TMP_DIR"' EXIT

echo "Generating $HASHSUM checksums for all files in $REMOTE..."

# 1. Get all checksums in ONE command and create sidecar files locally.
# The 'while read' loop only processes text here; it does not run new commands.
rclone hashsum "$HASHSUM" "$REMOTE" | while read -r checksum file; do
# Re-create the directory structure within the temp folder
mkdir -p "$TMP_DIR/$(dirname "$file")"

# Write the checksum file
echo "$checksum $file" > "$TMP_DIR/$file.$HASHSUM"
done

echo "Uploading all .$HASHSUM files to $REMOTE..."

# 2. Upload all the generated checksum files in ONE command.
rclone copy "$TMP_DIR" "$REMOTE" --include="**.$HASHSUM"

echo "$HASHSUM checksum files have been added to the remote."
```

@@ Line 17: / Line 17: @@
 ```
-#!/bin/bash
+set -euox pipefail
 REMOTE="$1"
-HASHSUM="MD5" # Or another hash like SHA1
+MODE="${2:---mode=dir}" # Default to 'file' mode if not provided. Accepts --mode=file or --mode=dir
 TMP_DIR=$(mktemp -d)
+# Ensure the tmp dir is removed when the script exits
 trap 'rm -rf -- "$TMP_DIR"' EXIT
-echo "Generating $HASHSUM checksums for all files in $REMOTE..."
+echo "Starting hash generation with mode: ${MODE#--mode=}"
 rclone hashsum "$HASHSUM" "$REMOTE" | while read -r checksum file; do
-   # Write the checksum file
+   if [[ "$MODE" == "--mode=file" ]]; then
-  echo "$checksum  $file" > "$TMP_DIR/$file.$HASHSUM"
+    mkdir -p "$TMP_DIR/$(dirname "$file")"
 done
 echo "Uploading all .$HASHSUM files to $REMOTE..."
-# 2. Upload all the generated checksum files in ONE command.
+rclone copy "$TMP_DIR" "$REMOTE" --include="**MD5SUMS"
-echo "$HASHSUM checksum files have been added to the remote."
+echo "Checksum files have been added successfully."
 ```

@@ Line 5: / Line 5: @@
 ```
 rclone backend set-md5 --all "azdst:test/foo"
 ```