Rclone md5 per file

MD5 for blobs

Adds MD5 hashsum per file on blob that doesn't support the backend command

if backend

rclone backend set-md5 --all "azdst:test/foo"

Example of

test.txt
test.txt.MD5
d8e8fca2dc0f896fd7cb4cb0031ba249  test.txt

If not

#!/bin/bash
set -euxo pipefail

REMOTE="$1"
HASHSUM="MD5" # Or another hash like SHA1

# Create a temporary directory to stage the checksum files
TMP_DIR=$(mktemp -d)

# Ensure the temporary directory is removed when the script exits
trap 'rm -rf -- "$TMP_DIR"' EXIT

echo "Generating $HASHSUM checksums for all files in $REMOTE..."

# 1. Get all checksums in ONE command and create sidecar files locally.
#    The 'while read' loop only processes text here; it does not run new commands.
rclone hashsum "$HASHSUM" "$REMOTE" | while read -r checksum file; do
  # Re-create the directory structure within the temp folder
  mkdir -p "$TMP_DIR/$(dirname "$file")"

  # Write the checksum file
  echo "$checksum  $file" > "$TMP_DIR/$file.$HASHSUM"
done

echo "Uploading all .$HASHSUM files to $REMOTE..."

# 2. Upload all the generated checksum files in ONE command.
rclone copy "$TMP_DIR" "$REMOTE" --include="**.$HASHSUM"

echo "$HASHSUM checksum files have been added to the remote."