Nexus ETH Zurich¶

This guide describes how to automatically transfer data from Nexus (ETH Zurich) to a local or project storage directory using rclone, SSH authentication, and cron scheduling.

1. Configure SSH Access¶

Add the following configuration to ~/.ssh/config:

Text Only

Host vault.nexus.ethz.ch
    HostName vault.nexus.ethz.ch
    User nexus-sftp
    Port 32523
    IdentityFile ~/.ssh/nexus
    IdentitiesOnly yes

Ensure that IdentityFile points to your private SSH key.

Set appropriate permissions:

Bash

chmod 700 ~/.ssh
chmod 600 ~/.ssh/config

2. Configure rclone¶

Configure an SFTP remote:

Bash

rclone config

Create a remote named nexus using the following connection details:

Parameter	Value
Remote name	`nexus`
Storage type	SFTP
Host	`vault.nexus.ethz.ch`
Port	`32523`
Username	`nexus-sftp`
SSH key	Path to your private SSH key

Test the connection:

Bash

rclone lsd nexus:
rclone ls nexus:files

3. Create the Transfer Script¶

Create a directory for the transfer script and its logs:

Bash

mkdir -p ~/scripts/nexus

Create the script:

Bash

nano ~/scripts/nexus/sync_nexus_data.sh

Define the remote source and local destination:

Bash

REMOTE="nexus:files"
DEST="/path/to/local/destination"

Use rclone copy to transfer the data:

Bash

rclone copy "$REMOTE" "$DEST" \
    --transfers 8 \
    --checkers 16 \
    --ignore-existing \
    --log-file "$HOME/scripts/nexus/sync_nexus_data.log" \
    --log-level INFO

The complete script should also: - Create the destination directory if it does not exist. - Record the execution start and end times. - Log the execution status (SUCCESS or FAILED). - Record the total execution duration.

Make the script executable:

Bash

chmod +x ~/scripts/nexus/sync_nexus_data.sh

4. Test the Transfer¶

Run the script manually:

Bash

/bin/bash ~/scripts/nexus/sync_nexus_data.sh

Verify that the files are available in the destination directory and inspect the log:

Bash

tail -f ~/scripts/nexus/sync_nexus_data.log

5. Automate Using Cron (Optional)¶

Open the crontab editor:

Bash

crontab -e

Add the following entry to schedule the transfer daily at 02:00 AM:

Text Only

0 2 * * * /bin/bash /path/to/scripts/nexus/sync_nexus_data.sh

Replace the script path with its actual location.

Verify the scheduled job:

Bash

crontab -l

6. Monitor Transfers¶

Transfer logs are stored alongside the script:

Text Only

~/scripts/nexus/sync_nexus_data.log

Check the latest execution summaries:

Bash

grep -E '\[(SUCCESS|FAILED)\]' ~/scripts/nexus/sync_nexus_data.log | tail -10

Important Notes¶

Ensure the SSH private key and rclone configuration are accessible to the cron user.
--ignore-existing skips files that already exist at the destination, even if their contents have changed.
Ensure sufficient storage space and write permissions at the destination.
Check logs periodically to identify failed transfers.
Consider using flock to prevent overlapping scheduled transfers.

Expected outcome: Data from Nexus is automatically transferred to the specified destination every day at 02:00 AM. Existing files are skipped, and transfer activities and execution results are recorded in the log file.