Transferring Data

Overview

Most data transfer methods support both sending and receiving, so you may initiate the transfer from either Merlin or the other system — depending on network visibility.

• From PSI Network to Merlin: Merlin login nodes are visible from the PSI network, so direct transfers using rsync, or ftp are generally preferable. Transfers from Merlin7 to PSI may require special firewall rules.
• From Merlin to the Internet: Merlin login nodes can access the internet with a limited set of protocols:
  • HTTP-based protocols on ports 80 or 445 (e.g., HTTPS, WebDAV).
  • Other protocols (e.g., SSH, FTP, rsync daemon mode) require admin configuration, may only work with specific hosts, and might need new firewall rules.
• From the Internet to PSI: Systems outside PSI can access the PSI Data Transfer Service at datatransfer.psi.ch using SSH-based protocols or Globus.

SSH-based protocols using port 22 to most PSI servers are generally not permitted.

• However, transfers from any PSI host to Merlin7 using port 22 are allowed.

Port 21 is also available for FTP transfers from PSI to Merlin7.

Choosing the best transfer method

The next chapters contain detailed information about the different transfer methods available on Merlin7.

Direct Transfer via Merlin7 Login Nodes

The following methods transfer data directly via the login nodes. They are suitable for use from within the PSI network.

Rsync (Recommended for Linux/macOS)

Rsync is the preferred method for small datasets from Linux/macOS systems. It supports resuming interrupted transfers and skips already transferred files. Syntax:

rsync -avAHXS <src> <dst>

An example for transferring local files to a Merlin project directory

rsync -avAHXS ~/localdata $USER@login001.merlin7.psi.ch:/data/project/general/myproject/

SCP

SCP works similarly to rsync but does not support resuming interrupted transfers. It may be used for quick one-off transfers. Example:

scp ~/localfile.txt $USER@login001.merlin7.psi.ch:/data/project/general/myproject/

Secure FTP

A vsftpd service is available on the login nodes, providing high-speed transfers. Choose the server based on your speed vs. encryption needs:

• login001.merlin7.psi.ch: Encrypted control & data channels. Use if your data is sensitive. Slower, but secure.
• service03.merlin7.psi.ch: Encrypted control channel only. Use if your data can be transferred unencrypted. Fastest method.

UI-based Clients for Data Transfer

WinSCP (Windows)

Available in the Software Kiosk on PSI Windows machines.

• Using your PSI credentials, connect to
  • when using port 22, connect to login001.merlin7.psi.ch or login002.merlin7.psi.ch.
  • when using port 21, connect to:
    • ftp-encrypted.merlin7.psi.ch: Fast transfer rates. Both control and data channels encrypted.
    • service03.merlin7.psi.ch: Fastest transfer rates, but data channel not encrypted.

• Drag and drop files between your PC and Merlin.

• FTP (port 21)

FileZilla (Linux/MacOS/Windows)

Download from FileZilla Project, or install from your Linux software repositories if available.

• Using your PSI credentials, connect to
  • when using port 22, connect to login001.merlin7.psi.ch or login002.merlin7.psi.ch.
  • when using port 21, connect to:
    • ftp-encrypted.merlin7.psi.ch: Fast transfer rates. Both control and data channels encrypted.
    • service03.merlin7.psi.ch: Fastest transfer rates, but data channel not encrypted.
• Supports drag-and-drop file transfers.

Sharing Files with SWITCHfilesender

SWITCHfilesender is a Swiss-hosted installation of the FileSender project — a web-based application that allows authenticated users to securely and easily send arbitrarily large files to other users. Features:

• Secure large file transfers: Send files that exceed normal email attachment limits.
• Time-limited availability: Files are automatically deleted after the chosen expiration date or number of downloads.
• Voucher system: Authenticated users can send upload vouchers to external recipients without an account.
• Designed for research & education: Developed to meet the needs of universities and research institutions.

About the authentication:

• It uses SimpleSAMLphp, supporting multiple authentication mechanisms: SAML2, LDAP, RADIUS and more.
• It’s fully integrated with PSI’s Authentication and Authorization Infrastructure (AAI).
• PSI employees can log in using their PSI account:
  1. Open SWITCHfilesender.
  2. Select PSI as the institution.
  3. Authenticate with your PSI credentials.

The service is designed to send large files for temporary availability, not as a permanent publishing platform. Typical use case:

1. Upload a file.
2. Share the download link with a recipient.
3. File remains available until the specified expiration date is reached, or the download limit is reached.
4. The file is automatically deleted after expiration.

PSI Data Transfer

From August 2024, Merlin is connected to the PSI Data Transfer service, datatransfer.psi.ch. This is a central service managed by the Linux team. However, any problems or questions related to it can be directly reported to the Merlin administrators, which will forward the request if necessary.

The PSI Data Transfer servers supports the following protocols:

• Data Transfer - SSH (scp / rsync)
• Data Transfer - Globus

Notice that datatransfer.psi.ch does not allow SSH login, only rsync, scp and Globus access is allowed.

Access to the PSI Data Transfer uses Multi factor authentication (MFA). Therefore, having the Microsoft Authenticator App is required as explained here.

Connecting to Merlin7 from outside PSI

Merlin7 is fully accessible from within the PSI network. To connect from outside you can use:

• VPN (alternate instructions)
• SSH hopx
  • Please avoid transferring big amount data through hop
• No Machine
  • Remote Interactive Access through ‘nx.psi.ch’
  • Please avoid transferring big amount of data through NoMachine