Gobus/GridFTP Data Transfers

Globus Overview

Globus Connect enables a computer to use the Globus file transfer and sharing service. It makes it simple to create a Globus endpoint on practically any system, from a personal laptop to a supercomputer. Endpoints are computers between which files need to be transferred. Globus Connect is free to install and use for users at non-profit research and education institutions.

The main advantages of Globus Connect are speed and the user interface. Globus uses a multi-threaded transfer technology (GridFTP) to increase the transfer speeds considerably over other electronic transfer options (e.g., ssh, rsync, ftp). The user interfaces include a file explorer-type web client and a command line interface (CLI).

Globus Connect comes in two flavors: Globus Connect Personal is designed for use by a single user on a personal machine; Globus Connect Server is designed to be installed by a system administrator on multi-user computing and storage resources.

Globus on the IWDW

The IWDW has Globus Connect Servers configured for downloading modeling platforms and data archives from the warehouse. To connect to the IWDW endpoints, users must:

Have the Globus Connect Server or Personal endpoint set up to receive data from the IWDW endpoints (see below)
Have a login to the IWDW Server endpoint (Register with the IWDW)
Know where to go on the IWDW data archive for specific files/platforms

IWDW Endpoints

iwdwcira#viking

iwdwcira#valkyr

Using Globus Shares for Platform Distributions

Globus can be used to create shared endpoints that take users directly to IWDW modeling platforms (e.g., WAQS 2011b). When a new user signs up for Globus, say an IWDW collaborator, they can be invited to be a member of group for which there is access to a shared endpoint.

Globus Interfaces

There are two ways to connect/initiate transfers from the IWDW servers using Globus. The web client and a command line interface (CLI). For both interfaces, users need to have a Globus login, which can be created when Globus Connect is installed, and a login to the IWDW servers.

Web Client

If transferring files between two servers, and not a personal endpoint, users can log into the web client from the globus.org website via a browser. You'll need to know the Globus endpoint handles (e.g., iwdwcira#viking) for the two servers for configuring the transfer.

If transferring between a server and a personal endpoint, you must first activate the personal endpoint by running the Globus Connect Personal application on the endpoint. The Globus documentation is very detailed and will guide you through this process for creating an endpoint on a Linux, MacOS, or Windows machine. Once that is done, you can connect via a web browser for doing drag-and-drop file transfers between endpoints.

An example of the Globus Connect Web Client is shown below.

Globus Web Client

Globus Command Line Interface (CLI)

Globus also has a command line interface for transferring files between endpoints. The Globus CLI is particularly useful for scripting batch transfers.

The basic form of the CLI is to login into Globus using ssh, and then run a transfer command:

$ ssh -l [globus user ID] cli.globusonline.org

$ transfer -- [origin endpoint]/[directory/filename] [destination endpoint]/[directory/filename]

Here's an example csh script for transferring files between two endpoints.

# Globus user ID
set UID = zacadelman

# This is the endpoint where the data are coming from
set origin_ep =  iwdwcira#viking
# This is the endpoint where the data are going
set destination_ep =uncch#unc-rc-dm
# Set the directory path on the destination where the files should go
set dest_dir = /path/to/put/files

# Set the files to transfer
set filedir = /path/to/files
cd $filedir
foreach file ( `ls *` ) 

# Get a Globus process ID
set TID = `ssh -l [globus user ID] cli.globusonline.org transfer --generate-id`

# Use Globus to transfer the file from origin to destination
echo "${origin_ep}$cwd/$file ${destination_ep}$dest_dir/$file" | ssh -l  $UID cli.globusonline.org transfer --taskid=$TID 
# Wait for the task to finish before starting the next transfer
ssh -l $UID cli.globusonline.org wait $TID
end

Here's an example csh script for transferring directories between two endpoints.

# Globus user ID
set UID = zacadelman

# This is the endpoint where the data are coming from
set origin_ep =  iwdwcira#viking
# This is the endpoint where the data are going
set destination_ep =uncch#unc-rc-dm
# Set the directory path on the destination where the data should go
set dest_dir = /path/to/put/directories

# Set the directories to transfer
set indir = /path/to/directories
cd $indir
foreach dir ( `ls  $indir` ) 

# Get a Globus process ID
set TID = `ssh -l [globus user ID] cli.globusonline.org transfer --generate-id`

# Use Globus to transfer the directories and all contents from origin to destination
echo "${origin_ep}$cwd/$dir/ ${destination_ep}$dest_dir/$dir/ -r" | ssh -l  $UID cli.globusonline.org transfer --taskid=$TID 
# Wait for the task to finish before starting the next transfer
ssh -l $UID cli.globusonline.org wait $TID
end

Configuration Notes

Viking Configuration