Gobus/GridFTP Data Transfers

Globus Overview

URL: http://www.globus.org

Globus Connect enables a computer to use the Globus file transfer and sharing service. It makes it simple to create a Globus endpoint on practically any system, from a personal laptop to a national supercomputer. Endpoints are computers between which files need to be transferred. Globus Connect is free to install and use for users at non-profit research and education institutions.

The main advantages of Globus Connect are speed and the user interface. Globus uses a multi-threaded transfer technology (GridFTP) to increase the transfer speeds considerably over other electronic transfer options (e.g., ssh, rsync, ftp). The user interfaces include a file explorer-type web client and a command line interface (CLI).

Globus Connect comes in two flavors: Globus Connect Personal is designed for use by a single user on a personal machine; Globus Connect Server is designed to be installed by a system administrator on multi-user computing and storage resources.

Globus on the IWDW

The IWDW has Globus Connect Servers configured for downloading modeling platforms and data archives from the warehouse. To connect to the IWDW endpoints, users must:

• Have the Globus Connect Server or Personal endpoint set up to receive data from the IWDW endpoints (see below)
• Have a login to the IWDW Server endpoint (request login)
• Know where to go on the IWDW data archive for specific files/platforms

IWDW Endpoints

iwdwcira#viking

iwdwcira#valkyr

Globus Interfaces

There are two ways to connect/initiate transfers from the IWDW servers using Globus. The web client and a command line interface (CLI). For both interfaces, users need to have a Globus login, which can be created when Globus Connect is installed, and a login to the IWDW servers.

Web Client

If transferring files between two servers, and not a personal endpoint, users can log into the web client from the globus.org website via browser. If transferring between a server and a personal endpoint, you must first activate the personal endpoint by running the Globus Connect Personal application on the endpoint. Once that is done, you can connect via a web browser. An example of the Globus Connect Web Client is shown below.

Globus Command Line Interface (CLI)

$ ssh -l [globus user ID] cli.globusonline.org

$ transfer -- [origin endpoint]/[directory/filename] [destination endpoint]/[directory/filename]

Here's an example csh script for transferring files between two endpoints.

# Globus user ID
set UID = zacadelman

# This is the endpoint where the data are coming from
set origin_ep =  iwdwcira#viking
# This is the endpoint where the data are going
set destination_ep =uncch#unc-rc-dm
# Set the directory path on the destination where the files should go
set dest_dir = /path/to/put/files

# Set the files to transfer
set filedir = /path/to/files
cd $filedir
foreach file ( `ls *` ) 

# Get a Globus process ID
set TID = `ssh -l [globus user ID] cli.globusonline.org transfer --generate-id`

# Use Globus to transfer the file from origin to destination
echo "${origin_ep}$cwd/$file ${destination_ep}$dest_dir/$file" | ssh -l  $UID cli.globusonline.org transfer --taskid=$TID 
# Wait for the task to finish before starting the next transfer
ssh -l zacadelman$UID cli.globusonline.org wait $TID
end

Here's an example csh script for transferring directories between two endpoints.

# Globus user ID
set UID = zacadelman

# This is the endpoint where the data are coming from
set origin_ep =  iwdwcira#viking
# This is the endpoint where the data are going
set destination_ep =uncch#unc-rc-dm
# Set the directory path on the destination where the data should go
set dest_dir = /path/to/put/directories

# Set the directories to transfer
set indir = /path/to/directories


cd $indir
foreach dir ( `ls  $indir` ) 

# Get a Globus process ID
set TID = `ssh -l [globus user ID] cli.globusonline.org transfer --generate-id`

# Use Globus to transfer the directories and all contents from origin to destination
echo "${origin_ep}$cwd/$dir/ ${destination_ep}$dest_dir/$dir/ -r" | ssh -l  $UID cli.globusonline.org transfer --taskid=$TID 
# Wait for the task to finish before starting the next transfer
ssh -l zacadelman$UID cli.globusonline.org wait $TID
end

Configuration Notes

Viking Configuration