Gobus/GridFTP Data Transfers
Globus Overview
URL: http://www.globus.orgGlobus Connect enables a computer to use the Globus file transfer and sharing service. It makes it simple to create a Globus endpoint on practically any system, from a personal laptop to a supercomputer. Endpoints are computers between which files need to be transferred. Globus Connect is free to install and use for users at non-profit research and education institutions.
The main advantages of Globus Connect are speed and the user interface. Globus uses a multi-threaded transfer technology (GridFTP) to increase the transfer speeds considerably over other electronic transfer options (e.g., ssh, rsync, ftp). The user interfaces include a file explorer-type web client and a command line interface (CLI).
Globus Connect comes in two flavors: Globus Connect Personal is designed for use by a single user on a personal machine; Globus Connect Server is designed to be installed by a system administrator on multi-user computing and storage resources.
Globus on the IWDW
The IWDW has Globus Connect Servers configured for downloading modeling platforms and data archives from the warehouse. To connect to the IWDW endpoints, users must:
- Have the Globus Connect Server or Personal endpoint set up to receive data from the IWDW endpoints (see below)
- Have a login to the IWDW Server endpoint (Register with the IWDW)
- Know where to go on the IWDW data archive for specific files/platforms
IWDW Endpoints
iwdwcira#vikingiwdwcira#valkyr
Using Globus Shares for Platform Distributions
Globus can be used to create shared endpoints that take users directly to IWDW modeling platforms (e.g., WAQS 2011b). When a new user signs up for Globus, say an IWDW collaborator, they can be invited to be a member of group for which there is access to a shared endpoint.
To create a shared endpoint with restricted access, use the following steps.
- Login to globus.org using the iwdwcira account
- From the iwdwcira#viking endpoint navigate to the directory path for the share to create. For example, iwdwcira#viking:/data2/3SDW/3SAQS/2011/Base11b/EMIS is being shared as "Base11b
Emissions"Emissions". Once the directory is found, click the "share" button and configure the endpoint as needed - Created a group called "WAQS Base 2011b" and made this share only accessible by that group (read-only)
- Created a user called iwdwbase11b and made a member of the group WAQS Base 2011b.
Users of the IWDW who are looking for access the 2011b platform can be given the login/password for iwdwbase11b. When they login to the globus web client, the Base2011b endpoints will be the only thing that they have access to. Alternatively, if a user already has a Globus login (this would be the preferred approach) then they can be invited into the Base11b group for access to the base11b platform files.
Globus Interfaces
There are two ways to connect/initiate transfers from the IWDW servers using Globus. The web client and a command line interface (CLI). For both interfaces, users need to have a Globus login, which can be created when Globus Connect is installed, and a login to the IWDW servers.
Web Client
If transferring files between two servers, and not a personal endpoint, users can log into the web client from the globus.org website via a browser. You'll need to know the Globus endpoint handles (e.g., iwdwcira#viking) for the two servers for configuring the transfer.If transferring between a server and a personal endpoint, you must first activate the personal endpoint by running the Globus Connect Personal application on the endpoint. The Globus documentation is very detailed and will guide you through this process for creating an endpoint on a Linux, MacOS, or Windows machine. Once that is done, you can connect via a web browser for doing drag-and-drop file transfers between endpoints.
An example of the Globus Connect Web Client is shown below.
Globus Command Line Interface (CLI)
Globus also has a command line interface for transferring files between endpoints. The Globus CLI is particularly useful for scripting batch transfers.The basic form of the CLI is to login into Globus using ssh, and then run a transfer command:
$ ssh -l [globus user ID] cli.globusonline.org $ transfer -- [origin endpoint]/[directory/filename] [destination endpoint]/[directory/filename]
Here's an example csh script for transferring files between two endpoints.
# Globus user ID
set UID = zacadelman
# This is the endpoint where the data are coming from
set origin_ep = iwdwcira#viking
# This is the endpoint where the data are going
set destination_ep =uncch#unc-rc-dm
# Set the directory path on the destination where the files should go
set dest_dir = /path/to/put/files
# Set the files to transfer
set filedir = /path/to/files
cd $filedir
foreach file ( `ls *` )
# Get a Globus process ID
set TID = `ssh -l [globus user ID] cli.globusonline.org transfer --generate-id`
# Use Globus to transfer the file from origin to destination
echo "${origin_ep}$cwd/$file ${destination_ep}$dest_dir/$file" | ssh -l $UID cli.globusonline.org transfer --taskid=$TID
# Wait for the task to finish before starting the next transfer
ssh -l $UID cli.globusonline.org wait $TID
end
Here's an example csh script for transferring directories between two endpoints.
# Globus user ID
set UID = zacadelman
# This is the endpoint where the data are coming from
set origin_ep = iwdwcira#viking
# This is the endpoint where the data are going
set destination_ep =uncch#unc-rc-dm
# Set the directory path on the destination where the data should go
set dest_dir = /path/to/put/directories
# Set the directories to transfer
set indir = /path/to/directories
cd $indir
foreach dir ( `ls $indir` )
# Get a Globus process ID
set TID = `ssh -l [globus user ID] cli.globusonline.org transfer --generate-id`
# Use Globus to transfer the directories and all contents from origin to destination
echo "${origin_ep}$cwd/$dir/ ${destination_ep}$dest_dir/$dir/ -r" | ssh -l $UID cli.globusonline.org transfer --taskid=$TID
# Wait for the task to finish before starting the next transfer
ssh -l $UID cli.globusonline.org wait $TID
end