Not logged in - Login

Gobus/GridFTP Data Transfers

Globus Overview

URL: http://www.globus.org

Globus Connect enables a computer to use the Globus file transfer and sharing service. It makes it simple to create a Globus endpoint on practically any system, from a personal laptop to a supercomputer. Endpoints are computers between which files need to be transferred. Globus shares are specific locations/collections on an endpoint. While an endpoint is an entire computer/server filesystem, you can identify a specific directory as a share point.

The main advantages of Globus Connect are speed and the user interface. Globus uses a multi-threaded transfer technology (GridFTP) to increase the transfer speeds considerably over other electronic transfer options (e.g., ssh, rsync, ftp). The user interfaces include a file explorer-type web client and a command line interface (CLI).

Globus Connect comes in two flavors: Globus Connect Personal is designed for use by a single user on a personal machine; Globus Connect Server is designed to be installed by a system administrator on multi-user computing and storage resources.

How to Connect to the IWDW with Globus

The IWDW has Globus Connect Servers configured for downloading modeling platforms and data archives from the warehouse. To connect to the IWDW using Globus:

  1. Install and create a Globus Connect Server or Personal Endpoint account.
  2. Have an invitation to connect to the IWDW Endpoint or share. To receive an invitation, email your GlobusID (received when you signed up for a Globus account) to Rodger Ames at CIRA. He will invite you to access the IWDW endpoints and shares.
  3. Accept the invitation to join (you'll get an email) and bookmark the IWDW Endpoints and shares.
  4. Test the file transfer using the Globus web interface. Interact with the IWDW project team to check the transfer speeds realized with Globus

IWDW Endpoints

iwdwcira#viking

iwdwcira#valkyr

IWDW Shares

Base11b Emissions

Base11b CMAQ

Base11b CAMx

Globus Interfaces

There are two ways to connect/initiate transfers from the IWDW servers using Globus. The web client and a command line interface (CLI). For both interfaces, users need to have a Globus login, which can be created when Globus Connect is installed, and a login to the IWDW servers.

Web Client

If transferring files between two servers, and not a personal endpoint, users can log into the web client from the globus.org website via a browser. You'll need to know the Globus endpoint handles (e.g., iwdwcira#viking) for the two servers for configuring the transfer.

If transferring between a server and a personal endpoint, you must first activate the personal endpoint by running the Globus Connect Personal application on the endpoint. The Globus documentation is very detailed and will guide you through this process for creating an endpoint on a Linux, MacOS, or Windows machine. Once that is done, you can connect via a web browser for doing drag-and-drop file transfers between endpoints.

An example of the Globus Connect Web Client is shown below.

Globus Web Client

Globus Command Line Interface (CLI)

Globus also has a command line interface for transferring files between endpoints. The Globus CLI is particularly useful for scripting batch transfers.

The basic form of the CLI is to login into Globus using ssh, and then run a transfer command: 

$ ssh -l [globus user ID] cli.globusonline.org

$ transfer -- [origin endpoint]/[directory/filename] [destination endpoint]/[directory/filename]

Here's an example csh script for transferring files between two endpoints. 

# Globus user ID
set UID = zacadelman

# This is the endpoint where the data are coming from
set origin_ep =  iwdwcira#viking
# This is the endpoint where the data are going
set destination_ep =uncch#unc-rc-dm
# Set the directory path on the destination where the files should go
set dest_dir = /path/to/put/files

# Set the files to transfer
set filedir = /path/to/files
cd $filedir
foreach file ( `ls *` ) 

# Get a Globus process ID
set TID = `ssh -l [globus user ID] cli.globusonline.org transfer --generate-id`

# Use Globus to transfer the file from origin to destination
echo "${origin_ep}$cwd/$file ${destination_ep}$dest_dir/$file" | ssh -l  $UID cli.globusonline.org transfer --taskid=$TID 
# Wait for the task to finish before starting the next transfer
ssh -l $UID cli.globusonline.org wait $TID
end

Here's an example csh script for transferring directories between two endpoints. 

# Globus user ID
set UID = zacadelman

# This is the endpoint where the data are coming from
set origin_ep =  iwdwcira#viking
# This is the endpoint where the data are going
set destination_ep =uncch#unc-rc-dm
# Set the directory path on the destination where the data should go
set dest_dir = /path/to/put/directories

# Set the directories to transfer
set indir = /path/to/directories
cd $indir
foreach dir ( `ls  $indir` ) 

# Get a Globus process ID
set TID = `ssh -l [globus user ID] cli.globusonline.org transfer --generate-id`

# Use Globus to transfer the directories and all contents from origin to destination
echo "${origin_ep}$cwd/$dir/ ${destination_ep}$dest_dir/$dir/ -r" | ssh -l  $UID cli.globusonline.org transfer --taskid=$TID 
# Wait for the task to finish before starting the next transfer
ssh -l $UID cli.globusonline.org wait $TID
end

Configure Globus on the IWDW (for IWDW Admins)

Login to viking 

sudo vi /etc/globus-connect-server.conf
sudo globus-connect-server-setup --force

Using Globus Shares for Platform Distributions

Globus can be used to create shared endpoints that take users directly to IWDW modeling platforms (e.g., WAQS 2011b). When a new user signs up for Globus, say an IWDW collaborator, they can be invited to be a member of group for which there is access to a shared endpoint.

To create a shared endpoint with restricted access, use the following steps.

  1. Login to globus.org using the iwdwcira account
  2. Create a group called "WAQS Base 2011b". This should only need to be done once per platform.
  3. From the iwdwcira#viking endpoint, navigate to the directory path for the share to create. For example, iwdwcira#viking:/data2/3SDW/3SAQS/2011/Base11b/EMIS is being shared as "Base11b Emissions". Once the directory is found, click the "share" button and configure the endpoint as needed
  4. In the configuration menu, make this share only accessible by the applicable group, e.g. WAQS Base2011b (read-only)
  5. Create a user called iwdwbase11b and make a this user a member of the group WAQS Base 2011b.

Users of the IWDW who are looking for access the 2011b platform can be given the login/password for iwdwbase11b. When they login to the globus web client, the Base2011b endpoints will be the only thing that they have access to. Alternatively, if a user already has a Globus login (this would be the preferred approach) then they can be invited into the Base11b group for access to the base11b platform files.

Viking Configuration Notes

Viking Configuration