FTP Retrieval System
OverviewThis page documents the configuration and process for allowing warehouse data requesters to retrieve their data over ftp.
The goals of this system are as follows:
- Allow users to retrieve their requested data over the internet.
- Limit users to retrieving only the data they have requested
- Disallow shell access for users - File retrieval only
- Require that users provide a username and password when retrieving their data.
- Require users to retrieve their data within a specified time frame.
- Avoid duplicating data on the server.
- Prevent users from changing data housed on the server.
- Maintain the directory structure of requested data.
- Automate the system configuration as much as possible.
The basic procedure is as follows:
- Configure the ftp server to restrict users to their home directory
- Create a user account and home directory for the data requester. The account is created with an expiration date and prohibition of ssh access.
- Create a directory tree to match that of the requested data within the users home directory.
- Use mount --bind to effectively symlink the tree in the users home directory to the requested data in the main data store.
- Generate a wget script containing the users credentials to download the requested data from the mounts in their home directory.
Preliminary ConfigurationThis section describes the steps necessary to configure the ftp server to accomplish the goals above. These steps have already been applied to viking.
Create a user group for ftponly usersThis step is necessary to allow the ssh daemon to differentiate ftp-only users from general users. FTP is normally served (on viking/centos) byt vsftpd, but in order to allow ftp-only access and prohibit shell access the ssh daemon has an ftp-only subsystem. The following creates a group called sftponly:
Create an alternate home directory for ftponly usersThis step is not strictly necessary but may simplify maintenance later. Users with ftp-only access are given their own home directory which is distinct from /home/ which is shared among legitimate users. The following creates an alternate root directory called /home-ftponly/
sudo mkdir /home-ftponly/
Enable sftp-only subsystem for the ssh daemonThe ssh daemon has an ftp-only subsystem. This is an alternate to the normal ftp server vsftpd. Using the ftp subsystem of the ssh daemon is an easy way to block shell access for ftp-only users. The following configuration matches any user belonging to the ftponly group (here called sftponly) and forces those users into their own home directory (shown here as /home-ftponly/[username])
Configuration of /etc/ssh/sshd_config
##The following section enables sftp-only access for users in the sftponly group Subsystem sftp internal-sftp Match Group sftponly ChrootDirectory /home-ftponly/%u AllowTCPForwarding no X11Forwarding no ForceCommand internal-sftp
In order to fully enable the ftp access a few more steps are necessary, these steps are performed when a new ftponly user is created using the script described in another section.
- Give root ownership of /home-ftponly/username
sudo chown root:root /home-ftponly/username
- Give read and execute permissions to group and other
sudo chmod go+rx /home-ftponly/username sudo chmod go+rx /home-ftponly/username/any/mounts/
then create directories and mount. Script performs user setup /var/DWUtils/FTPAccount/usermaker.sh Script for user deletion /var/DWUtils/FTPAccount/userdel.sh Created user account ftpacctmaker added to sudoers with nopassword access for above scripts only. web codebehind executes above script under ftpacctmaker account to create account and set up mounts.
user account on viking (if not existing) useradd -m -e 2012-02-25 -s /etc/ftponlyshell username -m make a home directory -e YYYY-MM-DD specify expiration -s /etc/ftponlyshell set users shellto the warning shell Update existing user account usermod -e 2012-02-25 -s /etc/ftponlyshell username extend expiration? ensure shell is /etc/ftponlyshell
Per Request #Build matching directory structure for requested data in user home mkdir -p /home/username/path_to_data/ #Expose main data store directory to user mount --bind /main_data_store/path_to_data/ /home/username/path_to_data/ Generate wget retrieval script with user credentials and correct path OUT_DIR=. DWUN=username DWPW=password wget -r -c -nH -P $OUT_DIR ftp://$DWUN:$DWPW@viking.cira.colostate.edu/path_to_data/ Tear Down Per User (?) Delete user account and home directory Per Request Un-mount relevant directories in /home/user/ Fix-up ##The following section enables sftp-only access for users in the sftponly group Subsystem sftp internal-sftp Match Group sftponly ChrootDirectory /home/%u AllowTCPForwarding no X11Forwarding no ForceCommand internal-sftp It is necessary to give root ownership of /home/username
and chmod 755 /home/username
then create directories and mount.
Script performs user set /var/DWUtils/FTPAccount/usermaker.sh
Script for user deletion /var/DWUtils/FTPAccount/userdel.sh
Created user account ftpacctmaker
added to sudoers with nopassword access for above scripts only.
web codebehind executes above script under ftpacctmaker account to create account and set up mounts.