NESUG 2009

Coders' Corner

A Secure Automated FTP Process to Transfer Data from a Remote Server

Shankar Yaddanapudi, SAS Consultant, Minneapolis, MN

ABSTRACT
A SAS Risk Dimensions based system was implemented for use by an investment group of a bank. The system requires up to date market data as input. An interface to Bloomberg was implemented which consists of a set of macros which request and download required market data automatically from Bloomberg server every business day. One of the requirements for the process was that the transfer of data files to and from Bloomberg server should be completely automatic and as fail-proof as possible. Another requirement was that the Bloomberg login credentials as well as reply files from Bloomberg need to be encrypted, for security reasons. The password encryption requirement was handled by PROC PWENCODE in conjunction with filename FTP engine which accepts encrypted password. The more challenging task of making the process near fail-proof was handled by a combination of SAS functions which will be discussed in this paper. The requirement was challenging in this case because the files are generated dynamically on the fly on Bloomberg server, and in some situations the files might not be available for several reasons. The process was implemented in such a way that it would gracefully recover in abnormal situations.

INTRODUCTION
SAS/BASE provides a convenient tool to transfer files to and from a remote server through filename FTP engine. The typical situation presents little challenge in terms of execution, and in case something goes wrong, for example if transfer fails, another attempt could easily be made. In this paper we present a situation where the FTP needs to be done in an automated way with little user interaction, while being able to gracefully recover in abnormal situations. First, the context in which a secure, automated FTP process is needed is presented. Then conventional FTP statement is presented, along with its limitations. Finally, a FTP process is presented to overcome these limitations. SAS/Risk Dimensions is an Enterprise Risk Management solution to help financial institutions manage their risk. To implement a market risk solution, SAS/RD requires two major inputs, portfolio data and market data. The acquisition and management of portfolio and market data presents a significant challenge in a typical SAS/RD implementation. While portfolio data acquisition can be managed more easily, the market data management presents some unique challenges. Market data can be sourced from several vendors, including Bloomberg. While Bloomberg provides different ways to obtain market data, for automated SAS/RD solution, the appropriate method is to use the batch FTP process. This process needs to be secure, automated and near fail-proof; Secure, to keep the data confidential; Automated, to help run the SAS/RD solution with no user interaction; A brief explanation about the process to obtain market data from Bloomberg follows, to help better understand the challenges involved. The first step is to create a request file, which contains a description of required market data, in a format defined by Bloomberg. These files end with an extension .req, and are uploaded to Bloomberg server by FTP. Once a .req file is placed on Bloomberg server, Bloomberg examines the file, and if the file is in correct format, generates a .copied file. And finally Bloomberg retrieves market data and places the reply in a .reply file. Once a .reply file appears on Bloomberg server, the file can be downloaded for further processing. It needs to be noted that, a reply file is not always guaranteed to be generated. There might be several reasons for this, for example, incorrect format of request file. The transfer of files, both request and reply files can be done using a conventional FTP statement; A brief overview of FTP filename engine is presented next.

FILENAME FTP ENGINE

The filename statement allows the user to define a handle which can be used to execute a FTP command: filename ftpdir ftp "&reqfile" cd='/' user="&USERNAME" pass="&PASSWORD host="&SERVER";

The above statement defines a handle ftpdir. reqfile is a macro variable which resolves to request file to be uploaded, &username and &password are login credentials to remote server. 1

NESUG 2009

Coders' Corner

The following statements make use of the handle and execute the FTP statement: filename locfile "&outdir.\&reqfile"; data _null_; infile locfile end=eof; input; file ftpdir("&reqfile"); put _infile_; run; The result of the above statements is to upload a file to the remote server. While this approach works well usually, very rarely it fails and issues the message ERROR: Physical file does not exist". There are several reasons for this message to appear, including the possibility that the file in question does not exist, or access to that file has been denied for some reason. However, the message might also appear even if the file exists and is available for download. It is possible that this could be a bug, and in fact SAS Tech support does mention this bug and offers two remedies. Re-run the FTP code. Apply a fix on remote server as suggested in :

http://support.sas.com/kb/5/883.html Changing the setting is ruled out in this case as the remote server in question is Bloomberg. In our experience, re-running the code always works, however the goal is to create a fail-proof process which can be ported to a production environment. It was important to avoid a failure in the first place, because that would mean significant delays for the client. The bug was traced to INFILE statement in the above code snippet, and the solution was to avoid the INFILE statement altogether. We provide an alternative approach in the next section.

FTP BY FILE-HANDLING FUNCTIONS

There are a group of file-handling functions originally part of SCL language, which provide an alternative method of transferring files. These functions include filename, fopen, fclose, fread and fwrite. Figure 1.0. illustrates the use of these functions to open and close a file on a remote server. While the most common use of filename function is to assign a reference to files existing on local system, it can also be used to refer to files on a remote sever. The syntax of filename function is: FILENAME(fileref, file-name <,device-type <,host-options <,dir-ref>>>) Referring to Figure 1.0, cHandle is the file reference to be created, &copiedfile resolves to the file in question, and the third option ftp, indicates the device-type, followed by host-options. The file reference cHandle, can then be used by other functions to open, close, read or write contents. A quick review of fopen and fclose functions follows. The syntax is:

FOPEN(fileref<,open-mode<,record-length<,record-format>>>) FCLOSE(file-id) The fopen function opens an external file and returns a file identifier. In Figure 1.0 fopen is shown using cHandle, and file_id is the identifier returned. If the file open fails the returned identifier is 0, and a non-zero return indicates a successful open. The argument I, indicates that the file is opened in read-only mode. The fclose function closes the file referenced by file_id. Once the remote file is open, fread and fget functions can be used to read the contents and write them out to a local file. Their syntax is: FREAD(file-id) FGET(file-id,variable<,length>)

The fread function reads a record from an external file into the File Data Buffer (FDB) while the fget function copies data from the File Data Buffer (FDB) into a variable. The use of fread and fget functions is demonstrated in Figure 2. The advantage of this approach is that it offers tight error control, i.e. a series of checks can be done and if any step fails, some kind of action can be taken. That kind of flexibility is not available with FTP filename engine. 2

NESUG 2009

Coders' Corner

FTP BY FILE-HANDLING FUNCTIONS: DOWNLOADING A FILE TO REMOTE SERVER

Figure 2. shows the technique employed for downloading file from a remote server. The basic idea is to use filename function to define a reference to remote file and then use fopen, fread and fget functions to read the contents of file and write them to a local file. After each invocation of the functions, the return code is checked for success. Another thing shown is an outer loop controlling the number of attempts made to download the file. The download can fail for any number of reasons, in which case few more attempts are made depending on the value of the macro variable nTries. Finally the file reference is cleared using filename clear statement. The file handle created by fopen is closed by using fclose function. Both these steps are essential to prevent unexpected errors. It is to be noted this technique can be used to download any type of file, including zipped files.

FTP BY FILE-HANDLING FUNCTIONS: UPLOADING A FILE TO REMOTE SERVER

Figure 3. shows the technique employed for uploading file to a remote server. The basic idea is to convert the file to be uploaded into a SAS data set and then use filename FTP engine to upload the file. This approach avoids using infile statement in the context of a FTP engine statement.

ENCRYPTING THE PASSWORD

The login password has been encrypted using PROC PWENCODE. The encrypted password is stored in a file, and a macro creates a macro variable password which is used in FTP statements shown in the code. It is to be noted that while FTP statements accept the encrypted password, this approach would not work if one wishes to use the DOS-FTP using say, X command. For an excellent recent discussion on issues surrounding encrypting password, please see the paper by Paul D. Sherman and Arthur L. Carpenter in SAS Global Forum 2009 Conference.

CONCLUSIONS
An approach is presented for transferring files to and from a remote server. The approach offers better error control compares to conventional FTP approach.

DISCLAIMER:
The contents of this paper are the work of the author and do not represent the opinions, recommendations, or practices of SAS Institute.

REFERENCES
Paul D. Sherman and Arthur L. Carpenter. "Secret Sequel: Keeping Your Password From the Log." Proceedings of the SAS Global Forum 2009 Conference.

ACKNOWLEDGMENTS
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are registered trademarks or trademarks of their respective companies.

CONTACT INFORMATION
Please contact the author if you have any questions or comments: Shankar Yaddanapudi Shankar.stat@gmail.com

NESUG 2009

Coders' Corner

Fig 1. Code to demonstrate filename and fopen functions

data _null_; length rec $ 32767; rec = ''; rc1 = filename( 'cHandle', "&copiedfile", 'ftp', "host='&server1'" || "cd='/'" || " user='&username' pass='&password'"); file_id = fopen('cHandle','I'); if file_id eq 0 then do; put 'NOTE: file open failed:' file_id; stop; end; if file_id ne 0 then do; rc4 = fclose(file_id); put 'NOTE: Request File succesfully co pied by Bloomberg'; end; run;

NESUG 2009

Coders' Corner

Fig 2. Code to demonstrate downloading a file from a remote server

%let outdir=C:\NESUG\code; %let repfile=testfile.txt; %let nTries = 4; %macro getFile(); %local iter1 fsize; %let iter1=0; %let fsize=0; %do %while (&iter1 le &nTries and &fsize ne 1); filename outfile "&outdir.\&repfile"; %let iter1=%eval(&iter1+1); data _null_; length rec $ 32767; rec = ''; rc1 = filename( "fHandle", "&repfile", "ftp", "host='&server1'" || " cd='/'" || " user='&username' pass='&password'" ); file_id = fopen('fHandle','I'); if file_id ne 0 then check=fread(file_id); else check=1; file outfile; if file_id ne 0 then do; do while (check eq 0); rc3 = fget(file_id,rec,32767); if rc3 eq 0 then do; reclen = length(rec); put rec $varying. reclen; end; check=fread(file_id); end; rc4 = fclose(file_id); if rc3 eq 0 then call symput('fsize',put(1,best.)); end; run; %end; filename outfile clear; %mend getFile;

NESUG 2009

Coders' Corner

Fig 3. Code to demonstrate uploading a file to a remote server

%let outdir=C:\NESUG\code; %let reqfile=testfile.txt; %let nTries=4; filename ftpdir ftp "&reqfile" cd='/' user="&USERNAME" pass="&PASSWORD" host="&SERVER1" recfm=v; /* create a SAS dataset from clear file */ data work.reqfile; length rec $32767; infile "&outdir.\&reqfile" missover pad lrecl=32767; input rec $32767; run; %macro upLoad(); %local upload; %let upload=0; /* will become 1 upon successful upload */ %local iter0; %let iter0=0; %do %while (&iter0 le &nTries and &upload ne 1); %let iter0=%eval(&iter0+1); %if %sysfunc(exist(reqfile)) and %sysfunc(fileref(ftpdir)) le 0 %then %do; data _null_; set work.reqfile; file ftpdir; reclen = length(rec); put rec $varying. reclen; if _ERROR_ eq 0 then call symput('upload',put(1,best.)); run; %end; %end; %mend upLoad;