Category Archives: Bridge to R

Slowly Getting there…

Gosh, have you ever had one of those weeks where everything seems to work against you? That has been the last week or so for me. I tend to make lists of tasks that I want to accomplish for the next seven days. One of the tasks was to update the website and push out a Study Edition of the Bridge to R for WPS (which we refer to around here as Bridge to R SE). Our grand plan was to release the Bridge to R SE for both the Macintosh as well as for Windows at the same time.

The code was all put together and running quite nicely on both OS platforms when our Mac Mini started experiencing some strange behavior. At first I could not get the mouse to work properly. The mouse buttons were backwards and the scroll button didn’t function at all. Then I noticed that at times when I rebooted, it would not ask me for my password credentials. Finally, when I would go into System Preferences, I could not open any icon. It was as if it was locked down.

So, off to the Apple Store to have a Genius look at it. I explained the problems and he plugged in a monitor, mouse and keyboard. As we was testing everything I said was not working, it of course worked for him. As he continued to look at the system, funky things started happening. First the video output flashed a few times and then stopped working. He quickly checked his cables and switched to the HDMI port. After a few minutes, the HDMI video stopped working. My Mac as disintegrating right before our eyes!

Long story short, the Mac is in the shop and I expect it to be back in a few days. Once I get the machine back, I can restore the Bridge to R and compile it and make it available as an SE version. I’m excited to make the Bridge to R available for WPS users who 1) want to learn about R, and 2) want to learn about the Bridge and evaluate it at their own leisure.

You can register to download the Bridge to R SE at MineQuest.com or by clicking here. The Windows version is available immediately. The OS X version will be available in a few days. If you are an OS X user and select the OS X build, we will keep your registration information and send out an email and the appropriate links as soon as we build and upload the latest version.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

Bridge to R Demo Reel

Finally making some headway on a short demo reel for the Bridge to R. We decided against narrating anything and just included some pop-ups and pointers to show what is happening in the video.

The video shows a few things that I believe are important. First, that the Bridge to R handles R statements and gets the text listing and log back into the Eclipse Workbench using WPS. Secondly, in addition to handling the text side of things, the Bridge can handle graphs, plots and maps. Third, and quite important actually is that the Bridge to R does a lot of work in the background to catch errors coming back from R and reporting them in the log.

The video can be viewed at:

http://www.minequest.com/downloads/Bridge2Rvids/bridgev3hd/Bridge2rV3HD.mp4

or tinyurl:

http://tinyurl.com/89rsrcr

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

A Self Study Release of the Bridge to R?

We’ve been thinking about releasing an Evaluation/Self Study (we’re not sure what to call it yet) version of the Bridge to R for those WPS users who want to learn and play with R. We have a build in house for both Windows and OS X that is limited to 1500 records. This is probably enough observations to play with and learn about the R language and how the Bridge to R integrates in to the Eclipse Workbench.

The Self Study release would only include the interface into R and not the other utilities that come with the full-blown version. It would be an easy way for R novices to manipulate data in WPS using the language of SAS and experiment with R.

The hope is that some organizations who have WPS in-house will realize the value that the Bridge to R brings to their BI stack and consider licensing the software for their desktops and servers. The Self Study version will also serve as an evaluation license. This will reduce the amount of work on our end to compile a specific time limited version for an organization wanting to look at the Bridge.

What are your thoughts? We would appreciate hearing from you in the comments section below.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

Bridge to R v3

The newest version of the Bridge to R is finally done. The Installation and User Guide has been written and the sample programs have been updated. This release is a milestone for us because we have the Bridge to R running on three major platforms. These platforms are Windows (both desktop and servers), Mac OS X on x86, and Linux on x86.

The manuals are available online and can be found at http://www.minequest.com/Bridge2R.html and select the appropriate guide from the left hand side of the page. We will be revising the guides over the next few months as we port some additional utilities over to the OS X and Linux specific versions.

We will continue to provide the Bridge to R to customers who license WPS from MineQuest as a no cost solution.  If you have licensed WPS from another reseller or from World Programming, there is a cost involved to acquire the Bridge to R. The cost for the Bridge is $240 on a desktop (Windows or OS X) without regard to the number of logical CPU’s on the workstation.

The cost for the Bridge to R on a Windows or Linux server is $599 annually per logical CPU. Running R on a large server is advantageous because you can often access  much more memory than a desktop traditionally offers and servers tend to have faster CPU’s.

If you have licensed WPS from MineQuest and desire a copy of the new version, contact us and we will begin shipping the files on Saturday, March 31st.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.


Adventures in Porting

We’ve been busy porting the new version of the Bridge 2 R over to both the Mac and Linux platforms from Windows. The Windows release of the Bridge always allowed for the use of the Bridge from within the Eclipse Workbench. WPS didn’t have the Workbench as the GUI on the Max or Linux until version 3 which is the latest release. So, here’s what I found porting a large program that has to talk to different operating platforms (i.e. calls to the OS) for such things as delete files, move files, copy files, read directories, etc… and still interface with R.

The mundane part of porting was converting a lot of “\” to “/” throughout the code. In retrospect, we could have done a better job writing the Bridge in the first place to accommodate these conventions, but we didn’t have the intention of porting code back then either.

Here’s a couple of the gotcha’s that we experienced. When you read a directory on Linux or OS X, the structure is slightly different between the two and you have to accommodate that issue. The other BIG issue is that the pathnames are much longer on Linux and OS X when reading and writing to the WPS work folders. We ended up resizing our string variables to handle that specific difference.

The above might sound trivial but one think we discovered is that when you restart your server on OS X and Linux, the new work folder is contained inside the previous folder. For example, your original folder, let’s call it work1 is now hosting work2, your new folder. Now the path name is /work1/work2. But in reality, the names of the work folders are not work1 or work2 but long strings that can be hundreds of characters long. If you have a user who likes to restart their WPS Server, you can eat up a lot of string space quickly.

Since we store a lot of metadata for the Bridge 2 R inside the work folder, R has to be able to cope with very long filenames and I’m not convinced that it really copes all that well. Speaking of file names, here’s another anomaly between Windows and Linux/OS X systems. if you have a filename such as “myfile.txt ” (note the blank space at the end of .txt) Windows handles that just fine. Windows will interpret that as meaning you wanted “myfile.txt” However, if you write such a file or try to read a file with that name under Linux or OS X, then those two names are distinctly different. On Ubuntu or Fedora, that name shows up as “myfile.txt\” when you list the files from the terminal.

It took us about three days to port the Linux version of the Bridge over from Windows. Much of that time was spent dealing with the issues in the previous paragraph. We then took the ported Linux code and tested it on OS X. It took about 20 minutes to modify the section dealing with the difference in reading directories between the two platforms, and we then had a new version of the Bridge to R running on OS X.

In retrospect, porting the code over to Unix/Linux systems was worth the effort. It took a few days for us to do the porting and much of that was due to being naive about the new ported destinations. I will talk soon about the new enhancements (and a programming change users will have to make) in the Bridge to R in my next post.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

Introducing WPS Version 3

It’s a new year and version 3 of WPS is out and available. There’s a lot of new items in this release and I think it has been well worth the wait. I personally feel that this is the block buster release that will turn heads in the corporate environment due to so many added features. With version 3, World Programming has made the software more friendly and easier to use, added new procedure support both statistically and in the core language, and added support for more language elements.

I’ve been beta testing WPS 3 for a few months now and so I’ve had a bit of time to try the software out and evaluate it extensively. Below is a list of items and features that really stood out to me in WPS V3. It’s far from being exhaustive but these are probably the most pertinent for many WPS users and SAS customers who are considering WPS as an alternative to SAS.

Workbench Support on Additional Platforms

The Eclipse workbench is available on Unix, Linux, Windows, AIX, Solaris and OS X platforms. The GUI that you’ve been using on Windows is now available on all the platforms except for z/OS. Amazingly, the Eclipse Workbench is identical (and I mean identical) across the platforms. Now when companies and organizations add servers to their data center and the users will have the same identical experience (read no learning curve) on all the platforms.

Apple OS X Support

WPS has been available on Apple’s OS X for a while, but now it has the Eclipse Workbench as the GUI. The WPS implementation on OS X is native, meaning you don’t have to run Parallels or Boot Camp to run WPS on your Mac. WPS is a true 64-bit implementation on OS X. The technical specifications for the Mac OS X implementation is that it needs to be an x86 chip and you will need version 10.5 (Leopard) or higher. WPS V3 works fine on Lion as you can see in the screen shot below.

Click here to view larger image.

About a month or two ago I bought a Mac Mini just to test WPS and to port the Bridge to R over to it. The Mac implementation is a bit prettier than on Windows or Linux in my opinion. It may be the font, but it does look nice as you can see in the image above. In testing WPS on OS X, I was amazed how easy it was to transfer my knowledge of the WPS implementation on Windows over to OS X.

There’s a lot of folks who love their Macs and WPS running natively gives the Mac faithful a SAS language compatible system that takes full advantage of their hardware. Mac users give up nothing over the Windows or Linux versions of WPS and desktop pricing for the Mac is the same as for the other desktop platforms. I’m excited for the university students where there is a preponderance of Macs and I can see the take up of WPS because of the hardware and operating system support.

Linux Support

As mentioned earlier, the workbench is also available on Linux. I have it running on both Ubuntu 10.4 LTS and Fedora 13. As you can see in the screen shots below, WPS looks the same on each of the platforms. One thing to note about the Linux version, it is only available as a server license. But for shops that have a Linux platform, WPS executes exactly like the Windows version in that it provides the end users the same experience as it pertains to the GUI.

Click here to view larger image.

Many shops run Red Hat as their preferred version of Linux and WPS does run just dandy on that platform. I find that many of the larger customers we have do use Red Hat, but I suspect that the smaller businesses use Ubuntu or Fedora instead.

AIX and Solaris

The workbench is also available on these platforms. I don’t have a license or even a machine to run these two implementations on, but I would suspect that the experience is identical to the Apple and Linux versions discussed above.

WPS Link

This is an exciting feature that is being introduced in WPS 3.0. WPS Link is a Client/Server architecture that allows you to remotely execute code on Linux or Unix servers. So for example, you can be running WPS on your Windows or Mac OS X desktop and submit your WPS code over to a WPS Linux server. The log and listing files come back from the remote machine to your desktop. WPS Link is different from SAS/Connect in that this is just a client to execute your programs on the remote box. You can’t upload and download files from the desktop to the server. But, if a lot of your processing takes place on the server and that’s where your data sets reside, this is a nifty solution for processing data on a remote server.

WPS Link is easy to setup from you WPS Desktop. Simply go to the Server Connection tab and select Local. Right mouse click on Local and you get a drop down menu. Select New || Server Connection || Remote SSH.

From that point you get a dialog box where you simply fill out the hostname (or IP Address) the Connection Name (how about “WPS Remote Server”) your user name and the Launch Command. The Launch command was the most difficult part for me. It’s simply the location of the folder where WPSLinks module is located. On my server it is:

/home/minequest/wps-3.0.0.0.19782-private/bin/wpslinks

Make sure you have SSH running on your Linux server and you will be able to remotely start WPS on the server from your desktop!

Submitting code from the desktop to the WPS Server is relatively easy. Simply go to the Submit Icon and you will find a downward facing arrow next to it. You can select whether you want to run the code Locally or on the Server. One of the cool innovations with WPS Link is that you can have multiple servers registered and choose which WPS server you would like to run your code on. Note the options in the drop down menu shown below.

WPS Link comes standard as part of your WPS purchase. So if you license WPS on the desktop, WPS Link will be available to you as part of your desktop license. Also be aware that in this first release, WPS Link requires WPS running on an AIX, Solaris, Linux Servers and Linux on Mainframe System/z. I suspect that you will see Windows Servers being supported in a later release. But for our customers running large volumes of data on Linux or Unix, this feature will be very welcome.

One other note of importance on WPS Link. You can connect from a desktop to a server and server-to-server. You cannot link from desktop-to- desktop.

Procedure Support

In version 3, World Programming has added support for the following statistical procedures.

  • T-Test
  • GLM
  • GLMMOD
  • STDIZE
  • PRINCOMP
  • DISTANCE
  • FACTOR

Database Engine Support

One of the things that SAS does that I really don’t like is the up-charge for database engines. With WPS, you get the database engines as part of the WPS license. You don’t buy these separately. With version 3 of WPS, there is now support for an XML Libname engine and Sybase databases.

If you’re a shop that has a few different databases from different vendors, it would be financially wise to take a look at WPS just for the cost reduction with the included access engines.

Other Enhancements

WPS has now implemented multi-threaded support for PROC MEANS and PROC SUMMARY. It’s always fun to look at the execution time for CPU and Real Time when dealing with multi-threaded procedures. From some of the WPS documentation you can see that the multi-threaded support that is used in MEANS and SUMMARY has made it into procedures that make use of this code such as PROC TTEST and I assume PROC CORR.

Also, for those of you who were unaware, PROC SORT is also multi-threaded and is very fast. If you do a fair amount of sorting in your environment, it would be beneficial to be running on a 64-bit platform and load up with memory. The more memory you have available to WPS, the more data it can store in RAM when sorting and this results in faster sort times.

There is now an import and export wizard for importing files into WPS and writing out WPS data sets to text files. This is pretty big in my opinion. I could never remember the PROC IMPORT statements even though I use them often. Probably everyone uses a code template for importing and exporting but this just makes life easier.

The data set viewer has been enhanced over the previous release. You can view the variables labels instead of just the variable names. You can also show and hide variable columns.

Also supported for the first time is the SYSTASK and WAITFOR commands. I’ve been waiting for these two language statements to be supported for a while. As part of the Bridge to R on Windows there is a module called MPExec. MPExec allows you to run multiple programs in parallel. I’ve had to resort to using other methods in lieu of SYSTASK and WAITFOR to get this to work. Now that these two commands are available, we will update MPExec to use these features instead, thus MPExec will become portable to other platforms.

Performance

Performance has improved in each release of WPS and WPS v3 is no exception. With multi-threaded support for more procedures and better data handling, WPS is a very viable candidate to replace SAS at many organizations. MineQuest has many customers who run large data sets (millions, ten millions and hundred millions) of records through the WPS System and performance is more than adequate. These companies are able to save loads of money over the competing software system.

Expandability

With affordable pricing, customers can use WPS to perform reporting and analytics in areas and for departments that just wasn’t justifiable before. Using WPS in lieu of SAS allows many organizations to expand the analytics platform and run more data through the system. This is because such advantageous pricing allows them to purchase an additional server at great savings over our competitors.

Data Service Providers

We love Data Service Providers! Seriously, we find that DSP’s are some of our best customers. With WPS V3, licensing terms stay pretty much the same. World Programming LTD does not have DSP fees and you are free to use WPS to service your customers by providing them with data sets, reports and analytics. For any organization that is paying DSP fees, we can dramatically reduce your license fees. Your customer is your customer.

Caveats

Before you download the latest release of WPS, there is one caveat to keep in mind for upgrading. If you are using batch command files to run your jobs, you need to modify the name of the WPS executable in your .bat or .cmd file. The program name you want to use is WPS.EXE (as opposed to the former name which was WPSI.EXE).

Also in WPS V3 the WPD dataset has changed and is faster and more robust. You can still read WPD V2 datasets but V3 datasets are the default in this release.

Finally, you will need a new license key to be able to install V3. Don’t expect to simply download it and run it using the V2 key as you have been able to do previously. If you are a MineQuest customer, contact us if you want to upgrade to V3 and we will facilitate getting you a license key.

Evaluations

MineQuest is offering free 30 day evaluations of WPS V3. You can contact us at (614) 457-3714 to request an evaluation or by email. If you prefer to request your evaluation by email, send a request to info@minequest.com.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

Life with a Mac Mini

Well I finally decided to break down and buy a Mac Mini. I’ve used Apple products before but have never become a fan because my consulting business has been mostly Windows platforms. I’ve also enjoyed the OS wars as it pertains to the Apple faithful and the Windows users.

One thing I’ve noticed is when I go to a college campus, the number of Apple computers to Windows computers. I ask nieces and kids of friends of mine who go to major universities the ratio of Apple computers to Windows computers. I’ve received feed back that the ratio is as high as 15 to 1 at some schools and the lowest has been a one-to-one ratio. That’s astounding to me and when I do visit a college campus, I often see at least a three to one ratio of Apple computers to Windows computers.

I brought the Mac Mini home and setup couldn’t be easier. It took literally five minutes to take it out of the box, plug the cables in, and press the start button. One thing I didn’t appreciate was the price gouging for extra memory. Apple wanted an extra $300 to take the machine from 2GB of memory to 8GB of memory. I went out to Newegg and found 8GB of memory for $29 after a $10 rebate. That in my mind is a huge price savings.

So far, I’ve installed R, Libre Office, and of course some of my other favorite statistical packages. Performance is OK, nothing stellar. But I didn’t expect that to be the case with a single 2.5 inch hard drive either.

This is a pretty decent little machine for developing and writing software. One of my goals is to have the Bridge to R running on the Mac Mini using the Linux version of the Bridge to R. I don’t expect any hassles in this respect.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

Clean Work Utility for WPS

Another new feature that will be included in the Bridge to R is the added functionality to clean temp workspaces. As most of you are aware, orphaned directories and files can take a large amount of space out of your temp work folder.

CleanWork is another Windows only application (at this time) that can be run at anytime. CleanWork will only delete old orphaned files and directories and not a currently used temp folder. A network administrator could easily setup CleanWork to run via a scheduler on a daily basis or even more often. This would ensure that your temp work space always has the most available space.

CleanWork is easy to use. For example:

%CleanWork;

will find your systems temp work space and delete all the old folders and files.

CleanWork will be available in the next release of the Bridge to R which is expected later next month.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

PDF Support for the Bridge to R

We’re still developing and adding on to the Bridge to R and want to announce a major enhancement to the product. We will be supporting output to PDF under the Windows version. A number of customers have asked for PDF support and we believe adding this is an easy and quick way for our customers to start creating reports using a PDF destination.

Here is a code snippet that shows how you would create a PDF file. We introduce two new macros that do the work. The first one, %PDF takes the parameters for creating the PDF file. The second macro, %PDFClose terminates the code for what will appear in the PDF file.

   1: options ls=133 ps=60;

   2:

   3: %pdf(autoview=TRUE,

   4:      pdffile='c:\temp\pdfnewandbig.pdf',

   5:      pdfoptions= -pfn200 -pps4 -pfs8 -pot);

   6:

   7:    proc print data=a;

   8:       title 'Sample PDF Output using the Bridge to R';

   9:       footnote 'Generated the Bridge to R by MineQuest';

  10:    RUN;

  11:

  12:    Proc Means data=a n nmiss min max range std maxdec=4;

  13:    run;

  14:

  15: %pdfclose;

We allow you to add a number of parameters to modify the PDF output. The parameters above tell the macro which font to use (-pfn200), the page size (-pps4), the point size (-pfs8) and to use a landscape orientation (-pot). There are many other parameters that you can use including encryption.

The PDF functionality will be available in the next release of the Bridge to R which should be available by the end of October. There will be some updated documentation that will include PDF support and the new documents will remove any references to the Bridge running under the SAS System since we don’t support that any longer.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

How Important are Gecoding Services?

I’ve been thinking about writing a Geocoder for WPS for some time now and have elements of it done. One thing that strikes me as being problematic ,and I should have thought about this aspect earlier, is the issue of keeping address and map tables up-to-date. I’m not sure how often these tables would have to be up-dated but the amount of data is tremendous that would have to be downloaded on say a quarterly basis.

So, I’m now reconsidering the direction I took and think a better way is to simply allow the user (i.e. a WPS licensee holder who purchased their WPS license from MineQuest) to move their address data to a remote service for geocoding. There are a number of free and pay services offering geocoding and I don’t like the idea of paying for such a thing if it is at all avoidable.

Looking at the pay services, Bulk Geocoder has pricing on their website and they charge $500 for 100,000 records. I personally think that’s a substantial amount of money to pay. The free services limit the amount of data (i.e. the number of addresses) they will geocode for free but some of them are quite lenient. For example, Microsoft will allow a maximum of 200,000 addresses to be geocoded at a time. That’s pretty decent and I suspect that would handle the bulk requirements of many WPS license holders who need to geocode addresses.

Since this would be a bulk geocoding system, i.e. not processing a record interactively, a web service written for and using WPS to communicate with the geocoding organization would be very useful and quite economic. SAS likes to add a tax to companies who are Data Service Providers and many of these so called DSP’s are in the advertising and marketing industries. If they can realize a savings over using the SAS System, then this is something that we need to strongly consider offering.

There’s some time needed for further research and I have questions on turnaround time, but if the turnaround time is low, then the cost savings over using an outside geocoding service or if you are a SAS user and have to use SAS/GIS (think $$$) then this would be a very reasonable solution.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.