Category Archives: Ubuntu

Ubuntu 16.04 Released and Quick Test Drive

In the last week, Canonical has brought forth a new release of Ubuntu and it is pretty nice! Version 16.04 has a number of great features that should be of value to those who use Linux. One thing that Ubuntu has at this point is a vertical line of products. I can’t think of any other vendor who has an OS that runs on Phones, tablets, notebooks/workstations, servers and mainframes.

I decided to give it a try on one of my workstations running it in an Oracle Virtual Machine (Virtualbox to be specific) to see how WPS runs on this new release. Just to cut to the chase, it runs quite well. As a matter of fact, once I got the VM to use all of its allotted storage, WPS ran like a charm.

clip_image002

A couple of things that might be of interest to potential Ubuntu upgraders. First, Ubuntu 16.04 supports ZFS. That might be important to a few sites. The second is the support for LXD 2.0. From the Ubuntu website –

LXD 2.0

Ubuntu 16.04 LTS includes LXD, a new, lightweight, network-aware, container manager offering a VM-like experience built on top of Linux containers.

LXD comes pre-installed with all Ubuntu 16.04 server installations, including cloud images and can easily be installed on the Desktop version too. It can be used standalone through its simple command line client, through Juju to deploy your charms inside containers or with OpenStack for large scale deployments.

All the LXC components – LXC, LXCFS and LXD – are at version 2.0 in Ubuntu 16.04 LTS.

In addition to trying Ubuntu 16.04 in a VM, I have also tested it on a small server (6 LCPU with 32GB of RAM) running WPS. Although I have not benchmark tested this exhaustively, it does appear that using v16.04 with WPS 3.3.2 (which is the latest release) provides a modest performance increase. This is easily observed with multi-threaded Procedures such as Means and Summary.

About the author: Phil Rack is President of MineQuest Business Analytics, LLC located in beautiful Tucson Arizona. Phil has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and is an authorized reseller of WPS in North America.

A Summer Project

One of my summer projects is building and performance tuning a relatively inexpensive analytics server. Many of the parts that are being used have been scavenged from another server or two that have been retired. One thing I want to do this summer is report on what I have discovered in performance tuning a modest server.

The server consist of a six core AMD processor and 16GB of RAM to start out with. I would like to experiment with different combinations of RAM, hard drives, hard disk controller cards and perhaps an SSD or two. The OS will be Linux, Ubuntu 14.04 specifically.

My baseline build has just two work drives in RAID-0 and use the SATA 3 ports on the motherboard. I will use the Workstation Performance Assessment Program that I wrote about back in 2012. I’ve slightly modified that program so that it doesn’t spew output in the listing with the exception of the actual performance benchmark.

One thing I have already learned is that you need to make sure that you have the Write Cache enabled. In Ubuntu, you would do this by going to Disks and clicking on the options button at the top right of the dialog box and then selecting Drive Settings. Simply select the Write Cache and click on Enable Write Cache. You will need to do this for each disk in the raid array.

ubuntu_disk_cache

When I enable the write cache, my timings for the PROCs and data steps that took place on data sets that existed on the work array dropped 35%. That’s a big improvement!

About the author: Phil Rack is President of MineQuest Business Analytics, LLC located in Grand Rapids, Michigan. Phil has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and is a authorized reseller of WPS in North America.

Thursday Ramblings

Does anyone do comparisons of graphics cards and measure performance in a VM? Specifically, do certain graphics cards boost performance when running VM’s on the desktop? I like to see my windows “snap” open when I switch from VM to VM. As a developer, I often wonder if spending an additional $150 on a popular graphics card will yield a perceptible performance boost.

Speaking of graphics cards, we recently bought a couple of used Nvidia Quadro graphics cards from a local CAD/CAM company that is upgrading their workstations. I got these at about 5% of their original retail price so I’m happy. We were having problems getting a couple of servers to go into sleep mode using Lights Out and we discovered that we needed a different graphics card to accomplish this. The plus side is that these are Nvidia cards with 240 CUDA cores and 4GB of RAM. So we now have the opportunity to try our hand at CUDA development if we want. I’m mostly interested in using CUDA for R.

One drawback to using CUDA, as I understand it, is that it is a single user interface. Say you have a CUDA GPU in a server, only one job at a time can access the CUDA cores. If you have 240 CUDA cores on your GPU and would like to appropriate 80 CUDA cores to an application — thinking you can run three of your apps at a time, well that is not possible. What it seems you have to do is have three graphics cards installed on the box and each user or job has access to a single card.

There’s a new Remote Desktop application coming out from MS that will run on your android device(s) as well as a new release from the Apple Store. I use the RDC from my mac mini and it works great. I’m not sure what they could throw in the app to make it more compelling however.

Toms Hardware has a fascinating article on SSD’s and performance in a RAID setup. On our workstations and servers, we have SSD’s acting as a cache for the work and perm folders on our drive arrays. According to the article, RAID0 performance tends to top out with three SSD’s for writes and around four on reads.

FancyCache from Romex Software has become PrimoCache. It has at least one new feature that I would like to test and that is L2 caching using an SSD. PrimoCache is in Beta so if you have the memory and hardware, it might be advantageous to give it a spin to see how it could improve your BI stack. We did a performance review of FancyCache on a series of posts on Analytic Workstations.

FYI, PrimoCache is not the only caching software available that can be used in a WPS environment. SuperSpeed has a product called SuperCache Express 5 for Desktop Systems. I’m unsure if SuperCache can utilize an SSD as a Level 2 cache. It is decently priced at $80 for a desktop version but $450 for a standard Windows Server version. I have to admit, $450 for a utility would give me cause for pause. For that kind of money, the results would have to be pretty spectacular. SuperSpeed offers a free evaluation as well.

If you are running a Linux box and want to enjoy the benefits of SSD caching, there’s a great blog article on how to do this for Ubuntu from Kyle Manna. I’m very intrigued by this and if I find some extra time, may give it the old Solid State Spin. There’s also this announcement about the Linux 3.10 Kernel and BCache that may make life a whole lot easier.

About the author: Phil Rack is President of MineQuest Business Analytics, LLC located in Grand Rapids, Michigan. Phil has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and is a authorized reseller of WPS in North America.

Creating a WPS Launch Icon in Ubuntu

I use Ubuntu for my WPS Linux OS and it’s pretty easy to install. However, unlike the vast majority of people out there who run it in batch mode; I like to run it in interactive mode using the Eclipse Workbench. Hence I want an icon that I can click on to start WPS. Here’s how to do it.

On the Ubuntu desktop, right mouse click on an empty part of the screen and you will get a little option menu. Click on “Create Launcher…” You will see a dialog box pop up that looks like:

clip_image002

On my Ubuntu Linux Server, I installed WPS into a folder named wps-3.0.1. The directions below use that folder name as our example. You may have installed WPS into another folder so be sure to consider that when performing the tasks below.

Name: WPS 3.01

Command: /home/minequest/wps-3.0.1/eclipse/workbench

Comment: WPS 3.0.1 Linux

Click on the icon on the upper left hand of the Create Launcher Dialog Box (the little spring) and you will get a choose icon list box. Simply go to the WPS install folder and go into the eclipse folder. There you will find a file named icon.xpm. Click on icon.xpm and then click Open and then click OK.

That’s all there is to it. You should have the WPS icon installed and available from your desktop.

About the author: Phil Rack is President of MineQuest Business Analytics, LLC located in Grand Rapids, Michigan. Phil has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and is a authorized reseller of WPS in North America.

Cool and Useful Software

I always enjoy reading other folks blogs on how they work and the tools they use most every day. It’s a great way to learn about new tools and how to work more efficiently. I have to rave about the phone system we use here at MineQuest. We use VOIP and our provider is VOIPO out of Texas. The quality is tremendous as well as the support. The cost is amazing for what you get. The benefits of VOIPO are numerous for a small business, but the one I like the most is a softphone. I can travel and still be able to use my phone system almost as well as if I was in the office. You can visit the VOIPO web site to get pricing and view all the features that they offer.

Of course I use Skype. I can use Skype to call overseas and to text message with friends, family and business contacts. I have contacts that are almost always on Skype and the number of Skype contacts that I have just continues to grow. If you want or need to do business overseas, then Skype maybe the only way you can do so cost effectively. I hope to see more integration of Skype into other products and services and the availability of an easier to use API. If you don’t have at least a free Skype account, you should visit the Skype website and get Skype today.

I recently started to use a new Linux distribution called ZorinOS. I have version 6.1 and essentially, ZorinOS is Ubuntu Linux with the coolest GUI interface. With ZorinOS, you can change the interface to mimic Windows XP, Windows Vista, Windows 7, Mac OS X and the Ubuntu Unity Interface. If you are a Windows User and want to start using Linux with minimum fuss and frustration, ZorinOS is something to try. Check out the ZorinOS website to learn more.

I also signed up for Microsoft’s Office 365. There are a number of plans available and you can see all of them at the Microsoft web site. But if you have multiple machines like I do, desktop, laptop and a Mac, Office 365 gives you five simultaneous installs for $100. This is an annual license and I love that I get Outlook on all my desktop machines. I love the simplicity and the fact that I get cloud storage to store my documents so I can access them from anywhere.

About the author: Phil Rack is President of MineQuest Business Analytics, LLC located in Grand Rapids, Michigan. Phil has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and is a authorized reseller of WPS in North America.

Bridge to R v3

The newest version of the Bridge to R is finally done. The Installation and User Guide has been written and the sample programs have been updated. This release is a milestone for us because we have the Bridge to R running on three major platforms. These platforms are Windows (both desktop and servers), Mac OS X on x86, and Linux on x86.

The manuals are available online and can be found at http://www.minequest.com/Bridge2R.html and select the appropriate guide from the left hand side of the page. We will be revising the guides over the next few months as we port some additional utilities over to the OS X and Linux specific versions.

We will continue to provide the Bridge to R to customers who license WPS from MineQuest as a no cost solution.  If you have licensed WPS from another reseller or from World Programming, there is a cost involved to acquire the Bridge to R. The cost for the Bridge is $240 on a desktop (Windows or OS X) without regard to the number of logical CPU’s on the workstation.

The cost for the Bridge to R on a Windows or Linux server is $599 annually per logical CPU. Running R on a large server is advantageous because you can often access  much more memory than a desktop traditionally offers and servers tend to have faster CPU’s.

If you have licensed WPS from MineQuest and desire a copy of the new version, contact us and we will begin shipping the files on Saturday, March 31st.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.


Introducing WPS Version 3

It’s a new year and version 3 of WPS is out and available. There’s a lot of new items in this release and I think it has been well worth the wait. I personally feel that this is the block buster release that will turn heads in the corporate environment due to so many added features. With version 3, World Programming has made the software more friendly and easier to use, added new procedure support both statistically and in the core language, and added support for more language elements.

I’ve been beta testing WPS 3 for a few months now and so I’ve had a bit of time to try the software out and evaluate it extensively. Below is a list of items and features that really stood out to me in WPS V3. It’s far from being exhaustive but these are probably the most pertinent for many WPS users and SAS customers who are considering WPS as an alternative to SAS.

Workbench Support on Additional Platforms

The Eclipse workbench is available on Unix, Linux, Windows, AIX, Solaris and OS X platforms. The GUI that you’ve been using on Windows is now available on all the platforms except for z/OS. Amazingly, the Eclipse Workbench is identical (and I mean identical) across the platforms. Now when companies and organizations add servers to their data center and the users will have the same identical experience (read no learning curve) on all the platforms.

Apple OS X Support

WPS has been available on Apple’s OS X for a while, but now it has the Eclipse Workbench as the GUI. The WPS implementation on OS X is native, meaning you don’t have to run Parallels or Boot Camp to run WPS on your Mac. WPS is a true 64-bit implementation on OS X. The technical specifications for the Mac OS X implementation is that it needs to be an x86 chip and you will need version 10.5 (Leopard) or higher. WPS V3 works fine on Lion as you can see in the screen shot below.

Click here to view larger image.

About a month or two ago I bought a Mac Mini just to test WPS and to port the Bridge to R over to it. The Mac implementation is a bit prettier than on Windows or Linux in my opinion. It may be the font, but it does look nice as you can see in the image above. In testing WPS on OS X, I was amazed how easy it was to transfer my knowledge of the WPS implementation on Windows over to OS X.

There’s a lot of folks who love their Macs and WPS running natively gives the Mac faithful a SAS language compatible system that takes full advantage of their hardware. Mac users give up nothing over the Windows or Linux versions of WPS and desktop pricing for the Mac is the same as for the other desktop platforms. I’m excited for the university students where there is a preponderance of Macs and I can see the take up of WPS because of the hardware and operating system support.

Linux Support

As mentioned earlier, the workbench is also available on Linux. I have it running on both Ubuntu 10.4 LTS and Fedora 13. As you can see in the screen shots below, WPS looks the same on each of the platforms. One thing to note about the Linux version, it is only available as a server license. But for shops that have a Linux platform, WPS executes exactly like the Windows version in that it provides the end users the same experience as it pertains to the GUI.

Click here to view larger image.

Many shops run Red Hat as their preferred version of Linux and WPS does run just dandy on that platform. I find that many of the larger customers we have do use Red Hat, but I suspect that the smaller businesses use Ubuntu or Fedora instead.

AIX and Solaris

The workbench is also available on these platforms. I don’t have a license or even a machine to run these two implementations on, but I would suspect that the experience is identical to the Apple and Linux versions discussed above.

WPS Link

This is an exciting feature that is being introduced in WPS 3.0. WPS Link is a Client/Server architecture that allows you to remotely execute code on Linux or Unix servers. So for example, you can be running WPS on your Windows or Mac OS X desktop and submit your WPS code over to a WPS Linux server. The log and listing files come back from the remote machine to your desktop. WPS Link is different from SAS/Connect in that this is just a client to execute your programs on the remote box. You can’t upload and download files from the desktop to the server. But, if a lot of your processing takes place on the server and that’s where your data sets reside, this is a nifty solution for processing data on a remote server.

WPS Link is easy to setup from you WPS Desktop. Simply go to the Server Connection tab and select Local. Right mouse click on Local and you get a drop down menu. Select New || Server Connection || Remote SSH.

From that point you get a dialog box where you simply fill out the hostname (or IP Address) the Connection Name (how about “WPS Remote Server”) your user name and the Launch Command. The Launch command was the most difficult part for me. It’s simply the location of the folder where WPSLinks module is located. On my server it is:

/home/minequest/wps-3.0.0.0.19782-private/bin/wpslinks

Make sure you have SSH running on your Linux server and you will be able to remotely start WPS on the server from your desktop!

Submitting code from the desktop to the WPS Server is relatively easy. Simply go to the Submit Icon and you will find a downward facing arrow next to it. You can select whether you want to run the code Locally or on the Server. One of the cool innovations with WPS Link is that you can have multiple servers registered and choose which WPS server you would like to run your code on. Note the options in the drop down menu shown below.

WPS Link comes standard as part of your WPS purchase. So if you license WPS on the desktop, WPS Link will be available to you as part of your desktop license. Also be aware that in this first release, WPS Link requires WPS running on an AIX, Solaris, Linux Servers and Linux on Mainframe System/z. I suspect that you will see Windows Servers being supported in a later release. But for our customers running large volumes of data on Linux or Unix, this feature will be very welcome.

One other note of importance on WPS Link. You can connect from a desktop to a server and server-to-server. You cannot link from desktop-to- desktop.

Procedure Support

In version 3, World Programming has added support for the following statistical procedures.

  • T-Test
  • GLM
  • GLMMOD
  • STDIZE
  • PRINCOMP
  • DISTANCE
  • FACTOR

Database Engine Support

One of the things that SAS does that I really don’t like is the up-charge for database engines. With WPS, you get the database engines as part of the WPS license. You don’t buy these separately. With version 3 of WPS, there is now support for an XML Libname engine and Sybase databases.

If you’re a shop that has a few different databases from different vendors, it would be financially wise to take a look at WPS just for the cost reduction with the included access engines.

Other Enhancements

WPS has now implemented multi-threaded support for PROC MEANS and PROC SUMMARY. It’s always fun to look at the execution time for CPU and Real Time when dealing with multi-threaded procedures. From some of the WPS documentation you can see that the multi-threaded support that is used in MEANS and SUMMARY has made it into procedures that make use of this code such as PROC TTEST and I assume PROC CORR.

Also, for those of you who were unaware, PROC SORT is also multi-threaded and is very fast. If you do a fair amount of sorting in your environment, it would be beneficial to be running on a 64-bit platform and load up with memory. The more memory you have available to WPS, the more data it can store in RAM when sorting and this results in faster sort times.

There is now an import and export wizard for importing files into WPS and writing out WPS data sets to text files. This is pretty big in my opinion. I could never remember the PROC IMPORT statements even though I use them often. Probably everyone uses a code template for importing and exporting but this just makes life easier.

The data set viewer has been enhanced over the previous release. You can view the variables labels instead of just the variable names. You can also show and hide variable columns.

Also supported for the first time is the SYSTASK and WAITFOR commands. I’ve been waiting for these two language statements to be supported for a while. As part of the Bridge to R on Windows there is a module called MPExec. MPExec allows you to run multiple programs in parallel. I’ve had to resort to using other methods in lieu of SYSTASK and WAITFOR to get this to work. Now that these two commands are available, we will update MPExec to use these features instead, thus MPExec will become portable to other platforms.

Performance

Performance has improved in each release of WPS and WPS v3 is no exception. With multi-threaded support for more procedures and better data handling, WPS is a very viable candidate to replace SAS at many organizations. MineQuest has many customers who run large data sets (millions, ten millions and hundred millions) of records through the WPS System and performance is more than adequate. These companies are able to save loads of money over the competing software system.

Expandability

With affordable pricing, customers can use WPS to perform reporting and analytics in areas and for departments that just wasn’t justifiable before. Using WPS in lieu of SAS allows many organizations to expand the analytics platform and run more data through the system. This is because such advantageous pricing allows them to purchase an additional server at great savings over our competitors.

Data Service Providers

We love Data Service Providers! Seriously, we find that DSP’s are some of our best customers. With WPS V3, licensing terms stay pretty much the same. World Programming LTD does not have DSP fees and you are free to use WPS to service your customers by providing them with data sets, reports and analytics. For any organization that is paying DSP fees, we can dramatically reduce your license fees. Your customer is your customer.

Caveats

Before you download the latest release of WPS, there is one caveat to keep in mind for upgrading. If you are using batch command files to run your jobs, you need to modify the name of the WPS executable in your .bat or .cmd file. The program name you want to use is WPS.EXE (as opposed to the former name which was WPSI.EXE).

Also in WPS V3 the WPD dataset has changed and is faster and more robust. You can still read WPD V2 datasets but V3 datasets are the default in this release.

Finally, you will need a new license key to be able to install V3. Don’t expect to simply download it and run it using the V2 key as you have been able to do previously. If you are a MineQuest customer, contact us if you want to upgrade to V3 and we will facilitate getting you a license key.

Evaluations

MineQuest is offering free 30 day evaluations of WPS V3. You can contact us at (614) 457-3714 to request an evaluation or by email. If you prefer to request your evaluation by email, send a request to info@minequest.com.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and a reseller of WPS in North America.

Timing WPS in Ubuntu and Fedora VM’s

In my previous blog post, I touched on the difference in execution time when running WPS in a VirtualBox VM between Fedora 13 and the Ubuntu 10.4 distributions. I expected that there would be a difference between the two, but I was a bit surprised to see how much faster Ubuntu is than Fedora.

I wrote a fairly generic SAS program that creates some data (29 variables) and 100K or one million records. I ran those records through PROC MEANS, SORT, CORR, UNIVARIATE, another data step, a PROC APPEND, and a PROC FREQ. I ran that program five times on Fedora and Ubuntu for 100,000 records and one million records respectively, and I made sure the VMS’s were exactly the same.

The characteristics for each VM are:

  1. Two Logical CPU’s.
  2. 3072 MB of RAM.
  3. SATA Port Controller that uses host I/O cache.
  4. 64MB of Video RAM.
  5. Each VM is 32GB in size.

The table below is the outcome based on running the program five times for each sized data set. I took the lowest time for the five runs and used that as the basis for comparison. I also made sure that no other programs were running on either the host PC or the client VM’s.

100,000 OBS

1,000,000 OBS

Fedora 13 x32
Real Time

00:22.36

04:41.23

CPU Time

00:22:32

04:17.46

Ubuntu 10.4 x32
Real Time

00:17.68

03:59.72

CPU Time

00:17.67

03:32.55

Times above are in minutes:seconds.ms

What is interesting when I look at the log is that the times for the PROCS that typically just read data and create a report, like MEANS, FREQ and CORR, the execution times between Ubuntu and Fedora are very close. When I look at the data steps and procedures like SORT and APPEND, ones that read and write data, Ubuntu just crushes Fedora. Writing data in Ubuntu is much more efficient in a VM than Fedora.

About the author: Phil Rack is President of MineQuest, LLC. and has been a SAS language developer for more than 25 years. MineQuest provides WPS and SAS consulting and contract programming services and is a reseller of WPS in North America.