Despite its popularity among the research community, R still seems inflexible in fully exploiting the latest developments in computer software and hardware. As there are no inherent constructs for parallelizing computations, it is up to the developer to adapt the R code, in order to take advantage of the resources available by multi-core CPUs. Moreover, when the data computed are too big to fit in main memory, no simple solution is considered standard. Packages that offer workarounds are indeed available, but their use by non-expert R users is rarely considered straightforward. These are real issues faced by researchers, whose needs for processing collected data continually increase both in computational demand and in size.

Working with large datasets in R can be cumbersome because of the need to keep objects in physical memory. The need to keep whole objects in memory becomes a challenging task to those who might want to work interactively with large datasets. Several packages attempt to overcome problems when accessing big volumes of data. The bigmem package (Kane et al. 2013) is designed to handle massive data sets that are not larger than the available RAM. It overcomes the restriction of R using matrices or data frames that, even though they fit in RAM, no space is available to handle the overhead of working with them. Furthermore, it extends and augments the R statistical programming environment, thus enabling more powerful parallel analyses and data mining of massive data sets, although it is restricted to the available RAM size (even though some options for connecting its objects with file-backed mappings can be implemented).

The Programming with Big Data in R (pbdR) project (Ostrouchov et al. 2012) seeks to elevate the R language to supercomputers. Most of its functionalities revolve around parallelization features, yet the pbdDMAT package (Schmidt et al. 2012) of the project offers an implicitly parallel system for doing distributed matrix computation in R. The bigmem and pbdDMAT packages are useful solutions, but do not always provide the level of flexibility needed in handling complex constructs because of the fact that they rely on their own constructs to handle big data.

Nowadays, the dplyr package (Wickham and Francois 2015) has become very popular for data manipulation, providing a repertoire of functions for accessing data stored in databases. Coupled with packages dedicated for specific database implementations, such as RPostgreSQL (Conway et al. 2013), it offers the possibility to write R scripts that access the underlying databases from within the R environment, but with the look and feel of relational data manipulation. This approach is more attractive for a large scale implementation, such as the one designed for the LifeWatchGreece portal.

As far as parallelization is concerned, a multitude of packages have emerged, such as snow (Tierney et al. 2015), multicore (Urbanek 2009) and parallel. Due to the complexity of tasks required in vegan, as well as in other packages for the LifeWatchGreece project, we found the existing approaches for parallel computing rather restrictive. Our decision in this project was to go into the low-level whenever necessary and implement custom parallel solutions which can provide more flexibility. To assist us in this process, packages that provide interfaces to MPI (Message-Passing Interface) for R are proved really valuable. Rmpi (Yu 2002) is one of the most popular solutions that can port low level MPI functions into R, abstracting the complexities of writing C or Fortran code. The more recent pbdR project (Ostrouchov et al. 2012) also offers such a wrapper through the pbdMPI library, which is intended for Single Program Multiple Data (SPMD) programming with big data.

After considering the benefits offered, we decided to adopt pbdMPI (Chen et al. 2012, as our primary package for parallelization within LifeWatchGreece, and couple it with other solutions for parallelization or optimization of code, whenever necessary.

Summarizing, the work conducted in the context of the project aims at applying optimization techniques for data on two different levels of abstraction described in detail in the project description. This is done by:

1) Using the pbdMPI package to implement Single Program Multiple Data (SPMD) parallelization on primitive mathematical operations, allowing for interplay with functions of the vegan package.

2) Using the dplyr and RPostgreSQL packages in order to offer secondary storage solutions whenever memory demands exceed available RAM resources (memory exhaustion)

Option (1) may be employed in conjugation with option (2) to address memory exhaustion issues. While, for speed-up and job segmentation issues we only use option (1).

We finally evaluate our optimization results using two test case scenarios with real data obtained from environmental ecologists in the standard file formats, commonly utilized in the field. We also demonstrate results and visualization outcomes obtained through the graphical user interface available through the LifeWatchGreece portal.

In this section we describe the RvLab web application and how a user can access it through LifeWatchGreece portal. The following screenshots illustrate the web pages a user goes through when using RvLab. After registering and logging in at portal's landing page (Fig. 6 - left image), the user comes to portal's Home Page (Fig. 6- right image) where direct access to RvLab (and other virtual laboratories) is available.

The main interface of RvLab is comprised by four panels (Fig. 7). The Workspace panel (top left) allows users to upload .csv files that can be used as input and monitor their available storage space. The Functions panel (right) allows users to select a statistical function, configure it and submit a new job to run for execution. The Jobs panel (bottom left) allows the user to keep track of his submitted jobs, monitor their status, view the results or delete the ones that are not needed anymore. The Help panel contains information about RvLab and its usage policies. Example datasets (Fig. 8) can also be found there, if someone wants to try out RvLab without using his own input files.

Once a job is completed, the user can view the results page by clicking on the Job ID link (Fig. 9 - left image). The result page (Fig. 9 - right image) may contain textual information (i.e. "Significance values and taxonomic indices values"), graphical results (i.e. "Taxonomic indices funnel plots") or file results. Result files can be downloaded or added to the workspace for downstream analyses and further utilization in additional RvLab functions.

Screen shots showing examples of the graphical results generated by RvLab can be seen in Fig. 10. The interactive SUMMARIZEplot function, utilizing JavaScript Data Drive Documents (D3.js) and HTML (Fig. 10a) allows users to observe distributions of species per station in bar charts, as well as pie charts, generated from the most abundant species found in each station. The results of regression analysis, i.e. the linear model relationship between environmental factors measurements during sampling, such as maximum depth and bathythemetry, are shown in Fig. 10b. Outputs of Analysis of Variance (Anova) (Fig. 10c, left image) and Analysis of similarities (Anosim) (Fig. 10c, right image) can be used to provide statistical significance for the relationship between environmental factors selected by the user. Finally, Principal Component Analysis (PCA) showing an ordination (grouping) of stations in a lower dimensional space, given their species abundance, is presented in Fig. 10d. The user is given flexibility in assigning colour-codes according to selected factors, such as the location of the sampling stations.

RvLab is developed in Hypertext Preprocessor (PHP) and has been integrated in the LifeWatchGreece portal allowing registration to the common user database used for all virtual laboratories available via the portal. This integration utilizes some background Hypertext Transfer Protocol (HTTP) communication between the portal's core and RvLab's web application that involves exchanging information regarding credentials and access control privileges. Moreover, cron jobs have been deployed to ensure that policies are enforced and job status is updated regularly through Asynchronous JavaScript and XML (AJAX) calls. Although job execution takes place on a cluster, RvLab has direct access to all job folders by mounting (through SSH) the rellevant cluster directories to the web server. RvLab uses Portable Batch System (PBS) scripts to schedule each job for execution on the cluster.

Mobile RvLab application (mobRvLab)

The RvLab mobile application (mobRvLab) has been developed by utilizing Unity3D Platform and C# scripting language. The application is available for android and ios platforms and functions on a dynamical and autonomous basis. It receives data in json format from the LifeWatchGreece portal in real time by exploiting the appropriate web services that have been developed. Whenever data are required, a secure proper communication channel is established between the device and the portal.

As previously mentioned, the RvLab is available after registration and login to the LifeWatchGreece portal. This is a pre-requisite in order the user to access mobRvLab by utilizing the same account credentials. MobRvLab employs the same functionalities adopted by the RvLab. The mobRvLab main page includes general information about the application and also displays three main tabs: “Functions”, “Files” and “Jobs”. The “Functions” tab shows all available functions, same as in the portal version. The user can select which function to run, including parallel implementations of available functions. In the “Jobs” tab the user can view a jobs log file and keep track of each job status. The jobs are presented and ordered by date. Each job can be selected for viewing results or for deletion. The “Files” tab displays the user workspace and allows file management by uploading or deletion of data files. In principle, the mobRvLab provides a direct link, via mobile access, to the RvLab user account created in the LifeWatchGreece portal; jobs are executed in the LifeWatchGreece cluster and not locally, hence allowing user access to the high performance computational resources of the cluster via mobile appplication. The user can choose to submit jobs and view results from either of the virtual laboratories, benefiting by the usability and flexibility of RvLab in data analysis, as well as by the results acquisition. Fig. 11 displays the data exchange web services and overall functionalities of the mobRvLab.

RvLab mobile application is available for download at https://portal.lifewatchgreece.eu/mobile_apps

The desired target audience is the average R user, without requiring expertise and prior training in the field of algorithm optimization and parallelization. However, basic statistical knowledge for using the analytical routines and for the interpretation of results is required.

We conducted a series of experiments to evaluate the generic methodology described in previous sections, in order to study (i) the speedup achieved with the new functions when exploiting the resources of a multiprocessor environment and (ii) to identify optimal allocation of resources given the size of the input data. The reported times are the average of 3 runs for each configuration. They were conducted in a controlled environment, where all external access was blocked; despite being an idealized environment, this setting allowed us to reach consistent conclusions about the behaviour of the functions.

Experimental setup

Our experiments varied between functions requiring large amounts of memory to handle input data, as well as functions performing computationally intensive tasks. For the former category, datasets of increasing size have been used as input. The taxa2dist function, as well as the combined taxa2dist+taxondive, fall into this category, as they rely on the computation of a distance matrix that can become significantly big depending on the initial dataset. For the latter category, we varied the number of computations, namely the number of permutations that need to be executed before producing the result. For each of these cases, we measured the execution time of the parallel version of our functions when allocating a different number of processors and compared these times to the time needed to run the serial function, i.e., the version provided by the vegan package. Note that in certain cases the serial version could not be executed at all, e.g., when the available memory was not enough to handle computations.

All experiments were conducted on a cluster involving 10 Intel Xeon CPU E5-2667 2.9GHz cores with a total of 384Gb RAM. We measured times by allocating 1, 2, 4, 6, 8 or 10 CPUs to the parallel functions, in order to study their behaviour. The results are presented through a series of diagrams which are based on a comparative analysis; absolute timing is mentioned only for verification. Although the current version of the RvLab portal runs over a cluster having a different configuration, the messages conveyed by our experimental analysis are still valid, as we are not interested in the absolute times measured, but rather on the speed up that can be achieved.

Experiment 1. taxa2dist

The diagram in Fig. 12 shows the speedup achieved by running the parallel implementation of the taxa2dist function in comparison to the serial one, for datasets having approximately 1,700 lines of species (small), 16,900 lines (medium) and 42,300 lines (large), respectively. Specifically, the black line at 1x sets the boundary in speedup, as it denotes the execution of the serial program itself. Obviously, no matter how many processors we assign to this program, the execution time stays the same since no parallelization of operations is possible within its code. For example, with the small dataset as input, the execution time was 5.1 sec on average with 1 CPU and 5.0 sec with 10 CPUs; for the large dataset, these times where 21.2 min and 21.1 min, respectively. We can safely conclude that the number of processors do not significantly affect the serial program.

Any values below the 1x boundary denote execution times proportially higher to the serial ones, whereas values above the boundary denote how many times faster the execution was found to be. As mentioned above, we measured our parallel program having different allocations of processors, in order to check at which setting the maximum speedup is achieved. In other words, the diagram shows the behaviour (i.e., speedup) of the parallel program in comparison to the serial one (vertical axis) given two parameters, the number of processors assigned to the program (horizontal axis) and the size of the input data (colored lines).

A first observation that can be made is that for small datasets the serial version is somewhat faster than the parallel one. This is displayed by the points lying below the 1x threshold. Indicatively, while the serial version required 5.1 sec on average to run the small dataset as mentioned before, the parallel version with 2 CPUs needed 6.2 sec and with 4 CPUs 7.3 sec. For such small datasets, the differences in absolute times are very small, therefore any delays introduced due to the initialization of the cluster seem to play an important role.

The situation changes as the input dataset becomes bigger, where the parallel version is faster in many cases. For example, with 4 CPUs the parallel version required 2.14 min for the medium dataset and 14.5 min for the large one, whereas the serial version needed 3.4 min and 21.2 min, respectively.

Nevertheless, speedup decreases as we add more CPUs. This observation verifies a conclusion already known in the ICT community: the parallel solution is not panacea and proper allocation of resources needs to be made not only based on their availability, but also on other parameters, such as the size of the input in our case. It seems that the cost of communicating data between processors becomes considerable as we add more processors.

It should be noted here that none of the aforementioned functions was able to operate with datasets of larger size, due to memory overflow. For such datasets, one needs to revert to the PostgreSQL variation that stores data on secondary memory, which inevitably takes much longer times to execute.

Experiment 2. taxa2dist+taxondive

Since the output of taxa2dist is very often used as input to the taxondive function, we implemented the parallel version that combines the two, as described in a previous section. The taxondive computes indices of taxonomic diversity and distinctness, which are averaged taxonomic distances among species or individuals in the community (Clarke and Warwick 1998). This function has the added advantage that it manages to overcome memory barriers: due to the algorithmic structure of these functions, one can break the huge bi-products of taxa2dist into portions and complete the execution in an incremental manner.

Fig. 13 shows the results obtained, revealing impressive speedups. For instance, while the vegan functions require 4.1 hours to execute the large dataset, the parallel version, when exploiting all 10 CPUs of the cluster, manages to reduce this time to 20 min on average. Even for the smallest dataset, where the serial version needs 20.5 sec, the parallel version runs faster, takeing 6.6 sec with 4 CPUs and 4.4 sec with 10 CPUs. One reason for obtaining these results is the more efficient redefinition of the functions made when rewriting the code: we reduced the work needed by avoiding duplicate calculations of operations that exist in the vegan (serial) code of both the taxa2dist and taxondive functions, and we utilized the same structures, wherever possible.

What is even more impressive is that we even managed to run datasets of much larger sizes that cannot be executed otherwise: indicatively, we managed to complete the calculation of the taxa2dist+taxondive function for an input dataset having 168,931 lines of species in 6.6 hours when allocating 10CPUs.

Experiment 3. anosim

The next set of experiments concentrated on the anosim function. The preliminary profiling tests we performed showed that the main issues one needs to tackle are concentrated on the consumption of time and not on memory overflow. This time consumption is strongly correlated with the number of “permutations” we introduce to the function.

As we see in Fig. 14, as we provide more processors to the parallel version, the execution becomes faster. Another conclusion is that the number of permutations are counter analogous to the algorithm’s running time, but we obtain higher speed up as we increase the number of allocated processors for the increased amount of permutations. This experiment was conducted with a 9 Mb dataset as input .

The above diagram (Fig. 14) suggests that the parallel version with 10 processors completes up to 24 times faster the calculations in comparison to the serial anosim; more specifically, it needs only 1.1 sec as compared to 27 sec. Also, every parallel version is faster than the serial one, apart from the case where the available processors are less than 2. Also, when we have 10⁷ – 1 permutations (orange broken line), the parallel version is faster than the serial.

In Fig. 15, for larger datasets, we notice that most parallel versions improve the computational time, except for the case with 10³ – 1 permutations and with the number of processors being less than 8. The parallel version with 10⁴ – 1 permutations has neglectable difference with respect to the serial one up until 8 processors. The maximum optimization we achieve is 13 times with 10 processors, from 5.8 min to 28.9 sec. This experiment used a 230 Mb dataset.

Experiment 4. adonis

Our profiling tests with the adonis function revealed that no memory issues are likely to be met in this case; instead, the main bottleneck that should concern us is the scaling of computation effort. The dominant factor of time consumption is the number of permutations set as input to the function, therefore we broke the task of computing them into chunks to be assigned to each of the available processors.

Fig. 16 shows the speed up gains of the parallel versions. It is evident that the number of processors assigned, directly affects the speed up in each version. Also the number of permutations are counter analogous to the total computation time. As we increase the number of available processors the computational time is reduced up to 12 times compared to the serial version: for example, the serial version requires 8.1 min and the parallel with 10 processors 40 seconds for 10⁷ – 1 number of permutations. Another interesting observation from the diagram is that the execution of parallel computations with a small number of permutations (e.g., less than 10⁴) is not faster than the serial on (black line with circles) despite the resources allocated.

Experiment 5. simper

Similarly to the anosim and adonis functions, the simper function also relies on the number of computations requested and not on the available memory. As before, we break the permutations into chunks which are assigned to the available processors.

Fig. 17 shows the speed up gains implementing the parallel versions. The number of processors is analogous to the speed up in each version. Also the number of permutations are counter analogous to the total computation time, but as we increase the number of available processors the computation time falls up to 5 times compared to the serial version. For instance, the serial version needs 11 sec and the parallel with 10 processors 2 sec, for 10⁴ – 1 number of permutations.

As we already observed in the experiments with the previous functions, we notice again here that when the number of permutations is small parallelization is not always beneficial: in fact, increasing the number of processors may hinder performance, due to the communication costs involved.

Experiment 6. mantel

Similarly to the previous functions, the mantel function relies on recurrent computations without producing any memory leakage or overflow. These recurrent computations depend on the number of permutation.

For further investigation we performed two classes of experiments, one for a small input dataset, 9 Mb, and one for a large input dataset, 230 Mb. The diagrams in Figs 18, 19 show that the behaviour of both experiments follow the same principles as the experiments for the previous functions.

In Fig. 18, for the small dataset, as long as we assign less than 8 processors all parallel efforts are less efficient than the serial mantel. Beyond this threshold, we notice a speed up increase and the parallel version outperforms the serial one (black circled line). Finally, using all 10 processors we reach the maximum speed up.

In Fig. 19, for large dataset, all parallel versions improve the computational time. The maximum speed up gain is 15 times, for the parallel version with 10 processors, namely 1.2 sec instead of 20.2 sec for the serial mantel.

Experiment 7. bioenv

Bioenv also depends on repeated computations. The main difference with the previous approach is that instead of breaking the permutation’s linked computations into chunks, we break the ncol’s linked computations. This modification came along with a limitation. The limitation suggests that it is not useful to use more processors than the number of ncol.

In Fig. 20 we report results from an experiment with a dataset with 1 ≤ ncol ≤ 8. As long as we increase both the ncol parameter and the number of processors the speed up increased. The bioenv serial version is faster than all parallel versions for ncol = 1. After that, the limitation verified from the results, i.e., the experiment with ncol = 4 reaches 2 times speed up, for 4 processors, with respect to the serial bioenv, but if we increase further the number of processors the speed up decreases and converges to 0. A general conclusion drawn from Fig. 20 is that we gain the maximum speed up when the number of available processors becomes equal with the number of ncol. Consequently, the maximum speed up we can gain is about 4 times for the parallel version with 8 processors when ncol = 8, namely we decrease the consumption time from 4.9 sec (serial version) to 1.2 sec (8 processors for ncol = 8).

Future implementations of the RvLab will include additional functions that are important for environmental ecology, biodiversity, fisheries and modelling. The RvLab has the possibility to incorporate a variety of functions and R packages, apart from the ones already implemented, should the user contact the development team with a relevant request. Moreover, we are currently also investigating issues of assigning jobs as a function of available resources in order to ensure optimal core distribution and function execution for all jobs submitted to RvLab.

The RvLab is a very useful and powerful tool, both for users who are already familiar with R (and some of its functions) but also for students and/or scientists who are in favour of open source software and would like to dedicate some time to get familiar with its functions, without having to go through the steep command line R learning curve.

When compared with online virtual environments, such as the "Multivariate AnalysiS Applications for Microbial Ecology (MASAME)" suite, apart from the intrinsic similarities between the two platforms, it is obvious that the RvLab can implement a plethora of functions, some of which are parallelized. Thus, the user can benefit from the availability of newly designed functions if the dataset to be analysed requires their implementation.

The accessibility of RvLab is also one of its major advantages; apart from being part of the LifeWatchGreece Infrastructure, it is also a part of the LifeWatch Marine Virtual Research Environment (VRE). The LifeWatch Marine Virtual Research Environment (VRE) portal is bringing together several marine resources, databases, data systems, web services, tools, etc. into one marine virtual research environment, allowing researchers to retrieve and access a great variety of data resources and tools.

In addition, the RvLab is an interactive virtual laboratory; should the user require other types of functions, these can be added in the "laboratory" and become available online in a short time. Therefore, the more users are logging in the portal and using it for their analyses, the more they can improve the RvLab, given the enormous possibilities of its programming language.

The source code for the functions is available for download at the RvLab.