SciServer Compute uses Jupyter Notebooks running within server-side Docker containers attached to big data collections to bring advanced analysis to big data “in the cloud.” Compute extends the popular CasJobs and SkyServer systems with server-side computational capabilities and very large scratch storage space, and further extends their functions to a range of other scientific disciplines.
CasJobs has revolutionized database querying of big datasets like SDSS catalog data. Yet until now users had to download their data from the cloud to perform additional computational analysis close to the data, in a networking sense. SciServer Compute is a web app that runs in any modern browser, allowing users process their dataset in a cloud-based research environment.
Compute uses the SciServer Web Services API to access CasJobs datasets and MyDB. Intermediate results are stored in FileScratch, a large scratch space (hundreds of TB). Users can store their results in a permanent allocation on SciDrive, a Dropbox-like system for sharing and publishing files. Users can develop Notebooks using SciScript libraries, standard scientific Python and R libraries, and/or their own Python or R functions, and execute their Notebooks on a server next to the data. Implementation of scripting in Matlab and other languages is in development.
SciScript is the main programmatic access point for SciServer resources. SciScript technology allows users to write scripts in Python, R, or Matlab that offer access to all tools within existing programs.
SciScript is an integral component of Compute.