Hosted Datasets
SciServer hosts more than two Petabytes of scientific data in a variety of disciplines, including astronomy and fluid mechanics. Datasets come in two types:
Public Data Volumes can be accessed from SciServer Compute by mounting them onto a new container at the time you create the container.
Databases can be accessed from SciServer Compute by querying them with commands from the SkyServer.CasJobs module. See the SciServer module API Documentation to learn how to use these commands.
The sections below describe each of the datasets available through SciServer. Click on one of the gray boxes to see full information about that dataset.
SciServer makes a number of datasets directly available in SciServer Compute in the form of Public Data Volumes. To use these data volumes in your research or education activities, you will need to mount them to a virtual container at the time you create that container. See the instructions on How to create a new container to learn how to mount a Public Data Volume.
File-based data for the complete imaging of the Sloan Digital Sky Survey (SDSS) is now available for analysis through SciServer. The files come from the SDSS’s Data Release 9 (DR9), covering a footprint of 14,555 square degrees shown by the white and gray areas in this figure, which is further described at the Scope of DR9 page.
The images are corrected frames, which are the final step of the SDSS imaging pipeline and the starting point for generating the color images found in the SkyServer Navigate tool.
The images can be accessed via SciServer Compute using the standard file access python tools.
We are now in the process of acquiring the full contents of the SDSS Science Archive Server, which will provide similar Compute-based access to all SDSS data released since 2008.
This volume contains all the raw and processed file-based data from Data Release 7 (DR7) of the Sloan Digital Sky Survey (SDSS). The raw and pipeline-processed imaging and spectroscopic data products are available here (mostly) in binary FITS format.
The data on the SDSS-DAS volume can be accessed via SciServer Compute using the standard file access python tools. A copy of this data is also accessible via the SDSS DAS website, and the catalog version of this data is available from the SDSS DR7 SkyServer.
Montage is a set of tools, available as either command-line programs or Python modules, that allow users to reproject images, analyze and correct background differences between images, and create mosaics. Montage includes tools for searching image metadata for several datasets and downloading the images found. On the back end, it provides visualization of grayscale or 3-color composite images, catalog and image coverage metadata overlays, etc. Catalogs and other data can be found and retrieved using the included Astroquery package.
For more on how to use Montage with online Python notebooks, see the Montage Jupyter Notebooks page.
Recount2 provides processed and summarized expression data for over 70,000 human RNA-seq samples from the Sequence Read Archive (SRA), The Cancer Genome Atlas (TCGA), and The Genotype-Tissue Expression (GTEx) project (https://doi.org/10.1038/nbt.3838).
The associated Bioconductor package provides a convenient API for querying, downloading, and analyzing the data. Each processed study consists of meta- and phenotype data, the expression levels of genes and their underlying exons and splice junctions, and corresponding genomic annotation. By taking care of several preprocessing steps and combining many datasets into one easily-accessible website, we make finding and analyzing RNA-seq data considerably more straightforward.
SciServer hosts numerical model output of high-resolution Ocean General Circulation Models (GCMs) set up and run by the research group of Prof. Thomas W. N. Haine (Johns Hopkins University – Department of Earth and Planetary Sciences).These models allow users to trace the physical evolution of ocean currents across orders of magnitude in space and time, and to quickly analyze important aspects of model events in conjunction with observational data.
The goal of the SciServer Ocean Modeling User Case is to build a collaborative sharing environment where users can access and process high-resolution datasets. The analysis of these large datasets is often restricted by limited computational resources, so we have developed OceanSpy, a python package that facilitates extracting information from model output fields. SciServer users can either download subsets of data on their own machines, or run our tools online and store post-processing files on our servers.
Available Datasets
- EGshelfIIseas2km_ERAI: High-resolution (~2km) numerical simulation covering the east Greenland shelf (EGshelf), and the Iceland and Irminger Seas (IIseas). Surface forcing based on the global atmospheric reanalysis ERA-Interim (ERAI). Citation: Almansi et al., 2017
- EGshelfIIseas2km_ASR: High-resolution (~2km) numerical simulation covering the east Greenland shelf (EGshelf),and the Iceland and Irminger Seas (IIseas). Surface forcing based on the regional atmospheric Arctic System Reanalysis (ASR). Citation: Almansi et al., 2017
- EGshelfSJsec500m: Very high-resolution (500m) numerical simulation covering the east Greenland shelf (EGshelf) and the Spill Jet section (SJsec). Both hydrostatic and non-hydrostatic setup are available. Citation: Magaldi and Haine, 2015
References
- Almansi, M., T.W. Haine, R.S. Pickart, M.G. Magaldi, R. Gelderloos, and D. Mastropole, 2017: High-Frequency Variability in the Circulation and Hydrography of the Denmark Strait Overflow from a High-Resolution Numerical Model. J. Phys. Oceanogr., 47, 2999–3013, https://doi.org/10.1175/JPO-D-17-0129.1.
- Marcello G. Magaldi, Thomas W.N. Haine, Hydrostatic and non-hydrostatic simulations of dense waters cascading off a shelf: The East Greenland case, Deep Sea Research Part I: Oceanographic Research Papers, Volume 96, 2015, Pages 89-104, ISSN 0967-0637, https://doi.org/10.1016/j.dsr.2014.10.008.
SciServer offers a number of catalog datasets in the form of online SQL databases. These databases can be queried directly through CasJobs. From the CasJobs Query page, select a dataset from the Context menu to choose what to query. Enter your query in the query box, then click Quick (to return results to your browser, limited to 90 seconds) or Submit (to write results into your MyDB.All these databases can also be queried from within SciServer Compute by using the SciServer API libraries. You do not have to mount anything to query these databases, and the libraries will work from within any container.
Simply import these libraries into your scripts, then make use of them. Information about how to use the libraries, and what commands are available, can be found in the SciServer API libraries documentation. Examples of how to query databases are available in the SciServer Example Notebooks.
The Sloan Digital Sky Survey (SDSS) is an ongoing project to make a map of the Universe. It has observed images of hundreds of millions of stars and galaxies, and spectra for more than four million.The entire history of data releases from the Sloan Digital Sky Survey (SDSS) can be queried through SciServer Compute by importing and using the SciServer libraries.
The SDSS releases its data in discrete batches through sequentially-numbered Data Releases. The most recent is Data Release 15 (DR15); for new studies with SDSS data, begin with DR15. To enable replication and extension of prior studies, we make all previous data releases (DR1 through DR14) available as well. We also provide two additional datasets: Stripe82 contains all photometric data for the repeat observations of the SDSS supernova survey, while RunsDB contains all photometric data for all SDSS observations, including overlap areas.
To query one of the SDSS databases using CasJobs, select its name from the Context menu just above the query window. The contexts holding SDSS data are DR15, DR14, etc.
To access one of the SDSS databases using SciServer Compute, specify its name as the context in the appropriate place in your SciServer.CasJobs.executeQuery(sql, context) or SciServer.CasJobs.submitQuery(sql, context) commands: for example, as
SciServer.CasJobs.executeQuery("select top 10 * from photoobj", context="DR15")
Gaia is a European Space Agency mission to find distances and properties of more than one billion stars in our Milky Way Galaxy.SciServer hosts the complete catalog data for Gaia Data Release 2 (Gaia DR2).
To query the Gaia DR2 catalog using CasJobs, select GaiaDR2 from the Context menu just above the query window. To access Gaia DR2 data using SciServer Compute, specify its name as the context in the appropriate place in your SciServer.CasJobs.executeQuery(sql, context) or SciServer.CasJobs.submitQuery(sql, context) commands: for example, as
SciServer.CasJobs.executeQuery("select top 10 * from gaia_source", context="GaiaDR2")
The Galaxy Evolution Explorer (GALEX) is an ultraviolet space telescope that operated from 2003 to 2012. During that time, it observed hundreds of thousands of galaxies, helping to determine distances and star formation rates throughout the universe. SciServer offers access to all GALEX releases up to and including its final complete dataset, GALEX Release 6 (GR6). The GALEX data releases are referred to in SciServer as GALEXGR6, GALEXGR5, etc.To query one of the GALEX databases using CasJobs, select its name from the Context menu just above the query window.To access one of the GALEX databases using SciServer Compute, specify its name as the context in the appropriate place in your SciServer.CasJobs.executeQuery(sql, context) or SciServer.CasJobs.submitQuery(sql, context) commands: for example, as
SciServer.CasJobs.executeQuery("select top 10 ra, dec from acsData", context="GalexGR6")
The Two Micron All-Sky Survey (2MASS) is an all-sky survey at infrared wavelengths. SciServer offers access to the Point Source Catalog of the 2MASS All-Sky Data Release, enabling studies of populations of stars and other resolved objects in the Milky Way.To query this catalog using CasJobs, select 2MASS from the Context menu just above the query window. To access it using SciServer Compute, specify its name as the context in the appropriate place in your SciServer.CasJobs.executeQuery(sql, context) or SciServer.CasJobs.submitQuery(sql, context) commands: for example, asSciServer.CasJobs.executeQuery("select top 10 * from PhotoObjAll", context="2MASS")
The Two-degree-Field (2DF) Galaxy Redshift Survey is an all-sky survey at visible wavelengths, with the goal of understanding the large-scale structure of galaxies.To query this catalog using CasJobs, select 2DF from the Context menu just above the query window. To access it using SciServer Compute, specify its name as the context in the appropriate place in your SciServer.CasJobs.executeQuery(sql, context) or SciServer.CasJobs.submitQuery(sql, context) commands: for example, asSciServer.CasJobs.executeQuery("select top 10 * from PhotoObjAll", context="2DF")
The Faint Images of the Sky at Twenty cm (FIRST) is an all-sky survey at radio wavelengths.To query this catalog using CasJobs, select FIRST from the Context menu just above the query window. To access it using SciServer Compute, specify its name as the context in the appropriate place in your SciServer.CasJobs.executeQuery(sql, context) or SciServer.CasJobs.submitQuery(sql, context) commands: for example, asSciServer.CasJobs.executeQuery("select top 10 * from PhotoObjAll", context="FIRST")
The Minor Planet Center Orbit (MPCORB) database contains orbital parameters for more than 700,000 asteroids from the International Astronomical Union’s Minor Planet Center. The dataset available through SciServer is a snapshot of the database as of August 17, 2017.To query this catalog using CasJobs, select MPCORB from the Context menu just above the query window. To access it using SciServer Compute, specify its name as the context in the appropriate place in your SciServer.CasJobs.executeQuery(sql, context) or SciServer.CasJobs.submitQuery(sql, context) commands: for example, asSciServer.CasJobs.executeQuery("select top 10 * from mpcorb", context="MPCORB")
SciServer also offers astronomical cross-matching services through the SkyQuery web service. A list of all datasets you can cross-match among is on the SkyQuery website. You can use SkyQuery to match objects between these big datasets or against data that you upload.See the SkyQuery website for more information.