The Universe in Data Form

Articles

The (Raw) State of Space

I was looking at a moon. Sitting in my lounge just outside of Central London, I felt a totally unjustified sense of discovery. You see this wasn’t The Moon. This was another far more alien (albeit still greyly familiar) moon. This was Ariel. And she was spectacular.

 

Several hours earlier I had become intrigued in the concept of raw historical space imagery, thanks to this fantastic post (and subseqent comments) on Reddit by user mmokkp. In that post, mmokkp had created a seemingly unparalleled high-resolution image of Callisto through a process of downloading, converting, cleaning, and finally stitching multiple raw images — taken as part of the Voyager mission — together into a seamless mosaic. The resultant image is breathtaking, but the process involved fascinated me even more.

The simple fact that this raw data is freely available was something that hadn’t even crossed my mind. On searching further, I quickly realised that just below the surface of the curated space imagery (which has repeatedly and seemingly without-fail fascinated me for as long as I can remember), lies an almost impossibly vast body of raw image, telemetry, and sensor data. Gigabytes upon gigabytes, spanning decades of space exploration, that I’d never once thought to seek out.

A few hastily-written Python scripts later, and a small subset of that data — the Ariel images taken by Voyager 2 nearly three decades earlier — was ready to explore. Advancing from image to image, a small, out-of-focus grey blob danced around my laptop screen amidst the noisy, often-corrupted blackness. Then suddenly it was just there. Downloaded, converted, and clearly definable. I was looking at a moon.

Animation showing each of the downloaded Ariel Images in turn at full resolution. Raw Images Credit: NASA/JPL.

Whilst a step-by-step, code-focussed look at how I obtained the Ariel images will be the subject of an upcoming article, here I want to discuss two things: first, the major difference between the images described above and the space images we are more accustomed to seeing on the web; and second, where exactly you can go to find such raw data.


The Reasons for (and against) Raw Imagery

The last ten years of my life have been spent in the Visual Effects industry where, coincidentally, raw data also plays an exceptionally important role. Broadly speaking, in Visual Effects there are two main reasons for using raw image files over other formats :

  1. Using a raw image gives an artist access to a larger range of visual information, which in turn improves their ability to manipulate that image accurately.

    In VFX this often meant ensuring that detail was maintained in the darkest and brightest sections of an image; be it a photo captured on a low-lit set, or a computer-generated image of sunlight gleaming off of a robot’s shoulder. In both cases capturing in a raw format gives the final artist the greatest potential for creative change.

Animation demonstrating the difference in the highlights of a non-raw image (left) vs. raw image (right) when altering the exposure values. Using the raw image format preserves cloud and tree detail, whilst the non-raw format does not

  1. Raw images often contain extra non-visual information that you potentially lose once you convert to a non-raw format.

    This extra information (often termed ‘metadata’) can range from details on a camera’s lens and shooting settings, to time and location data — no-doubt familiar to those of you who have viewed or edited EXIF information for your own photos.

    Depending on the raw format used, there might be few limitations on what types of metadata the resultant files can contain, or on how that extra information might be used by the final artist.

However, there is also a strong case for not always using raw files in Visual Effects work. Larger file-size and accessibility issues aside, perhaps the biggest reason of all is one of presentation — oftentimes the extra data provided by a raw format is simply unnecessary in a final image, and presenting the raw, unedited state of the data might actually detract from what you are trying to convey.

Perhaps unsurprisingly, the same arguments apply when it comes to space imagery.

Just as the images you see in the cinema have been converted from their raw format and manipulated to elicit the director’s desired emotional response, the space images we commonly see on the internet are indeed also manipulated; oftentimes constructed from multiple separate raw-format images, taken with multiple separate filters, and then recombined, coloured and graded to stunning effect.

Of course, saying that these processed space images are in any way ‘incorrect’ would be the same as complaining that you’re not seeing the greenscreens when you go to see the next Marvel movie. A presentational choice was made, and the resultant images are the reason myself and millions of others are so passionate about the topic of space. But likewise, you wouldn’t use those same images for further analysis.

Simply put, there is a time and a place for both the raw data, and the edited, processed imagery. Although if you intend to manipulate, examine or process that image data in any way, then using a raw format is essential.


Raw Data Sources

On my trip down the rabbit-hole I’ve started to compile a list of sites that provide space imagery in a ‘raw’ form. Whilst it is in no-way designed as an exhaustive collection of every source in existance, I do intend to update the list as and when I find missing/incorrect information (or when said missing/incorrect information is kindly pointed out to me).

Before I present the list however, I feel I should briefly draw attention to the fact that all of this is only possible thanks to the fact that certain organisations (NASAJPLESA, & SETI to name but-a-few) not only achieve the momentous task of capturing these images in the first place, but then choose to make them available to a global audience, in both raw and processed forms — a staggeringly momentous task which I am incredibly grateful for.

Source Target Agency Year Data URL
Cassini-Huygens Saturn NASA 1997 PDS-Imaging : EDRs, RADAR, VIMS and Cartographic Maps
Chandrayaan-1 The Moon ISRO 2008 PDS-Imaging : M3 Data
Cluster II Earth’s Magnetosphere ESA/NASA 2000 Cosmos : Cluster Science Archive
Dawn Vesta & Ceres NASA 2007 PDS : Small Bodies Node
EXOSAT Everything ESA 1983 Cosmos : Spectra, Images and Light Curves
Galileo Jupiter NASA 1989 PDS-Imaging : SSI REDRs and NIMS EDRs/CUBEs
Giotto Halley’s Comet ESA 1985 Cosmos : Planetary Science Archive
Hayabusa 25143 Itokawa JAXA 2003 Planetary Data Archive : Hayabusa AMICA
Herschel Space Observatory Everything ESA 2009 Cosmos : Herschel Science Archive
Hubble Space Telescope Everything NASA/ESA 1990 Hubble Legacy Archive
ISO Everything ESA/ISAS/NASA 1995 ESA : ISO Data Archive
Juno Jupiter NASA 2011 NASA : Mission Juno - Media Gallery and Image Processing Gallery
Lunar Reconnaissance Orbiter The Moon NASA 2009 PDS-Imaging : LAMP & LROC, and LCROSS
Magellan Venus NASA 1989 PDS-Imaging : Mosaics, Global Altimetry & Radiometry/SAR Experiment Data Records
Mariner 10 Mercury & Venus NASA 1973 PDS-Imaging : EDRs
Mariner 9 Mars NASA 1971 PDS-Imaging : EDRs
Mars Exploration Rover (A & B) Mars NASA 2003 PDS-Imaging : EDRs and RDRs
Mars Express Mars ESA 2003 PDS-Imaging : HRSC RDRs, Map Projected Image Data and Orthophoto/DTM
Mars Global Surveyor Mars NASA 1996 PDS-Imaging : MOC Data Products/Records
Mars Odyssey Mars NASA 2001 PDS-Imaging : THEMIS EDRs and RDRs
Mars Pathfinder Mars NASA 1996 PDS-Imaging : ASI-MET, EDRs and Rover Cameras/APXS
Mars Reconnaissance Orbiter Mars NASA 2005 PDS-Imaging : HiRISE EDRs & RDRs, CTX EDRs and MARCI EDRs
Mars Science Laboratory (Curiosity) Mars NASA 2011 PDS-Imaging : HAZCAM, NAVCAM, Mastcam, MAHLI and MARDI EDRs & RDRs
Messenger Mercury NASA 2004 PDS-Imaging :  MEDR, CDR, DDR, BDR, MDR, MD3, HIE, HIW, LOI, MP5, RTM, and DEM
New Horizons Jupiter & Pluto NASA 2006 PDS : Small Bodies Node
Phoenix Mars NASA 2007 PDS-Imaging : OM, RAC and SSI EDRs
Planck Cosmic Background Radiation ESA 2009 Cosmos : Planck Legacy Archive
Proba-V Earth ESA 2013 Proba - Vegetation
Rosetta 67P/Churyumov–Gerasimenko ESA 2004 PDS : Small Bodies Node, and ESA : Archive Image Browser
Ulysses The Sun NASA/ESA 1990 ESA : Ulysses Final Archive
Venus Express Venus ESA 2005 ESA : Planetary Science Archive
Viking Mars NASA 1975 PDS-Imaging : EDRs, Maps, Mosaics and Processed Images
Voyager Outer Solar System NASA 1977 PDS-Imaging : EDRs & Processed Images
XMM-Newton Everything ESA 1999 ESA : XMM-Newton Sience Archive

Notes

  1. It’s worth mentioning that sometimes just getting the data into a usable format can be a long, technically-challenging process — some users over on the Unmanned Spaceflight forums work on single datasets for months before even beginning to stitch and grade imagery. Whilst this article is intended to serve as just a brief introduction, future articles will hopefully begin to at least scrape the surface of this demanding workflow.
  2. There are of course a multitude of ‘raw’ image types that I could cover here, however for more in-depth information on the different raw formats available I can highly recommend Bill Green’s fantastic post  ‘Finding and Using Space Data’ over at The Planetary Society.
  3. For further reading on the benefit of using raw archived data over converted formats, Emily Lakdawalla’s ‘Getting to the real science image data: It’s not that hard!” is an excellent, thouroughly convincing article.
  4. Yes, that ‘construction’ image really is over-simplified. I’ll be attempting to recreate a much more representative version that hopefully does the process a little more justice in an upcoming article.
  5. Finally please go and visit Planetary Bot. I think I love it a little…