Extremely Large Images: Considerations for Contemporary Approach by Bruce Berriman.
From the post:
This is the title of a paper by Kitaeff, Wicenec, Wu and Taubman recently posted on astro-ph. The paper addresses the issues of accessing and interacting with very large data-cube images that will be be produced by next generation of radio telescopes such as the Square Kilometer Array (SKA), the Low Frequency Array for Radio Astronomy (LOFAR) and others. Individual images may be TB-sized, and one SKA Reference Mission Project, “Galaxy Evolution in the Nearby Uni-verse: HI Observations,” will generate individual images of 70-90 TB each.
Data sets this large cannot reside on local disks, even with anticipated advances in storage and network technology. Nor will any new lossless compression techniques that preserve the low S/N of the data save the day, for the act of decompression will impose excessive computational demands on servers and clients.
(emphasis added)
Yes, you read that correctly: “generate individual images of 70-90 TB each.”
Looks like the SW/WWW is about to get a whole lot smaller, comparatively speaking.
But the data you will be encountering will be getting larger. A lot larger.
Bear in mind that the semantics we associate with data will be getting larger as well.
Read that carefully, especially the part about “…we associate with data…”
Data may appear to have intrinsic semantics but only because we project but do not acknowledge the projection of semantics.
The more data we have, the more space there is for semantic projection, by everyone who views the data.
Whose views/semantics do you want to capture?