Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

January 21, 2015

MrGeo (MapReduce Geo)

Filed under: Geo Analytics,Geographic Information Retrieval,Geography,Geospatial Data — Patrick Durusau @ 7:58 pm

MrGeo (MapReduce Geo)

From the webpage:

MrGeo was developed at the National Geospatial-Intelligence Agency (NGA) in collaboration with DigitalGlobe. The government has “unlimited rights” and is releasing this software to increase the impact of government investments by providing developers with the opportunity to take things in new directions. The software use, modification, and distribution rights are stipulated within the Apache 2.0 license.

…

MrGeo (MapReduce Geo) is a geospatial toolkit designed to provide raster-based geospatial capabilities that can be performed at scale. MrGeo is built upon the Hadoop ecosystem to leverage the storage and processing of hundreds of commodity computers. Functionally, MrGeo stores large raster datasets as a collection of individual tiles stored in Hadoop to enable large-scale data and analytic services. The co-location of data and analytics offers the advantage of minimizing the movement of data in favor of bringing the computation to the data; a more favorable compute method for Geospatial Big Data. This framework has enabled the servicing of terabyte scale raster databases and performed terrain analytics on databases exceeding hundreds of gigabytes in size.
…

The use cases sound interesting:

Exemplar MrGeo Use Cases:

  • Raster Storage and Provisioning: MrGeo has been used to store, index, tile, and pyramid multi-terabyte scale image databases. Once stored, this data is made available through simple Tiled Map Services (TMS) and or Web Mapping Services (WMS).
  • Large Scale Batch Processing and Serving: MrGeo has been used to pre-compute global 1 ArcSecond (nominally 30 meters) elevation data (300+ GB) into derivative raster products : slope, aspect, relative elevation, terrain shaded relief (collectively terabytes in size)
  • Global Computation of Cost Distance: Given all pub locations in OpenStreetMap, compute 2 hour drive times from each location. The full resolution is 1 ArcSecond (30 meters nominally)

    • I wonder if you started war gaming attacks on well known cities and posting maps on how the attacks could develop if that would be covered under free speech? Assuming your intent was to educate the general populace about areas that are more dangerous than others in case of a major incident.

    I first saw this in a tweet by Marin Dimitrov.

    December 8, 2013

    Mapping the open web using GeoJSON

    Filed under: Geo Analytics,Geographic Data,Geographic Information Retrieval,JSON,NSA — Patrick Durusau @ 5:59 pm

    Mapping the open web using GeoJSON by Sean Gillies.

    From the post:

    GeoJSON is an open format for encoding information about geographic features using JSON. It has much in common with older GIS formats, but also a few new twists: GeoJSON is a text format, has a flexible schema, and is specified in a single HTML page. The specification is informed by standards such as OGC Simple Features and Web Feature Service and streamlines them to suit the way web developers actually build software today.

    Promoted by GitHub and used in the Twitter API, GeoJSON has become a big deal in the open web. We are huge fans of the little format that could. GeoJSON suits the web and suits us very well; it plays a major part in our libraries, services, and products.

    A short but useful review of why GeoJSON is important to MapBox and why it should be important to you.

    A must read if you are interested in geo-locating data of interest to your users to maps.

    Sean mentions that Github promotes GeoJSON but I’m curious if the NSA uses/promotes it as well? 😉

    July 27, 2013

    …Spatial Analytics with Hive and Hadoop

    Filed under: Geo Analytics,Hadoop,Hive — Patrick Durusau @ 5:54 pm

    How To Perform Spatial Analytics with Hive and Hadoop by Carter Shanklin.

    From the post:

    One of the big opportunities that Hadoop provides is the processing power to unlock value in big datasets of varying types from the ‘old’ such as web clickstream and server logs, to the new such as sensor data and geolocation data.

    The explosion of smart phones in the consumer space (and smart devices of all kinds more generally) has continued to accelerate the next generation of apps such as Foursquare and Uber which depend on the processing of and insight from huge volumes of incoming data.

    In the slides below we look at a sample, anonymized data set from Uber that is available on Infochimps. We step through basics of analyzing the data in Hive and learn how a new using spatial analysis decide whether a new product offering is viable or not.

    Great tutorial and slides!

    My only reservation is the use of geo-location data to make a judgement about the potential for a new ride service.

    Geo-location data is only way to determine potential for a ride service. Surveying potential riders would be another.

    Or to put it another way, having data to crunch, doesn’t mean crunching data will lead to the best answer.

    September 1, 2012

    “What Makes Paris Look Like Paris?”

    Filed under: Geo Analytics,Geographic Data,Image Processing,Image Recognition — Patrick Durusau @ 3:19 pm

    “What Makes Paris Look Like Paris?” by Erwin Gianchandani.

    From the post:

    We all identify cities by certain attributes, such as building architecture, street signage, even the lamp posts and parking meters dotting the sidewalks. Now there’s a neat study by computer graphics researchers at Carnegie Mellon University — presented at SIGGRAPH 2012 earlier this month — that develops novel computational techniques to analyze imagery in Google Street View and identify what gives a city its character….

    From the abstract:

    Given a large repository of geotagged imagery, we seek to automatically find visual elements, e.g. windows, balconies, and street signs, that are most distinctive for a certain geo-spatial area, for example the city of Paris. This is a tremendously difficult task as the visual features distinguishing architectural elements of different places can be very subtle. In addition, we face a hard search problem: given all possible patches in all images, which of them are both frequently occurring and geographically informative? To address these issues, we propose to use a discriminative clustering approach able to take into account the weak geographic supervision. We show that geographically representative image elements can be discovered automatically from Google Street View imagery in a discriminative manner. We demonstrate that these elements are visually interpretable and perceptually geo-informative. The discovered visual elements can also support a variety of computational geography tasks, such as mapping architectural correspondences and influences within and across cities, finding representative elements at different geo-spatial scales, and geographically-informed image retrieval.

    The video and other resources are worth the time to review/read.

    What features do you rely on to “recognize” a city?

    The potential to explore features within a city or between cities looks particularly promising.

    March 29, 2012

    Intro to Map Suite DynamoDB Extension Technology Preview

    Filed under: Dynamo,Geo Analytics,Mapping — Patrick Durusau @ 6:40 pm

    Intro to Map Suite DynamoDB Extension Technology Preview

    Promotes Amazon’s DynamoDB, including pricing but an interesting presentation none the less.

    A couple of suggestions:

    The code mentioned in the presentation is unreadable. I am sure it worked at an actual presentation but doesn’t work on the web.

    The extension is downloadable but requires MS Studio to be opened. Understand why there is a version for one of the more popular programming IDE’s but the product should not be restricted to that IDE.

    Some resources that may be of interest:

    http://thinkgeo.com/

    Press Release on this extension.

    Looking for feedback on the technology.

    Great to be able to support GIS data robustly but the “killer” app for GIS data would be to integrate other data in real time.

    For example, take a map of a major metropolitan area and integrate real time GIS coordinates from police and fire units, across jurisdictions during major public events. While at the same time integrating encounters, arrests, intelligence reports, both with each other as well as the GIS positions.

    March 18, 2012

    Gisgraphy

    Filed under: Geo Analytics,Geographic Data,Geographic Information Retrieval,Gisgraphy — Patrick Durusau @ 8:53 pm

    Gisgraphy

    From the website:

    Gisgraphy is a free, open source framework that offers the possibility to do geolocalisation and geocoding via Java APIs or REST webservices. Because geocoding is nothing without data, it provides an easy to use importer that will automagically download and import the necessary (free) data to your local database (Geonames and OpenStreetMap : 42 million entries). You can also add your own data with the Web interface or the importer connectors provided. Gisgraphy is production ready, and has been designed to be scalable(load balanced), performant and used in other languages than just java : results can be output in XML, JSON, PHP, Python, Ruby, YAML, GeoRSS, and Atom. One of the most popular GPS tracking System (OpenGTS) also includes a Gisgraphy client…read more

    Free webservices:

    • Geocoding
    • Street Search
    • Fulltext Search
    • Reverse geocoding / street search
    • Find nearby
    • Address parser

    Services that you could use with smart phone apps or in creating topic map based collections of data that involve geographic spaces.

    February 11, 2012

    GeoMapApp

    Filed under: Geo Analytics,Geographic Data,Geographic Information Retrieval — Patrick Durusau @ 7:53 pm

    GeoMapApp

    From the webpage:

    GeoMapApp is an earth science exploration and visualization application that is continually being expanded as part of the Marine Geoscience Data System (MGDS) at the Lamont-Doherty Earth Observatory of Columbia University. The application provides direct access to the Global Multi-Resolution Topography (GMRT) compilation that hosts high resolution (~100 m node spacing) bathymetry from multibeam data for ocean areas and ASTER (Advanced Spaceborne Thermal Emission and Reflection Radiometer) and NED (National Elevation Dataset) topography datasets for the global land masses.

    See YouTube: GeoMapApp (21 video tutorial)

    More data for your merging pleasure. Not to mention a resource on how others prefer to understand/view their data.

    2011 Research Tools (Geo Data)

    Filed under: Geo Analytics,Geographic Data,Python — Patrick Durusau @ 7:52 pm

    2011 Research Tools

    A very good course/refresher on handling data sets for the Earth sciences. There are videos, podcasts, notes, etc.

    From the description:

    For the UNH Earth Science (ESCI) 895-03 class, I created extra videos. This is a part of the UNH Center for Coastal and Ocean Mapping (CCOM) / Joint Hydrographic Center (JHC). The class web page is: http://vislab-ccom.unh.edu/~schwehr/Classes/2011/esci895-researchtools/

    December 4, 2011

    All the software a geoscientist needs. For free!

    Filed under: Geo Analytics,Geographic Data,Geographic Information Retrieval — Patrick Durusau @ 8:17 pm

    All the software a geoscientist needs. For free! by John A. Stevenson.

    It is quite an impressive list and what’s more, John has provided a script to install it on a Linux machine.

    If you any mapping or geoscience type needs, you would do well to consider some of the software listed here.

    A handy set of tools if you are working with geoscience types on topic map applications as well.

    November 25, 2011

    GeoIQ API Overview

    Filed under: Geo Analytics,Geographic Data,Geographic Information Retrieval — Patrick Durusau @ 4:29 pm

    GeoIQ API Overview

    From the webpage:

    GeoIQ is the engine that powers the GeoCommons Community. GeoIQ includes a full Application Programming Interface (API) that allows developers to build unique and powerful domain specific applications. The API provides capability for uploading and download data, searching for data and maps, building, embedding, and theming maps or charts, as well as general user, group, and permissions management.

    The GeoIQ API consists of a REST API and a JavaScript API. REST means that it uses simple URL’s and HTTP methods to perform all of the actions. For example, a dataset is a specific endpoint that a user can create, read, update or delete (CRUD).

    Another resource for topic mappers who want to link information to “real” locations. 😉

    September 12, 2011

    LinkedGeoData Release 2

    Filed under: Geo Analytics,Geographic Data,Geographic Information Retrieval,Linked Data,RDF — Patrick Durusau @ 8:25 pm

    LinkedGeoData Release 2

    From the webpage:

    The aim of the LinkedGeoData (LGD) project is to make the OpenStreetMap (OSM) datasets easily available as RDF. As such the main target audience is the Semantic Web community, however it may turn out to be useful to a much larger audience. Additionally, we are providing interlinking with DBpedia and GeoNames and integration of class labels from translatewiki and icons from the Brian Quinion Icon Collection.

    The result is a rich, open, and integrated dataset which we hope to be useful for research and application development. The datasets can be publicly accessed via downloads, Linked Data, and SPARQL-endpoints. We have also launched an experimental “Live-SPARQL-endpoint” that is synchronized with the minutely updates from OSM whereas the changes to our store are republished as RDF.

    More geographic data.

    July 13, 2011

    GeoCommons Enterprise Features – Free!

    Filed under: Geo Analytics,Geographic Data,Geographic Information Retrieval — Patrick Durusau @ 7:30 pm

    GeoCommons Enterprise Features – Free!

    From the email announcement:

    • Analytics: Easy-to-use, advanced spatial analytics that users and groups can utilize to answer mission-critical questions. Select among numerous analyses such as filtering, buffers, spatial aggregation and predictive analysis.
    • Private Data Support: Keep proprietary data private and unsearchable by others. Now you can upload proprietary data, analyze it with other data and create compelling maps, charts and graphs all within a secure interface.
    • Groups and Permissions: Allow others in your group or organization to access and collaborate with you. Enable permissions at various levels to limit or expand data sharing. See a step-by-step guide of how to create groups and make your data private here from @seangorman.

    For groups and private data, see: Private Data and Groups for GeoCommons!!

    GeoCommons has 70,000 datasets.

    If you look around you might find something you like.

    Topic mappers should ask themselves: Why does this work? (more on that anon)

    June 12, 2011

    clusterPy: Library of spatially constrained
    clustering algorithms

    Filed under: Clustering,Geo Analytics,Geographic Data,Geographic Information Retrieval — Patrick Durusau @ 4:13 pm

    clusterPy: Library of spatially constrained clustering algorithms

    From the webpage:

    Analytical regionalization (also known as spatially constrained clustering) is a scientific way to decide how to group a large number of geographic areas or points into a smaller number of regions based on similarities in one or more variables (i.e., income, ethnicity, environmental condition, etc.) that the researcher believes are important for the topic at hand. Conventional conceptions of how areas should be grouped into regions may either not be relevant to the information one is trying to illustrate (i.e., using political regions to map air pollution) or may actually be designed in ways to bias aggregated results.

    May 29, 2011

    Geospatial Data Preservation

    Filed under: Data Source,Geo Analytics,Geographic Data,Geographic Information Retrieval,Metadata,Preservation — Patrick Durusau @ 7:03 pm

    Geospatial Data Preservation

    From the FAQ:

    What are geospatial information and geospatial data?

    “Geospatial Information” identifies, depicts or describes geographic locations, boundaries or characteristics of Earth’s inhabitants or natural or human-constructed features. Geospatial data include geographic coordinates that identify a specific location on the Earth; and data that are linked to geographic locations or have a geospatial component.

    Why preserve geospatial data?

    • Preserving and reusing geospatial data can save time and money
    • Geospatial data provide important baseline information for later comparison and analysis
    • Geospatial data can increase the value of important historical and administrative records
    • Geospatial data are often useful for purposes not originally envisioned

    Serious data acquisition, processing, storage and preservation folks. A good starting place to learn the community’s history, vocabulary, concerns and present preservation efforts.

    April 22, 2011

    Geo Analytics Tutorial – Where 2.0 2011

    Filed under: Geo Analytics,Hadoop,Mechanical Turk,Pig — Patrick Durusau @ 1:04 pm

    Geo Analytics Tutorial – Where 2.0 2011

    Very cool set of slides on geo analytics from Pete Skomoroch.

    Includes use of Hadoop, Pig, Mechanical Turk.

    Powered by WordPress