Mathew Hurst provides a quick listing of data engines (including his own, which merits a close look).
February 23, 2011
AI Mashup Challenge 2011
Due date: 1 April 2011
From the website:
The AI mashup challenge accepts and awards mashups that use AI technology, including but not restricted to machine learning and data mining, machine vision, natural language processing, reasoning, ontologies and the semantic web.
Imagine for example:
- Information extraction or automatic text summarization to create a task-oriented overview mashup for mobile devices.
- Semantic Web technology and data sources adapting to user and task-specific configurations.
- Semantic background knowledge (such as ontologies, WordNet or Cyc) to improve search and content combination.
- Machine translation for mashups that cross language borders.
- Machine vision technology for novel ways of aggregating images, for instance mixing real and virtual environments.
- Intelligent agents taking over simple household planning tasks.
- Text-to-speech technology creating a voice mashup with intelligent and emotional intonation.
- The display of Pub Med articles on a map based on geographic entity detection referring to diseases or health centers.
The emphasis is not on providing and consuming semantic markup, but rather on using intelligence to mashup these resources in a more powerful way.
This looks like an opportunity for an application that assists users in explicit identification or confirmation of identification of subjects.
Rather than auto-correcting, human-correcting.
Assuming we can capture the corrections, wouldn’t that mean that our apps would incrementally get “smarter?” Rather than starting off from ground zero with each request? (True, a lot of analysis goes on with logs, etc. Why not just ask?)
Big Oil and Big Data
Big Oil and Big Data Mike Betron, Marketing Director of Infoglide says that it is becoming feasible to mine “big data” and to exploit “entity resolution.”
Those who want to exploit the availability of big data have another powerful tool at their disposal – entity resolution. The ability to search across multiple databases with disparate forms residing in different locations can tame large amounts of data very quickly, efficiently resolving multiple entities into one and finding hidden connections without human intervention in many application areas, including detecting financial fraud.
By exploiting advancing technologies like entity resolution, systems can give organizations a distinct competitive advantage over those who lag in technology adoption.
I have to quibble about the …without human intervention… part, although I am quite happy with augmented human supervision.
Well, that and the implication that entity resolution is a new technology. In various guises, entity resolution has been in use for decades in medical epidemiology, for example.
Preventing subject identifications from languishing in reports, summaries, and the other information debris of a modern organization. So that organizational memories, documented and accessible organization memories prosper and grow, now that would be something different. (It could also be called a topic map.)
Comments Off on Big Oil and Big Data
Do You Tweet?
This is probably old news to most people who Tweet but I stumbled across: http://wthashtag.com, which maintains a directory of hash tags.
Hash tags for which you can submit definitions.
I have entered a definition for #odf.
There already is a definition for #topicmaps.
When you tweet about topic maps related material, events, etc., please use #topicmaps.
I try to use that plus hashtags from other relevant areas in hopes people will follow the topic maps tag as something unfamiliar that may be of interest.
Hard to say if it will be effective but I suspect no less effective than some marketing strategies I have seen.
Update: Apparently this service has been moved from this address to: http://hashtag.it/. And as far as I can tell, the hashtags for topicmaps and ODF have been lost.
Do note that the URL listed in the original tweet does not go to the tweet site but to “What the Trend? API.”
February 22, 2011
LingPipe Baseline for MITRE Name Matching Challenge
LingPipe Baseline for MITRE Name Matching Challenge.
Bob Carpenter walks though the use of LingPipe in connection with the MITRE Name Matching Challenge.
There are many complex issues in data mining but doing well on basic tasks is always a good starting place.
Comments Off on LingPipe Baseline for MITRE Name Matching Challenge
Luke
From the website:
Lucene is an Open Source, mature and high-performance Java search engine. It is highly flexible, and scalable from hundreds to millions of documents.
Luke is a handy development and diagnostic tool, which accesses already existing Lucene indexes and allows you to display and modify their content in several ways:
- browse by document number, or by term
- view documents / copy to clipboard
- retrieve a ranked list of most frequent terms
- execute a search, and browse the results
- analyze search results
- selectively delete documents from the index
- reconstruct the original document fields, edit them and re-insert to the index
- optimize indexes
- open indexes consisting of multiple parts, and located on Hadoop filesystem
- and much more…
Searching is interesting and I have several more search engines to report this week, but the real payoff is finding.
And recording the finding so that other users can benefit from it.
We could all develop our own maps of the London Underground, at the expense of repeating the effort of others.
Or, we can purchase a copy of the London Underground.
Which one seems more cost effective for your organization?
LOTR Timeline Graph
Interesting take on visualizing the time line in the Lord of the Rings.
Based on the entertaining but fairly flat movie time lines you will find at XKCD.
A much more subtle work could be based on the books. 😉
Not that movie time lines have to be flat, but there is only so much that be be done in the usual movie length presentation.
For LOTR enthusiasts:
- What other time lines or perspectives would you add?
- How would you represent deceptions concerning gender?
- What of Denethor’s “wrestling” with Sauron? How would you represent that?
- How would you deal with cross-overs between time lines, such as time as experienced by the Ents versus others?
- Your suggested issues/problems…
Quantum GIS
From the website:
QGIS is a cross-platform (Linux, Windows, Mac) open source application with many common GIS features and functions. The major features include:
1. View and overlay vector and raster data in different formats and projections without conversion to an internal or common format.
Supported formats include:
- spatially-enabled PostgreSQL tables using PostGIS and SpatiaLite,
- most vector formats supported by the OGR library*, including ESRI shapefiles, MapInfo, SDTS and GML.
- raster formats supported by the GDAL library*, such as digital elevation models, aerial photography or landsat imagery,
- GRASS locations and mapsets,
- online spatial data served as OGC-compliant WMS , WMS-C (Tile cache), WFS and WFS-T
2. Create maps and interactively explore spatial data with a friendly graphical user interface. The many helpful tools available in the GUI include:
- on the fly projection,
- print composer,
- overview panel,
- spatial bookmarks,
- identify/select features,
- edit/view/search attributes,
- feature labeling,
- vector diagram overlay
- change vector and raster symbology,
- add a graticule layer,
- decorate your map with a north arrow, scale bar and copyright label,
- save and restore projects
3. Create, edit and export spatial data using:
- digitizing tools for GRASS and shapefile formats,
- the georeferencer plugin,
- GPS tools to import and export GPX format, convert other GPS formats to GPX, or down/upload directly to a GPS unit
4. Perform spatial analysis using the fTools plugin for Shapefiles or the integrated GRASS plugin, including:
- map algebra,
- terrain analysis,
- hydrologic modeling,
- network analysis,
- and many others
5. Publish your map on the internet using the export to Mapfile capability (requires a webserver with UMN MapServer installed)
6. Adapt Quantum GIS to your special needs through the extensible plugin architecture.
I didn’t find this on my own. 😉 This and the T I G E R data source were both mentioned Paul Smith’s Mapping with Location Data presentation.
Data and manipulations you usually find have no explicit basis in subject identity but that is your opportunity to really shine.
Assuming you can discover some user need that can be met with explicit subject identity or met better with explicit subject identity than not.
Let’s try not to be like some vendors I could mention where a user’s problem has to fit the solution they are offering. I turned down an opportunity like that, some thirty years ago now, and see no reason to re-visit that decision.
At least in my view, any software solution has to fit my problem, not vice versa.
Comments Off on Quantum GIS
T I G E R – Topologically Integrated Geographic Encoding and Referencing system
T I G E R – Topologically Integrated Geographic Encoding and Referencing system
From the US Census Bureau.
From the website:
Latest TIGER/Line® Shapefile Release
- TIGER/Line®Shapefiles are spatial extracts from the Census Bureau’s MAF/TIGER database, containing features such as roads, railroads, rivers, as well as legal and statistical geographic areas.
- They are made available to the public for no charge and are typically used to provide the digital map base for a Geographic Information System or for mapping software.
- They are designed for use with geographic information system (GIS) software. The TIGER/Line®Shapefiles do not include demographic data, but they contain geographic entity codes that can be linked to the Census Bureau’s demographic data, available on American FactFinder
2010 TIGER/Line® Shapefiles Main Page — Released on a rolling basis beginning November 30, 2010.
and,
TIGER®-Related Products
- Census Bureau Maps — Census 2000 Map Series, Online Mapping, and Census 2000 Centers of Population
- Cartographic Boundary Files — Generalized boundary files in (.shp), (.e00) and (ASCII) formats.
- Census 2000 Block Relationship Files — 1990 to 2000 Census Blocks.
- Census 2000 Census Tract Relationship Files — 1990 to 2000 Census Tracts.
- LandView® — A Federal Geographic Data Viewer
Great source of geographic and other data.
Can use it for mashups or, you can push beyond mashups to creating topic maps.
For example, plotting all the crime in an area is a mashup.
Interesting I suppose for real estate agents pushing housing in better neighborhoods.
Having the crime reported in an area and the location of crimes committed by the same person (based on arrest reports) and known associates of that person, that is starting to sound like a topic map. Then add in real time observations and conversations of officers working the area.
Enhancing traditional law enforcement, the most effective way to prevent terrorism.
“Mapping with Location Data” by Paul Smith (February 2011)
“Mapping with Location Data” by Paul Smith (February 2011)
From the description:
With the recent announcement of Baltimore’s open-data initiative, “OpenBaltimore”, there’s been lots of buzz about what people can do with some of this city data. Enter Paul Smith, an expert on data and mapping. Paul will be talking about EveryBlock and how his company uses city data in their neighborhood maps, as well as showing off some cool map visualizations. He’ll also be providing some insight on how folks might be able to jump in and create their own maps based on their own location data.
Our speaker Paul Smith is co-founder and software engineer for EveryBlock, a “what’s going on in my neighborhood” website. He has been developing sites and software for the web since 1994. Originally from Maryland, he recently moved to Baltimore after more than a decade in Chicago, where we co-founded Friends of the Bloomingdale Trail and produced the Election Day Advent Calendar.
Great source of ideas for city data and use of the same.
Two ways where topic maps are a value-add:
1) The relationships between data sets are subjects that can be represented and additional information recorded about those relationships.
2) Identification of subjects can support reliable attachment of other data to the same subjects.
Comments Off on “Mapping with Location Data” by Paul Smith (February 2011)
See an Error at the Washington Post? Now You Can Easily Report It
See an Error at the Washington Post? Now You Can Easily Report It
I’m not sure what would leave me less impressed but I can say this story doesn’t do much for me.
How about you?
The news is that every story will have have a link to a form for reader feedback.
That’s better than current practice but here are a couple of things that might make more of a difference:
- Where possible, embed links directly in stories for websites or other online resources that are mentioned.
Why cite a report from an agency, commission, etc. that is online and not provide a link?
Leaves me with the impression you want me take your word for it.
- Provide permalinks so users can create mappings to news stories to be used with their data.
I would say that the permalinks should contain explicit subject identity but that is expecting too much.
If we can link to it, we can add explicit subject identity.
Comments Off on See an Error at the Washington Post? Now You Can Easily Report It
elasticsearch
From the website:
So, we build a web site or an application and want to add search to it, and then it hits us: getting search working is hard. We want our search solution to be fast, we want a painless setup and a completely free search schema, we want to be able to index data simply using JSON over HTTP, we want our search server to be always available, we want to be able to start with one machine and scale to hundreds, we want real-time search, we want simple multi-tenancy, and we want a solution that is built for the cloud.
“This should be easier“, we declared, “and cool, bonsai cool“.
elasticsearch aims to solve all these problems and more. It is an Open Source (Apache 2), Distributed, RESTful, Search Engine built on top of Lucene.
Another contender in the space for search engines.
Do you have a favorite search engine? If so, what about it makes it your favorite?
Comments Off on elasticsearch
February 21, 2011
Introducing Vector Maps
From the post:
Modern distributed data stores such as CouchDB and Riak, use variants of Multi-Version Concurrency Control to detect conflicting database updates and present these as multi-valued responses.
So, if I and my buddy Ola both update the same data record concurrently, the result may be that the data record now has multiple values – both mine and Ola’s – and it will be up to the eventual consumer of the data record to resolve the problem. The exact schemes used to manage the MVCC differs from system to system, but the effect is the same; the client is left with the turd to sort out.
This let me to an idea, of trying to create a data structure which is by it’s very definition itself able to be merged, and then store such data in these kinds of databases. So, if you are handed two versions, there is a reconciliation function that will take those two records and “merge” them into one sound record, by some definition of “sound”.
Seems to me that reconciliation should not be limited to records differing based on time stamps. 😉
Will have to think about this one for while but it looks deeply similar to issues we are confronting in topic maps.
BTW, saw this first at Alex Popescu’s myNoSQL blog.
Comments Off on Introducing Vector Maps
Still Building Memex & Topic Maps Part 1
Still Building the Memex, Stephen Davies writes:
We define a Personal Knowledge Base – or PKB – as an electronic tool through which an individual can express, capture, and later retrieve the personal knowledge he or she has acquired.
….Personal: Like Bush’s memex, a PKB is intended for private use, and its contents are custom tailored to the individual. It contains trends, relationships, categories, and personal observations that its owner sees but which no one else may agree with. Many of the issues involved in PKB design are also relevant in collaborative settings, as when a homogeneous group of people is jointly building a shared knowledge base. In this case, the knowledge base could simply reflect the consensus view of all contributors; or, perhaps better, it could simultaneously store and present alternate views of its contents, so as to honor several participants who may organize it or view it differently. This can introduce another level of complexity.
I am not sure that having “ear buds” from an intellectual IPod like a PKB is a good idea.
The average reader is already well insulated from the inconvenience of information or opinions dissimilar from their own. Reflecting the “consensus view of all contributors,” is a symptom, and not a desirable one at that.
We have had the equivalents of PKBs over both Republican and Democratic administrations.
The consequences of PKB or Personal Knowledge Bases, weapons of mass destruction in Iraq, the collapse of the housing market, serious mis-steps in both foreign and domestic policy, are too well known to need elaboration.
The problem posed by a PKB is simpler but what we need are PbKB, Public Knowledge Bases.
Even though more complex, a PbKB has the potential to put all citizens on a even footing with regard to debates over policy choices.
It may be difficult to achieve in practice and only every partially successful, but the result could hardly be worse than the echo chamber of a PKB.
(Still Building Memex & Topic Maps Part 2 – Beyond the Echo Chamber)
Comments Off on Still Building Memex & Topic Maps Part 1
Sphinx
Sphinx – Open Source Search Server
Benjamin Bock mentioned Sphinx in a Twitter posting and so I had to go see what he was reading about.
The short version from the website:
Sphinx is an open source full text search server, designed from the ground up with performance, relevance (aka search quality), and integration simplicity in mind. It’s written in C++ and works on Linux (RedHat, Ubuntu, etc), Windows, MacOS, Solaris, FreeBSD, and a few other systems.
Sphinx lets you either batch index and search data stored in an SQL database, NoSQL storage, or just files quickly and easily — or index and search data on the fly, working with Sphinx pretty much as with a database server.
A variety of text processing features enable fine-tuning Sphinx for your particular application requirements, and a number of relevance functions ensures you can tweak search quality as well.
Searching via SphinxAPI is as simple as 3 lines of code, and querying via SphinxQL is even simpler, with search queries expressed in good old SQL.
For more specific details, see: About Sphinx.
Comments welcome.
Comments Off on Sphinx
Topic Maps and the Semantic Web
As I pointed out in Topic Maps in < 5 Minutes, topic maps take it as given that:
salt
sodium chloride
NaCl
are all legitimate identifiers for the same subject.
That is called semantic diversity. (There are more ways to identify salt but let’s stick with those for the moment.)
Contrast that with the Semantic Web, that wants to have one identifier for salt and consequently, no semantic diversity.
You may ask yourself, what happens to all the previous identifications of salt, literally thousands of different ways to identify it?
Do we have to re-write all those identifiers across the vast sweep of human literature?
Or you may ask yourself, given the conditions that lead to semantic diversity still exist, how is future semantic diversity to be avoided?
Good questions.
Or for that matter, the changing information structures and growing information structures where we are storing petabytes of data. What about semantic diversity there as well?
I don’t know.
Maybe we should ask Tim Berners-Lee, timbl @ w3.org?
*****
PS: And this is a really easy subject to identify. Just think about democracy, human rights, justice, or any of the others of thousands of subjects, all of which are works in progress.
Topics Maps in < 5 Minutes
Get a cup of coffee and set your timer!
*****
The best way to explain topic maps is to say why we want them in the first place.
I have a set of food/recipe/FDA documents.
Some use the term salt.
Some use the term sodium chloride.
Some use the term NaCl.
Walmart wants to find information on salt/sodium chloride/NaCl, no matter how it is identified.
Not a hypothetical use case: Walmart Launches Major Initiative to Make Food Healthier and Healthier Food More Affordable
Topic maps would enable Walmart to use any one identification for salt, and to retrieve all the information about it, recorded using any of its identifications.*
How?
Topics represent subjects and can have multiple, independent identifications of the same subject.
That sounds cool!
Can I talk about relationships too? Like they do in the tabloids? 😉
Well, sure, but we call those associations.
Let’s keep with the salt theme.
What about: Mark Kurlansky wrote “Salt: A World History”.
How many subjects to you see?
Mark Kurlansky (subject) wrote “Salt: A World History” (subject).
Two right?
But Mark is probably in a lot of associations, both professional and personal. How do we keep those straight?
Topic maps identify another three subjects:
Mark Kurlansky (subject) (role – author) wrote (subject – written-by) “Salt: A World History” (subject) (role – work).
Mark is said to “play” the role of author in the association. “Salt: A World History” plays the role of a work in the association.
Means we can find all the places where Mark plays the author role, as opposed to husband role, speaker role, etc.
One more thing rounds out the typical topic map, occurrences.
Occurrences are used to write down places where we have seen a subject discussed.
That is we say that Mark Kurlansky (subject) occurs at:
http://www.bookbrowse.com/reviews/index.cfm?book_number=960
(How am I doing for time?)
That’s it.
To review:
- Topics represent subjects that can have multiple identifications.
- Use of any identifier should return information recorded using any identifiers for the same subject.
- Associations are relationships between subjects (role-players) playing roles.
- Occurrences are pointers to where subjects are discussed.
Exercise:
Choose any subject that you like to talk about. Now answer the following questions about your subject:
- Name at least three ways it is identified when people talk about it.
- Name at least one association and the roles in it for your subject.
- Name at least three occurrences (discussions) of your subject that you would like to find again.
Congratulations! Except for the syntax, you have just gathered all the information you need for your first topic map.
*****
* The salt example is only one of literally hundreds of thousands of multiple identifier type issues that confronts any consumer of information.
Walmart could use topic maps to collate information of interest to its 140 million customers every week and to deliver that as a service both to its customers as well as other commercial consumers of information.
A contrast to the U-Sort-It model of Google, which delivers dumpster loads of information, over and over again, for the same request.
PS: There are more complex issues and nuances of the syntaxes but this is the essence of topic maps.
February 20, 2011
Detecting Defense Fraud With Topic Maps
The New York Times, reported Sunday, Hiding Details of Dubious Deal, U.S. Invokes National Security extraordinary fraud in defense contracting.
Gen. Victor E. Renuart, Jr. of the Air Force, former commander of the Northern Command is quoted as saying:
We’ve seen so many folks with a really great idea, who truly believe their technology is a breakthrough, but it turns out not to be.
OK, but the technology in question was alleged to detect messages in broadcasts from Al Jazeera. (I suppose something more than playing them backwards and hearing, “number 9, number 9, number 9, ….”)
The fact that some nut-job wraps up in a flag and proclaim they want to save American lives should not short circuit routine sanity checks.
Here’s one where topic maps would be handy:
Build a topic map interface to the Internet Movie Database that has extracted all the technologies used in all the movies listed.
Give each of those technologies a set of attributes so a contracting officer can check them off while reading contract proposals.
For example, in this case:
- hidden messages
- TV broadcasts
- by enemy
Which would return (along with possibly others): Independence Day.
That should terminate the funding review with a referral to the U.S. Attorney for attempted fraud or the local district attorney for competency hearings.
New technologies are developed all the time but non-fraudulent proposals based upon them can be independently verified before funding.
If it works only for the inventor or in their lab, pass on the opportunity.
BTW, a topic map of who was being funded for what efforts could have made the Air Force aware other departments were terminating funding with this applicant.
Caution: It is always possible to construct topic maps (or other analysis) in hindsight, that avoid problems that have already happened. That said, topic maps can be used to navigate existing information systems, providing a low impact/risk way to evaluate the utility of topic maps in addressing particular problems.
Comments Off on Detecting Defense Fraud With Topic Maps
On Distributed Consistency — Part 1 (MongoDB)
On Distributed Consistency — Part 1 (MongoDB)
The first of a six part series on consistency in distributed databases.
From the website:
See also:
- Part 2 – Eventual Consistency
- Part 3 – Network Partitions
- Part 4 – Multi Data Center
- Part 5 – Multi Writer Eventual Consistency
- Part 6 – Consistency Chart
For distributed databases, consistency models are a topic of huge importance. We’d like to delve a bit deeper on this topic with a series of articles, discussing subjects such as what model is right for a particular use case. Please jump in and help us in the comments.
Consistency is an issue that will confront distributed topic maps so best to start learning the options now.
Comments Off on On Distributed Consistency — Part 1 (MongoDB)
Mio- Distributed Skip Graph based ordered KVS
Mio- Distributed Skip Graph based ordered KVS by Taro Minowa.
From the slide deck:
Mio is…
- a distributed orderd KVS
- memcached + range search
- Skip Graph based
- Written in Erlang
- http://github.com/higepon/mio
- In alpha quality
On skip graphs, see James Aspnes:
Skip graphs, with Gauri Shah. ACM Transactions on Algorithms, 3(4):37, November 2007. An earlier version appeared in Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, January 2003, pp. 384–393. Available as: arXiv:cs.DS/0306043. [PDF]
Not going to be important in all cases but perhaps important to have in your tool kit.
It will be interesting to see as topic maps develop, the techniques that are successful in particular fields.
So those lessons can be applied in similar situations.
No patterns just yet but not excluding the possibility.
Comments Off on Mio- Distributed Skip Graph based ordered KVS
Riak Search
From the website:
Riak Search is a distributed, easily-scalable, failure-tolerant, real-time, full-text search engine built around Riak Core and tightly integrated with Riak KV.
Riak Search allows you to find and retrieve your Riak objects using the objects’ values. When a Riak KV bucket has been enabled for Search integration (by installing the Search pre-commit hook), any objects stored in that bucket are also indexed seamlessly in Riak Search.
The Riak Client API can then be used to perform Search queries that return a list of bucket/key pairs matching the query. Alternatively, the query results can be used as the input to a Riak map/reduce operation. Currently the PHP, Python, Ruby, and Erlang APIs support integration with Riak Search.
The indexing of XML data (it takes path/element name as key) is plausible enough. Made me wonder about a slightly different operation.
What if as part of the indexing operation, additional properties were added to the key?
Could be as simple as the DTD/Schema that defines the element or more complex information about the field.
Comments Off on Riak Search
HSearch NoSQL Search Engine Built on HBase
HSearch NoSQL Search Engine Built on HBase
HSearch features include:
- Multi-XML formats
- Record and document level search access control
- Continuous index updation
- Parallel indexing using multi-machines
- Embeddable inside application
- A REST-ful Web service gateway that supports XML
- Auto sharding
- Auto replication
Original title and link: HSearch: NoSQL Search Engine Built on HBase (NoSQL databases © myNoSQL)
Another entry in the NoSQL arena.
I don’t recall but was parallel querying discussed for TMQL?
AllegroMCOCE: GPU-accelerated Cytoscape Plugin
TM Explorer?
AllegroMCOCE: GPU-accelerated Cytoscape Plugin
From the website:
AllegroMCODE is a high-performance Cytoscape plugin to find clusters, or highly interconnected groups of nodes in a huge complex network such as a protein interaction network and a social network in real time. AllegroMCODE finds the same clusters as the MCODE plugin does, but the analysis usually takes less than a second even for a large complex network. The plugin user interface of AllegroMCODE is based on MOCDE and has additional features. AllegroMCODE is an open source software and freely available under LGPL.
Cluster has various meanings according to the sources of networks. For instance, a protein-protein interaction network is represented as proteins are nodes and interactions between proteins are edges. Clusters in the network can be considered as protein complexes and functional modules, which can be identified as highly interconnected subgraphs. For social networks, people and their relationships are represented as nodes and edges, respectively. A cluster in the network can be considered as a community which has strong inter-relationship among their members.
AllegroMCODE exploits our high performance GPU computing architecture to make your analysis task faster than ever. The analysis task of the MCODE algorithm to find the clusters can be long for large complex networks even though the MCODE is a relatively fast method of clustering. AllegroMCODE provides our parallel algorithm implementation base on the original sequential MCODE algorithm. It can achieve two orders of magnitude speedup for the analysis of a large complex network by using the latest graphics card. You can also exploit the GPU acceleration without any special graphics hardware since it provides the seamless remote processing in our free GPU computing server.
You do not need to purchase any special GPU hardware or systems and also not to care about the tedious installation task of them. All you have to do are to install the AllegroMCODE plugin module on your computer and create a free account on our server.
Simply awesome!
The ability to dynamically explore and configure topic maps will be priceless.
A greater gap than between hot-lead type and a modern word processor.
Will take weeks/months to fully explore but wanted to bring it to your attention.
A thought on Hard vs Soft – Post
(nonIdentification vs. multiIdentification?)
A thought on Hard vs Soft by Dru Sellers starts off with:
With the move from RDBMS to NoSQL are we seeing the same shift that we saw when we moved from Hardware to Software. Are we seeing a shift from Harddata to Softdata? (emphasis in original)
See his post for the rest of the post and the replies.
Do topic maps address a similar hardIdentification vs. softIdentification?
By hardIdentification I mean a single identification.
But it goes further than that doesn’t it?
There isn’t even a single identification in most information systems.
Think about it. You and I both see the same column names and have different ideas of what they mean.
I remember reading in Doan’s dissertation (see Auditable Reconciliation) that a schema reconciliation project would have taken 12 person years but for the original authors being available.
We don’t have any idea what has been identified in most systems and no way to compare it to other “identifications.”
What is this? Write once, Wonder Many Times (WOWMT)?
So, topic maps really are a leap from nonIdentification to multiIdentification.
No wonder it is such a hard sell!
People aren’t accustomed to avoiding the cost of nonIdentification and here we are pitching the advantages of multiIdentification.
Pull two tables at random for your database and have a contest to see who outside the IT department can successfully identify what the column headers represent. No data, just the column headers.*
What other ways can we illustrate the issue of nonIdentification?
Interested in hearing your suggestions.
*****
*I will be posting column headers from public data sets and asking you to guess their identifications.
BTW, some will argue that documentation exists for at least some of these columns.
True enough, but from a processing standpoint it may as well be on a one way mission to Mars.
If the system doesn’t have access to it, it doesn’t exist. (full stop)
Gives you an idea of how impoverished our systems truly are.
IBM’s Watson (the computer, not IBM’s founder, who was also soulless) has been described as deaf and blind. Not only that, but it has no more information than it is given. It cannot ask for more. The life of pocket calculator, if it had emotions, is sad.
Comments Off on A thought on Hard vs Soft – Post
(nonIdentification vs. multiIdentification?)
(nonIdentification vs. multiIdentification?)
February 19, 2011
Cascalog: Clojure-based Query Language for Hadoop – Post
Cascalog: Clojure-based Query Language for Hadoop
From the post:
Cascalog, introduced in the linked article, is a query language for Hadoop featuring:
- Simple – Functions, filters, and aggregators all use the same syntax. Joins are implicit and natural.
- Expressive – Logical composition is very powerful, and you can run arbitrary Clojure code in your query with little effort.
- Interactive – Run queries from the Clojure REPL.
- Scalable – Cascalog queries run as a series of MapReduce jobs.
- Query anything – Query HDFS data, database data, and/or local data by making use of Cascading’s “Tap” abstraction
- Careful handling of null values – Null values can make life difficult. Cascalog has a feature called “non-nullable variables” that makes dealing with nulls painless.
- First class interoperability with Cascading – Operations defined for Cascalog can be used in a Cascading flow and vice-versa
- First class interoperability with Clojure – Can use regular Clojure functions as operations or filters, and since Cascalog is a Clojure DSL, you can use it in other Clojure code.
From Alex Popescu’s myNoSQL
There are a number of NoSQL query languages.
Which should be considered alongside TMQL4J in TMQL discussions.
Comments Off on Cascalog: Clojure-based Query Language for Hadoop – Post
ElephantDB
From the website:
ElephantDB is a database that specializes in exporting key/value data from Hadoop. ElephantDB is composed of two components. The first is a library that is used in MapReduce jobs for creating an indexed key/value dataset that is stored on a distributed filesystem. The second component is a daemon that can download a subset of a dataset and serve it in a read-only, random-access fashion. A group of machines working together to serve a full dataset is called a ring.
Since ElephantDB server doesn’t support random writes, it is almost laughingly simple. Once the server loads up its subset of the data, it does very little. This leads to ElephantDB being rock-solid in production, since there’s almost no moving parts.
ElephantDB server has a Thrift interface, so any language can make reads from it. The database itself is implemented in Clojure.
I rather like that, “…almost no moving parts.”
That has to pay real dividends over the id shuffle in some topic map implementations. Both in terms of processing overhead as well as in auditing.
Group Theoretical Methods and Machine Learning
Group Theory and Machine Learning
Description:
The use of algebraic methods—specifically group theory, representation theory, and even some concepts from algebraic geometry—is an emerging new direction in machine learning. The purpose of this tutorial is to give an entertaining but informative introduction to the background to these developments and sketch some of the many possible applications, including multi-object tracking, learning rankings, and constructing translation and rotation invariant features for image recognition. The tutorial is intended to be palatable by a non-specialist audience with no prior background in abstract algebra.
Be forewarned, tough sledding if you are not already a machine learning sort of person.
But, since I don’t post what I haven’t watched, I did watch the entire video.
It suddenly got interesting just past 93:08 when Risi Kondor started talking about blobs on radar screens and associating information with them…., wait, run that by once again, …blobs on radar screens and associating information with them.
Oh, that is what I thought he said.
I suppose for fire control systems and the like as well as civilian applications.
I am so much of a text and information navigation person that I don’t often think about other applications for “pattern recognition” and the like.
With all the international traveling I used to do, being a blob on a radar screen got my attention!
Has applications in tracking animals in the wild and other tracking with sensor data.
Another illustration of why topic maps need an open-ended and extensible notion of subject identification.
What we think of as methods of subject identification may not be what others think of as methods of subject identification.
Comments Off on Group Theoretical Methods and Machine Learning
IOM-NAE Health Data Collegiate Challenge – 27 April 2010 Deadline
IOM-NAE Health Data Collegiate Challenge
From the website:
The IOM and the National Academy of Engineering (NAE) of the National Academies invite college and university students to participate in an exciting, new initiative to transform health data into innovative, effective new applications (apps) and tools that take on the nation’s pressing health issues. With reams of U.S. Department of Health and Human Services (HHS) data and other health data newly available as part of the Health Data Initiative (HDI), students have an unprecedented opportunity to create interactive apps and other tools that engage and empower people in ways that lead to better health. Working in interdisciplinary teams that meld technological skills with health knowledge, the IOM and NAE believe that college students can generate powerful new products—the next “viral app”— to improve health for communities and individuals.
Along with spreading this one on college campuses, need to also point out the advantages of topic maps!
Comments Off on IOM-NAE Health Data Collegiate Challenge – 27 April 2010 Deadline