## Archive for the ‘Regexes’ Category

### Debuggex [Emacs Alternative, Others?]

Saturday, April 13th, 2013

Debuggex: A visual regex helper

Regexes (regular expressions) are a mainstay of data mining/extraction.

Debuggex is a regex debugger with visual cues to help you with writing/debugging regular expressions.

The webpage reports full JS regexes are not yet supported.

If you need a fuller alternative, consider debugging regex expressions in Emacs.

M - x regexp-builder

 

which shows matches as you type.

Be aware that regex languages vary (no real surprise).

One helpful resource: Regular Expression Flavor Comparison

### Working with Pig

Saturday, February 16th, 2013

Working with Pig by Dan Morrill. (video)

From the description:

Pig is a SQL like command language for use with Hadoop, we review a simple PIG script line by line to help you understand how pig works, and regular expressions to help parse data. If you want a copy of the slide presentation – they are over on slide share http://www.slideshare.net/rmorrill.

Very good intro to PIG!

Mentions a couple of resources you need to bookmark:

Input Validation Cheat Sheet (The Open Web Security Application Project – OWASP) – regexes to re-use in Pig scripts. Lots of other regex cheat sheet pointers. (Being mindful that “\” must be escaped in PIG.)

Regular-Expressions.info A more general resource on regexes.

I first saw this at: This Quick Pig Overview Brings You Up to Speed Line by Line.

### C++11 regex cheatsheet

Sunday, July 22nd, 2012

C++11 regex cheatsheet

A one page C++11 regex cheatsheet that you may find useful.

Curious though, how useful do you find colors on cheatsheets?

Or are there cheatsheets where you find colors useful and others not?

If so, what seems to be the difference?

Not an entirely idle query. I want to author a cheatsheet or two, but want them to be useful to others.

At one level, I see cheatsheets as being extremely minimalistic, no commentary, just short reminders of the correct syntax.

A step up from that level, perhaps for rarely used commands, a bit more than bare syntax.

Suggestions? Pointers to cheatsheets you have found useful?

### BADREX: In situ expansion and coreference of biomedical abbreviations using dynamic regular expressions

Friday, June 22nd, 2012

Abstract:

BADREX uses dynamically generated regular expressions to annotate term definition-term abbreviation pairs, and corefers unpaired acronyms and abbreviations back to their initial definition in the text. Against the Medstract corpus BADREX achieves precision and recall of 98% and 97%, and against a much larger corpus, 90% and 85%, respectively. BADREX yields improved performance over previous approaches, requires no training data and allows runtime customisation of its input parameters. BADREX is freely available from https://github.com/philgooch/BADREX-Biomedical-Abbreviation-Expander as a plugin for the General Architecture for Text Engineering (GATE) framework and is licensed under the GPLv3.

From the conclusion:

The use of regular expressions dynamically generated from document content yields modestly improved performance over previous approaches to identifying term definition–term abbreviation pairs, with the benefit of providing in-place annotation, expansion and coreference in a single pass. BADREX requires no training data and allows runtime customisation of its input parameters.

Although not mentioned by the author, a reader can agree/disagree with an expansion as they are reading the text. Could provide for faster feedback/correction of the expansion.

Assuming you accept a correct/incorrect view of expansions. I prefer agree/disagree as the more general rule. Correct/incorrect is the result of the application of a specified rule.

### Clojure and XNAT: Introduction

Saturday, February 4th, 2012

Clojure and XNAT: Introduction

Over the last two years, I’ve been using Clojure quite a bit for managing, testing, and exploratory development in XNAT. Clojure is a new member of the Lisp family of languages that runs in the Java Virtual Machine. Two features of Clojure that I’ve found particularly useful are seamless Java interoperability and good support for interactive development.

“Interactive development” is a term that may need some explanation: With many languages — Java, C, and C++ come to mind — you write your code, compile it, and then run your program to test. Most Lisps, including Clojure, have a different model: you start the environment, write some code, test a function, make changes, and rerun your test with the new code. Any state necessary for the test stays in memory, so each write/compile/test iteration is fast. Developing in Clojure feels a lot like running an interpreted environment like Matlab, Mathematica, or R, but Clojure is a general-purpose language that compiles to JVM bytecode, with performance comparable to plain old Java.

One problem that comes up again and again on the XNAT discussion group and in our local XNAT support is that received DICOM files land in the unassigned prearchive rather than the intended project. Usually when this happens, there’s a custom rule for project identification where the regular expression doesn’t quite match what’s in the DICOM headers. Regular expressions are a wonderfully concise way of representing text patterns, but this sentence is equally true if you replace “wonderfully concise” with “maddeningly cryptic.”

Interesting “introduction” that focuses on regular expressions.

If you don’t know XNAT (I didn’t):

XNAT is an open source imaging informatics platform, developed by the Neuroinformatics Research Group at Washington University. It facilitates common management, productivity, and quality assurance tasks for imaging and associated data. Thanks to its extensibility, XNAT can be used to support a wide range of imaging-based projects.

Important neuroinformatics project based at Washington University, which has a history of very successful public technology projects.

Never hurts to learn more about any informatics project, particularly one in the medical sciences. With an introduction to Clojure as well, what more could you want?

### How Google Code Search Worked

Tuesday, January 24th, 2012

In the summer of 2006, I was lucky enough to be an intern at Google. At the time, Google had an internal tool called gsearch that acted as if it ran grep over all the files in the Google source tree and printed the results. Of course, that implementation would be fairly slow, so what gsearch actually did was talk to a bunch of servers that kept different pieces of the source tree in memory: each machine did a grep through its memory and then gsearch merged the results and printed them. Jeff Dean, my intern host and one of the authors of gsearch, suggested that it would be cool to build a web interface that, in effect, let you run gsearch over the world’s public source code. I thought that sounded fun, so that’s what I did that summer. Due primarily to an excess of optimism in our original schedule, the launch slipped to October, but on October 5, 2006 we did launch (by then I was back at school but still a part-time intern).

I built the earliest demos using Ken Thompson’s Plan 9 grep, because I happened to have it lying around in library form. The plan had been to switch to a “real” regexp library, namely PCRE, probably behind a newly written, code reviewed parser, since PCRE’s parser was a well-known source of security bugs. The only problem was my then-recent discovery that none of the popular regexp implementations – not Perl, not Python, not PCRE – used real automata. This was a surprise to me, and even to Rob Pike, the author of the Plan 9 regular expression library. (Ken was not yet at Google to be consulted.) I had learned about regular expressions and automata from the Dragon Book, from theory classes in college, and from reading Rob’s and Ken’s code. The idea that you wouldn’t use the guaranteed linear time algorithm had never occurred to me. But it turned out that Rob’s code in particular used an algorithm only a few people had ever known, and the others had forgotten about it years earlier. We launched with the Plan 9 grep code; a few years later I did replace it, with RE2.

Russ covers inverted indexes, tri-grams, regexes, pointers to working code and examples of how to use the code searcher locally on Linux source code for example.

Extremely useful article as an introduction to indexes and regexes.