Introducing Watchtower – Like Light Table for Pig by Thomas Millar.
From the post:
There are no two ways around it, Hadoop development iterations are slow. Traditional programmers have always had the benefit of re-compiling their app, running it, and seeing the results within seconds. They have near instant validation that what they’re building is actually working. When you’re working with Hadoop, dealing with gigabytes of data, your development iteration time is more like hours.
Inspired by the amazing real-time feedback experience of Light Table, we’ve built Mortar Watchtower to bring back that almost instant iteration cycle developers are used to. Not only that, Watchtower also helps surface the semantics of your Pig scripts, to give you insight into how your scripts are working, not just that they are working.
Instant Feedback
Watchtower is a daemon that sits in the background, continuously flowing a sample of your data through your script while your work. It captures what your data looks like, and shows how it mutates at each step, directly inline with your script.
I am not sure about the “…helps surface the semantics of your Pig scripts…,” but just checking scripts against data is a real boon.
I continue to puzzle over how the semantics of data and operations in Pig scripts should be documented.
Old style C comments seem out of place in 21st century programming.
I first saw this at Alex Popescu’s Watchtower – Instant feedback development tool for Pig.