Portal de Periódicos da CAPES

Inspector gadget

2011; Association for Computing Machinery; Volume: 4; Issue: 12 Linguagem: Inglês

10.14778/3402755.3402758

2150-8097

Christopher Olston, Benjamin Reed,

Software System Performance and Reliability

We consider how to monitor and debug query processing dataflows, in distributed environments such as Pig/Hadoop. Our work is motivated by a series of informal user interviews, which revealed that monitoring and debugging needs are both pressing and diverse. In response to these interviews, we created a framework for custom dataflow instrumentation, called Inspector Gadget (IG). IG makes it easy to write a wide variety of monitoring and debugging behaviors, and attaches seamlessly to an existing, unmodified dataflow environment such as Pig. We have implemented a dozen user-requested tools in Inspector Gadget, each in just a few hundred lines of Java code. The performance overhead is modest in most cases. Our Pig-based implementation of IG, called Penny , is slated for public release in mid-2011, in conjunction with the upcoming Apache Pig v0.9 release.