Monday, March 12, 2007

Groovy Process Piping

I was playing around with using Groovy as a shell scripting language. While convenient, I found that shelling out to my trusted unix commands is many times faster for file operations than handling them within Groovy.

Creating and executing a process is trivial in groovy:

p = "ls -l".execute()
println p.text
But using pipes or redirection is not allowed:

p = "ls -l | sort".execute()
println p.text
For some reason it took me awhile to understand the java.lang.Process well enough to come up with this script, which contains a helper class for piping one Unix shell command into another.
def class Pipe {
private Process p

def Pipe(cmd) {
p = cmd.execute()
p.out.close()
}

def to(cmd) {
def p2 = cmd.execute()
p2.out << p.in
p2.out.close()
p = p2
this
}

def text() {
p.text
}
}

def output = new File('output.txt')
output.delete()

output << new Pipe("grep -h -e '^.* ERROR .*' input.txt")
.to("sort")
.to("tail -n50").text()


Note 1: The Process object is in the standard JDK so with a bit of alteration, one could create the Pipe utility in straight-up Java. But in regular Java I wonder how often one needs to run a shell command?

Note 2: If you are on Windows and are using Cygwin you'll need to be aware that the Windows "sort" command clobbers the Unix version and is hardly as robust. You may need to hard code which sort you want to use: "C:\\cygwin\\bin\\sort"

8 comments:

Yuri Schimke said...

I did some work on this usage in groovy about 3 years ago. It's unlikely to be still working, but probably wouldn't take too much effort to get it working.

def f = gsh.find('.', '-name', '*.java', '-ls');
f.pipeTo(lines);

http://groovy.codehaus.org/Process

CharlesAnderson said...

Did you try something like

p=['sh','-c','ls -l | sort'].execute()
println p.text

if it's winblows, I'd give up hope, of doing any meaningful command piping. But it does work

groovy> p=['cmd','/c','dir|sort'].execute()
groovy> println p.text
groovy> go


2 Dir(s) 13,654,196,224 bytes free
16 File(s) 21,723 bytes
Directory of C:\java\groovy-1.0\bin
Volume in drive C is WXPPRO
Volume Serial Number is A888-C3EA
01/03/2007 02:17 AM 491 groovyConsole.bat
01/03/2007 02:17 AM 494 groovy.bat
01/03/2007 02:17 AM 500 groovysh.bat
01/03/2007 02:17 AM 504 grok.bat
01/03/2007 02:17 AM 517 java2groovy.bat
01/03/2007 02:17 AM 518 groovyc.bat
01/03/2007 02:17 AM 571 groovyConsole
01/03/2007 02:17 AM 584 groovysh
01/03/2007 02:17 AM 589 grok
01/03/2007 02:17 AM 602 groovyc
01/03/2007 02:17 AM 605 java2groovy
01/03/2007 02:17 AM 736 groovy
01/03/2007 02:17 AM 1,583 startGroovy_cygwin
01/03/2007 02:17 AM 4,267 startGroovy.bat
01/03/2007 02:17 AM 7,359 startGroovy
03/12/2007 11:28 AM 1,803 out.xml
03/12/2007 11:28 AM <DIR> .
03/12/2007 11:28 AM <DIR> ..


===> null

groovy>

devilelephant said...

Thanks guys. I'll give your suggestions a try. There is the Groosh module for Groovy that I discovered after this post.

http://groovy.codehaus.org/Process

It seems to do what I wanted but probably better.

Anonymous said...

The one from Charles Anderson doesnt return the return code of the last thing piped to. This is inconvenient. Are we trying to find an easier way to do this?

def proc = 'ps -ef'.execute().pipeTo('grep nweb'.execute()).pipeTo('grep 7999'.execute())

proc.waitFor()

println proc.exitValue()

It doesnt seem that hard or unintuitive... or is this what you meant my "groosh module"?

Chris

devilelephant said...

Your example is intuitive but does not handle all shelling-to-unix use cases.

My original post was March 07 before piping via groovy was well documented. Still when I want to take the output of a process and dump it to a file:
ls | grep 'something' > file.txt

the groovy approach falls short. Unix does this much faster than piping the output to a FileWriter.

Anonymous said...

This has saved me alot of time. Thanks for putting this up.

Anthos said...

You ask: "in regular Java I wonder how often one needs to run a shell command?"
I'll answer: unfortunately sometimes you have to, because if your chief wants you to absolutely make a set of linux shell commands portable, you must do it.
I'll try using your solution :) thank you

Anthos said...

...this doesn't mean it makes sense ;-)