The tar utility program supports the option to use a 3rd party compression program with --use-compress-program. I'm currently alternating between lbzip2 and pbzip2 as options since they both support multi-threading multiple files passed to it.

The tar command is put into an Execute Shell step as follows:

tar --use-compress-program lbzip2 -cf test.tar.bzip2 app_directory

This works fine through command line:

$ tar --use-compress-program lbzip2 -cf test.tar.bzip2 app_directory
$ ls -l
total 1149104
drwxr-xr-x 3 user 832595117 102 Nov 4 10:09 app_directory
rw-rr- 1user 832595117 588338126 Nov 7 20:35 test.tar.bzip2

But the Jenkins job just flags failure:

+ tar -v --use-compress-program lbzip2 -cf test.tar.bzip2 app_directory
a app_directoryBuild step 'Execute shell' marked build as failure

Note that a normal bzip2 operation works, so I suspect that somehow the multi-threading of lbzip2 and pbzip2 (I tested both) have something to do with it, but the tar program itself doesn't exit right away with a non-zero value from what I can tell through command line.

I don't understand why Jenkins fails on that line instantly. Even having an echo $? after it, but it doesn't reach that line.