Do you remember the last time you though about making a command line interface (CLI) tool in java, and gave up because you just “could not do” something you really wanted (or needed) in there? No? But I do, but I also think problems in the category of “but you cannot do that in programming language X” are there to be busted (especially if I want to).

It all began when I got the idea of porting gittool to java, first try was back in 2014, but then I stranded at the same problems as I’m addressing here. The change now? This time I decided to make the tooling needed to make proper CLI tools with advanced terminal output first. And using proper Java 8, so I can use lambdas and streams and other pretty new java tooling.

Command Line Parser

Before I started out I needed a proper argument parsing library. There are probably quite a few available, but not many are “commonly” used. The 3 most widespread (and recommended) java CLI libraries I cound find was:

  • args4j: Pretty good, and really easy to use CLI argument parsing library. Supports simple “named” arguments and simple aliases. And a pretty complex system of handling ‘arguments’ (as opposed to ‘options’ are without the ‘–name’ distinction. But Does not support short option chaining, or java properties-like options.
  • commons-cli: Old and robust GNU-compliant CLI argument parsing, and has some other CLI tooling available. It is built around the ‘short’ options as it’s base, but has some limitations.
    • Since it is really backward compliant, it also does not support lots of the java 8 features, AFAIK it’s compiled with 1.6, and most of the code is even compatible to 1.5 or 1.4.
    • commons-cli requires pretty much boiler-plate that args4j has managed to solve, and requires you to verify and convert most of the values yourselves.
  • JCommander: An old-style java CLI library with java style options syntax, but is pretty much not updated for the last 5 years (a few critical bugs only). Good documentation was hard to find, so I simply don’t know the exact features it support.

But the real alternatives are args4j and commons-cli. But both of them have a specific feature that I’m not such a big fan of. That is that it requires an intermediate options object instance before the options can be reacted to. And commons-cli even just stores all the flag values (mostly as string values) internally. This means you have to go through the flags after parsing and move / push the values wherever they belong.

Note that args4j support OptionHandler’s that can arbitrarily reformat or instantiate special types for the option, but the values are still bound to the Options instance you declare for the parser.

Rich Terminal Input/Output

Another problem with java and “advanced” console output is that it is truly platform independent. Therefor it has no notion of a “terminal device”. But it has a very simple Console interface, which is enabled if (and only if) both the input and output is an interactive terminal, i.e. PowerShell in windows or bash in Linux. The problem hare is if you have only 1 of the two as interactive (e.g. used in scripting, piping etc), then the Console simply disappears. This requires more advanced CLI tools to handle both the interactive mode vs scripting mode of I/O.

But if we put platform independence out of the question, it is possible to control the terminal in java, you just need to cheat… This is done by calling the $ stty [.*] < /dev/tty from a shell, from a java sub-process. Essentially doing this:

public class STTY {
    // switch to RAW mode
    public static void raw() {
        Runtime.getRuntime().exec(new String[]{"/bin/sh", "-c", "stty raw -echo </dev/tty"});
    }

    // switch to COOKED mode
    public static void cooked() {
        Runtime.getRuntime().exec(new String[]{"/bin/sh", "-c", "stty -raw echo </dev/tty"});
    }
}

This way the terminal mode can be switched between RAW and COOKED, so we can switch between simple terminal output, and interactive mode where we can control the input echoing and the cursor (for re-printing and replacing lines in the terminal).

NOTE: The stty command will only work on Unix based systems, and may need some tweaking to behave nicely on bsd / mac (I have only tested it on linux so far).

Parsing Rich Terminal Input

Since when you press keys in the terminal, you will get a mixture of simple character keys (using ASCII or UTF-8 encoding) various control characters (mostly ASCII characters) and control sequences, which is a combination of a number of characters (after an initial ESC char), we also need a system to parse these “back” into key-press-like events.

Thankfully, the java input stream is a pretty simple (and raw) input stream. It will read one byte at a sime, as the terminal writes it, until the stream is closed. And I had done my homework this time. Between last time and this I had implemented a C++ library which essentially did the same (a small project I did to learn the new features of C++11).

Essentially:

  • '\033', the ESC character can be used to note the beginning of a control sequence, which can consist of up to 10-15 more characters.
  • If the input stream has no available bytes after the ESC byte, it is a single press of the Escape key instead, as for other control sequences, they are sent at the same time to standard in.
  • Control sequences can be grouped in two broad types; Terminal Control Modifiers, which have the '[' character after the ESC, and special keys, which use one or two ASCII letters (O + one, or any but O). these are usually used to either modify the output or input, and include test modifiers, cursor movements, and print control. E.g.:
    • Colors have the syntax: '\033[34;01m', where the numbers 34 and 01 are the text modifiers, or '\033', '[', a list of modifiers separated with ';', and 'm' at the end, and cursor movements (or arrow key pressing) ends with 'A' through 'C'. Except for colors, the modifier numbers are optional.
    • The lower F keys use the special key with 2 letters, e.g. F1 is '\033OP', but the higher use the compound syntax, e.g. F5 is '\033[15~'.
    • Also note that some keys have alternative control sequences, depending on the terminal type (or even platform), e.g. the ‘backspace’ key will return '\177' on most linux terminals, but '\010' on BSD and mac. And home and end have alternative control sequences as well.

But this is mostly just fiddly work, and not that difficult to make sense of in the first place, but it would be really good to put it into a library. E.g. to handle the alternatives, and ensure consistent multiple platform support.

Speed and binary size.

The last challenge for making proper CLI tools, is that you usually would want to have quite a couple of them, and still not fill the disk with unused java classes that in the end takes extra memory and time loading. In order to create a truly minimized jar, you can use the maven-shade-plugin. There are also optimize the runscript for tweaking the program startup parameters.

Maven shade plugin

The first rule of speeding up java is to make a stripped down fat jar file. This is simply done by using the maven-shade-plugin (what’s up with that name?). And secondly to strip down the dependencies in general. See example here.

Using the <minimizeJar> option can shave off up to 50% or more of the jar file size, but may need to be handled carefully, as it may remove classes that are only indirectly referenced via reflection. For example in providence-rpc-tool the jar was minimized from 5.0Mb to 2.8Mb (or a 44% reduction).

Proguard Maven plugin

The proguard-maven-plugin can trim the shaded jar even mode, but it is really tricky to set up correctly (I could not do it for providence-rpc-tool). The shade plugin will only minimize down based off of class references, but will do no trimming within each class, so if you use a single utility method in a class, you will need to drag in dependencies referenced anywhere within that class. What proguard does is to look at the actual method calls (you may need to force include some class methods manually, especially if proguard cannot determine what it references, e.g. with slf4j logging). See example here.

Run scripts

Tuning startup time further can be done by managing the memory. Since java 8 there is no real difference between “client” and “server” runs, so what is left is the options to tweak memory and garbage collection. If the tool is pretty fast, the GC will not have much to say, but the memory settings really may.

  • -Xmx set’s the maximum memory. For anything that is not 1st gen short-lived objects it should probably never be garbage collected. Make sure your max memory supports this. Mostly you would not need this.
  • -Xms is the initial allocated heap size. This needs to be enough to load the base of your tool, but reserving this memory (especially if really big), will slow down the startup time of the tool, but if your app allocates much more memory than this early during startup, you will slow down again (for allocating more heap space again).

Also there is a point about running the jar from the script, or to replace the script with the java -jar process. By using the bash exec function you will replace the script shell with the java process instead of forking a new process to run the command. This can also save a little bit of startup time.

#!/bin/bash

exec java -Xms128m -jar /usr/local/lib/my-tool.jar "$@"

Conclusion

In the end I can conclude that is it totally feasible to make basically any kind of CLI tool in java. But getting down the initial starting time seems to be one of the last remaining hurdles, but at <100ms it should be bearable, but makes it less useful as a sub-tool used heavily in scripting, unless what the job it does is also somewhat more compute time.

With a pretty large memory footprint, even just from the JVM, and an initial startup time that is pretty difficult to get rid of, java has a limited set of useful CLI tooling uses. And it is probably centered around more complex interaction models (with user input) or tasks where using a java library is a real boost, or the only alternative.

When it comes to trimming dependencies and limiting the runtime to load, java 9’s jigsaw modules has the potential to make this much faster, as we can further reduce the number of system classes to be made available.


Most of these (except for the speed and binary size parts) are available in the console-util library, with additions of an argument parser that utilized java functional interfaces, and interactive input and output helpers for the terminal.

And I will now be able to try again at my pet project with java-8. :-D