47. Programs, JSON and the Network¶
Desktop programs on Linux lean on other programs: Imagescan asks scanimage for a picture, Audio asks wpctl for the volume, WritePad hands documents to a converter. And some of them talk to web services: Mail signs in through the browser, Audio looks up CDs and radio stations. This chapter covers running other programs safely, testing without them, reading and writing JSON, and the rules the programs follow on the network, which is not (yet) a part of the library.
Running a program: CommandRunner¶
Package process · introduced in 31. Imagescan · source: CommandRunner
What it is for. Starting another program, and for short commands, running it to its end and collecting what it printed:
ProgramResult result = runner.run(List.of("scanimage", "-L"), Duration.ofSeconds(20));
String volume = runner.output(List.of("wpctl", "get-volume", "@DEFAULT_AUDIO_SINK@"), Duration.ofSeconds(3));
When to choose it. Whenever a program runs another one. Code takes a CommandRunner instead of calling ProcessBuilder itself, so that tests answer with canned output (below) and never depend on what is installed. CommandRunner.system() is the real one:
/** Starts real processes, with the environment of this program. */
static CommandRunner system() {
return command -> {
try {
return new ProcessBuilder(command).start();
} catch (IOException e) {
throw ProgramMissingException.check(command.getFirst(), e);
}
};
}
| Method | For |
|---|---|
start(command) |
A long-running program whose output is a stream, such as a scanned picture; read the Process yourself |
run(command, timeout) |
A short command: exit code and both outputs as bytes, in a ProgramResult |
output(command, timeout) |
A short command whose standard output is text; any failure is an IOException with the program's own message |
How it works. run handles the parts that are easy to get wrong. Standard input is closed at once, so a program that asks a question gets an end of file instead of waiting. Both outputs are read at the same time on virtual threads, because a program that fills one pipe while nobody reads it blocks for good. And the time limit holds even for a program that hangs without printing anything; when it runs out, the program is stopped:
/**
* Runs a command to its end and collects what it printed. Standard input is closed at once, both output
* streams are read at the same time on virtual threads, so that a program that fills one of them does
* not block, and the time limit holds even when the program hangs without printing anything.
*
* @throws ProgramMissingException if the program does not exist
* @throws IOException if it does not end within {@code timeout}; it is stopped then
* @throws InterruptedException if the calling thread is interrupted; the program is stopped then
*/
default ProgramResult run(List<String> command, Duration timeout) throws IOException, InterruptedException {
Process process = start(command);
process.getOutputStream().close();
ByteArrayOutputStream out = new ByteArrayOutputStream();
ByteArrayOutputStream err = new ByteArrayOutputStream();
Thread outReader = Thread.ofVirtual().start(() -> copy(process.getInputStream(), out));
Thread errReader = Thread.ofVirtual().start(() -> copy(process.getErrorStream(), err));
try {
if (!process.waitFor(timeout.toMillis(), TimeUnit.MILLISECONDS)) {
process.destroyForcibly();
throw new IOException(command.getFirst() + " did not finish within " + timeout.toSeconds() + " s");
}
// The streams end with the process; a child that inherited them could keep them open, so the
// readers get a moment, not for ever.
outReader.join(Duration.ofSeconds(5));
errReader.join(Duration.ofSeconds(5));
} catch (InterruptedException e) {
process.destroyForcibly();
throw e;
}
synchronized (out) {
synchronized (err) {
return new ProgramResult(process.exitValue(), out.toByteArray(), err.toByteArray());
}
}
}
output turns every failure into one IOException whose message says what happened, wpctl failed (exit code 1): Connection refused, ready for a status bar:
/**
* Runs a short command and returns its standard output as UTF-8 text.
*
* @throws IOException if the program is missing, fails (exit code other than 0, with its error message),
* takes longer than {@code timeout} or is interrupted
*/
default String output(List<String> command, Duration timeout) throws IOException {
ProgramResult result;
try {
result = run(command, timeout);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException("Interrupted while " + command.getFirst() + " ran", e);
}
if (!result.succeeded()) throw new IOException(result.failure(command.getFirst()));
return result.stdoutText();
}
Pitfall: reading the output after waitFor
The obvious code, waitFor() and then readAllBytes(), works until a program prints more than the pipe holds (64 KB on Linux). The program then waits for a reader, the caller waits for the program, and both wait forever. Read while waiting, as run does (chapter 31).
Pitfall: a shell
List.of("sh", "-c", "scanimage " + device) lets a device name with a ; in it run anything. Pass each argument as its own list element; no shell is involved, and nothing in an argument is interpreted.
Test: the time limit holds for a program that prints nothing
/** A program that hangs without printing anything must not hang its caller: the limit holds. */
@Test
@EnabledOnOs({OS.LINUX, OS.MAC})
void theTimeLimitHoldsForAProgramThatPrintsNothing() {
long start = System.nanoTime();
IOException e = assertThrows(IOException.class,
() -> system.run(List.of("sleep", "30"), Duration.ofMillis(300)));
assertTrue(e.getMessage().contains("did not finish"), e.getMessage());
assertTrue(System.nanoTime() - start < Duration.ofSeconds(10).toNanos());
}
What a program printed: ProgramResult¶
Package process · introduced in 31. Imagescan · source: ProgramResult
What it is for. The exit code and both outputs of a finished program. The outputs are bytes, because standard output may be a picture or text in any encoding; stdoutText() and stderrText() decode them as UTF-8, or in a charset you pass. succeeded() is true for exit code 0.
How it works. failure(program) builds the message for a failed run from the error output, or the standard output when a program prints its errors there:
/** "wpctl failed (exit code 1): Connection refused", from the error output, else the standard output. */
public String failure(String program) {
String detail = stderrText().strip();
if (detail.isEmpty()) detail = stdoutText().strip();
return program + " failed (exit code " + exitCode + ")" + (detail.isEmpty() ? "" : ": " + detail);
}
A program that is not installed: ProgramMissingException¶
Package process · introduced in 31. Imagescan · source: ProgramMissingException
What it is for. Telling "the program is not installed" apart from every other failure. A missing program is a message for the user, usually with the package to install (Imagescan: "The scanning program scanimage is not installed", followed by sudo apt install sane-utils), not an error report.
When to choose it. Catch it where a program offers a feature that needs another program: Imagescan without scanimage, Audio without wpctl. CommandRunner.system() throws it from start, run and output; a test runner can throw it to check the message.
How it works. The JDK reports a missing program as an IOException with "error=2" in its message. check turns that one into a ProgramMissingException and leaves other errors, such as a missing permission, as they are:
/**
* Turns the JDK's "Cannot run program" into a {@code ProgramMissingException}; other errors (no permission,
* no memory for the process) stay as they are.
*/
public static IOException check(String program, IOException e) {
if (e instanceof ProgramMissingException) return e;
String message = String.valueOf(e.getMessage());
// "Cannot run program "x": error=2, No such file or directory" on Linux, "CreateProcess error=2" on Windows
if (message.contains("error=2") || message.contains("No such file")) {
return new ProgramMissingException(program, e);
}
return e;
}
Tests without programs: FinishedProcess¶
Package process · introduced in 31. Imagescan · source: FinishedProcess
What it is for. A Process that has already ended with a given output. Because CommandRunner has one abstract method, a test answers with a lambda that returns one:
When to choose it. In every test of code that runs programs. printing(stdout) is a successful run, failing(exitCode, stderr) a failed one, and the constructor takes both outputs as text or bytes. The lambda can also record the commands, to check what would have been run.
Test: a fake runner that records the commands
@Test
void testsAnswerWithFinishedProcesses() throws Exception {
List<List<String>> commands = new ArrayList<>();
CommandRunner fake = command -> {
commands.add(command);
return command.getFirst().equals("wpctl") ? FinishedProcess.printing("Volume: 0.25")
: FinishedProcess.failing(1, "no such sink");
};
assertEquals("Volume: 0.25", fake.output(List.of("wpctl", "get-volume"), LIMIT));
IOException e = assertThrows(IOException.class, () -> fake.output(List.of("pactl", "info"), LIMIT));
assertEquals("pactl failed (exit code 1): no such sink", e.getMessage());
assertEquals(List.of(List.of("wpctl", "get-volume"), List.of("pactl", "info")), commands);
}
JSON¶
Package json · introduced in 33. Audio · source: Json
What it is for. Reading the answers of web services and token endpoints, and reading and writing small JSON files. Mail reads its sign-in tokens with it; Audio reads MusicBrainz, the radio station directory and its favourites file.
When to choose it. For JSON you read a few members from, and for small files you write. Objects become LinkedHashMaps, so members keep their order, arrays Lists, numbers Doubles. A program that maps large JSON documents onto many classes would want a real JSON library; none of the programs does.
How it works. parse is a small recursive parser and throws IllegalArgumentException on text that is not JSON:
/** Parses any JSON value. */
public static Object parse(String text) {
Json json = new Json(text == null ? "" : text);
Object value = json.value();
json.skipSpace();
if (json.pos != json.text.length()) throw json.error("Trailing characters");
return value;
}
@SuppressWarnings("unchecked")
public static Map<String, Object> parseObject(String text) {
if (!(parse(text) instanceof Map<?, ?> map)) throw new IllegalArgumentException("Not a JSON object");
return (Map<String, Object>) map;
}
@SuppressWarnings("unchecked")
public static List<Object> parseArray(String text) {
if (!(parse(text) instanceof List<?> list)) throw new IllegalArgumentException("Not a JSON array");
return (List<Object>) list;
}
The reading helpers are the reason the class exists. Web services leave members out, and send numbers as strings or strings as numbers. The helpers never throw on a missing or mistyped member, they return "", 0 or an empty list, so reading a release date is one line instead of five checks:
/** String member, or "" if missing or not a string/number. */
public static String str(Object object, String key) {
if (!(object instanceof Map<?, ?> map)) return "";
Object v = map.get(key);
if (v instanceof String s) return s;
if (v instanceof Double d) return d == Math.rint(d) ? Long.toString(d.longValue()) : d.toString();
return v == null ? "" : v.toString();
}
/** Number member (also a number written as a string), or 0. */
public static int integer(Object object, String key) {
if (object instanceof Map<?, ?> map && map.get(key) instanceof Number n) return n.intValue();
try {
return Integer.parseInt(str(object, key).strip());
} catch (NumberFormatException e) {
return 0;
}
}
/** Any member, or null; for nested objects: {@code Json.member(Json.member(o, "b"), "c")}. */
public static Object member(Object object, String key) {
return object instanceof Map<?, ?> map ? map.get(key) : null;
}
/** Array member, or an empty list. */
@SuppressWarnings("unchecked")
public static List<Object> list(Object object, String key) {
return member(object, key) instanceof List<?> l ? (List<Object>) l : List.of();
}
Map<String, Object> token = Json.parseObject(response.body());
String access = Json.str(token, "access_token"); // "" if missing
int seconds = Json.integer(token, "expires_in"); // 0 if missing
String name = Json.str(Json.member(answer, "artist"), "name"); // nested: "" if anything on the way is missing
write(value) writes maps, lists, strings, numbers, booleans and null, indented two spaces, for files people may read. Whole numbers are written without .0.
Test: it writes what it reads
@Test
void writesWhatItReads() {
Map<String, Object> m = new LinkedHashMap<>();
m.put("name", "Tab\tand \"quote\"");
m.put("n", 128);
m.put("list", List.of("a", 1.5, false));
m.put("empty", List.of());
String json = Json.write(m);
assertEquals("Tab\tand \"quote\"", Json.str(Json.parse(json), "name"));
assertEquals(128, Json.integer(Json.parse(json), "n"));
assertEquals(List.of("a", 1.5, false), Json.member(Json.parse(json), "list"));
}
The network¶
Not in amade-commons · see 22. Mail and 33. Audio
amade-commons has no network package, and that is deliberate. Mail and Audio both use the JDK's java.net.http.HttpClient, but what they need around it differs: Mail signs in with OAuth through the browser and loads remote images only when the user asks; Audio has proxy settings of its own, a cache of CD titles and a rate limit for MusicBrainz. What the two share is a handful of rules, not code yet:
package eu.amadeos.apps.audio.service.metadata;
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.Optional;
/** HTTP GET for the metadata services; replaced by a fake in tests. */
public interface HttpFetcher {
String USER_AGENT = "amade-audio/0.1.0 ( https://github.com/datazuul/amade-audio )";
/** The body, or empty for 404; other errors throw. */
Optional<byte[]> get(URI uri, String accept) throws IOException;
static HttpFetcher system() {
HttpClient client = eu.amadeos.apps.audio.service.net.NetworkProxy.INSTANCE.httpClient()
.connectTimeout(Duration.ofSeconds(8))
.followRedirects(HttpClient.Redirect.NORMAL)
.build();
return (uri, accept) -> {
HttpRequest request = HttpRequest.newBuilder(uri)
.header("User-Agent", USER_AGENT)
.header("Accept", accept)
.timeout(Duration.ofSeconds(20))
.GET().build();
try {
HttpResponse<byte[]> response = client.send(request, HttpResponse.BodyHandlers.ofByteArray());
int code = response.statusCode();
if (code == 404) return Optional.empty();
if (code != 200) throw new IOException("HTTP " + code + " from " + uri.getHost());
return Optional.of(response.body());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException("Interrupted", e);
}
};
}
}
- An interface in front of the client. The service takes an
HttpFetcher, not anHttpClient, so its tests answer with canned bytes and never go online, just asCommandRunnerkeeps tests away from installed programs. - Two time limits. A connect timeout for a server that does not answer at all, and a request timeout for one that answers and then stalls. Without them a request can wait for minutes.
- A
User-Agentthat names the program and where to reach its authors. Services run by volunteers, such as MusicBrainz, ask for it, and block clients that hide. - Off the EDT. Every request runs on the background executor; the result comes back through the
@Uiexecutor (chapter 41). - Never online by itself. A program goes online when the user asked for something that needs it: a CD lookup, a station search, images in a message. Web, the documentation viewer, never goes online at all (chapter 11).
- Answers are read with
Json's forgiving helpers, because a member that a service leaves out must not break the program.
When a third program needs the same client, the shared part (time limits, User-Agent, the proxy) will move into amade-commons, as every other part of it did.