Accessing Nested Data Structures in Java

Enterprise data structures are not usually flat. More often they are hierarchical, nesting one or more levels deep. For example, an account object might contain a customer object, which might contain an address object, and so on.

Such data structures are often returned by web services. A single call to the server requires less code and incurs less network overhead than multiple calls to retrieve the same information. Often, these data structures are returned as JSON and mapped to strongly typed equivalents such as Java beans on the client side. This works well when modeling the complete server response; however, it can be overkill when only a small subset of the returned data is required.

For example, the sample user service at typicode.com returns a collection of records structured like this:

{
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "Sincere@april.biz",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "phone": "1-770-736-8031 x56442",
  "website": "hildegard.org",
  "company": {
    "name": "Romaguera-Crona",
    "catchPhrase": "Multi-layered client-server neural-net",
    "bs": "harness real-time e-markets"
  }
} 

It would be straightforward to map this content to a bean representation. However, if the caller is only interested in the "catchPhrase" values, for example, it would still require the following class definitions at a minimum:

public static class User {
    private Company company;

    public Company getCompany() {
        return company;
    }

    public void setCompany(Company company) {
        this.company = company;
    }
}

public static class Company {
    private String catchPhrase;

    public String getCatchPhrase() {
        return catchPhrase;
    }

    public void setCatchPhrase(String catchPhrase) {
        this.catchPhrase = catchPhrase;
    }
}

Using HTTP-RPC‘s WebServiceProxy class, the following code could then be used to bind the response data to bean instances, extract the catch-phrase values, and convert them to a JSON list:

URL url = new URL("https://jsonplaceholder.typicode.com/users");

List<User> users = WebServiceProxy.get(url).invoke(BeanAdapter.typeOf(List.class, User.class));

List<String> catchPhrases = users.stream()
    .map(user -> user.getCompany().getCatchPhrase())
    .collect(Collectors.toList());

JSONEncoder jsonEncoder = new JSONEncoder();

jsonEncoder.write(catchPhrases, System.out);

For example:

[
  "Multi-layered client-server neural-net",
  "Proactive didactic contingency",
  "Face to face bifurcated interface",
  "Multi-tiered zero tolerance productivity",
  "User-centric fault-tolerant solution",
  "Synchronised bottom-line interface",
  "Configurable multimedia task-force",
  "Implemented secondary concept",
  "Switchable contextually-based project",
  "Centralized empowering task-force"
]

Alternatively, interfaces could be used in place of the bean types to eliminate some of the boilerplate code:

public interface User {
    Company getCompany();
}

public interface Company {
    String getCatchPhrase();
}

The code for extracting the catch-phrases would be identical to the previous example, and the resulting output would be the same:

URL url = new URL("https://jsonplaceholder.typicode.com/users");

List<User> users = WebServiceProxy.get(url).invoke(BeanAdapter.typeOf(List.class, User.class));

List<String> catchPhrases = users.stream()
    .map(user -> user.getCompany().getCatchPhrase())
    .collect(Collectors.toList());

JSONEncoder jsonEncoder = new JSONEncoder();

jsonEncoder.write(catchPhrases, System.out);

A third option would be to deserialize the "raw" JSON data and access the catch-phrases via successive calls to Map#get():

URL url = new URL("https://jsonplaceholder.typicode.com/users");

List<Map<String, Map<String, ?>>> users = WebServiceProxy.get(url).invoke();

List<String> catchPhrases = users.stream()
    .map(user -> (String)user.get("company").get("catchPhrase"))
    .collect(Collectors.toList());

JSONEncoder jsonEncoder = new JSONEncoder();

jsonEncoder.write(catchPhrases, System.out);

This uses less code than the bean or interface approaches, but still requires the declaration of a moderately complex generic, even for this fairly simple case.

A fourth alternative would be to use the valueAt() method of HTTP-RPC’s Collections class to access the nested values by key path:

URL url = new URL("https://jsonplaceholder.typicode.com/users");

List<?> users = WebServiceProxy.get(url).invoke();

List<String> catchPhrases = users.stream()
    .map(user -> (String)Collections.valueAt(user, "company", "catchPhrase"))
    .collect(Collectors.toList());

JSONEncoder jsonEncoder = new JSONEncoder();

jsonEncoder.write(catchPhrases, System.out);

This approach is the least verbose, as it allows the caller to retrieve the desired data directly, without the need for intermediate types or nested generics.

If a caller needs access to most or all of the data returned by a service, then binding to bean or interface types is probably the most practical solution. However, if access to only a targeted subset of nested data is required (e.g. for lightweight transformation or basic validation), then the generic map or valueOf() approach may be preferable.

For more information, see the project README.

Introducing QueryBuilder

Programmatically constructing SQL queries is a common but cumbersome task. For example, consider this query that retrieves information about a hypothetical IoT device:

select Device.*, Building.name as buildingName, Site.name as siteName 
from Device 
join Building on buildingID = Building.id
join Site on siteID = Site.id
where Device.id = :deviceID

Written in plain text, as above, the query is fairly easy to read and understand. However, in Java, the same query might be declared as follows:

String sql = "select Device.*, Building.name as buildingName, Site.name as siteName " 
    + "from Device "
    + "join Building on buildingID = Building.id "
    + "join Site on siteID = Site.id "
    + "where Device.id = :deviceID";

Even in this simple example, readability is negatively impacted by the addition of double quotes and string concatenation operators. Care must be taken to ensure that whitespace is properly managed. Longer, more complex, queries become increasingly difficult to write and maintain.

The new QueryBuilder class introduced in HTTP-RPC 7.5 simplifies the task of writing SQL queries in Java. Using QueryBuilder, the preceding example could be rewritten as follows:

String sql = QueryBuilder.select("Device.*", "Building.name as buildingName", "Site.name as siteName")
    .from("Device")
    .join("Building").on("buildingID = Building.id")
    .join("Site").on("siteID = Site.id")
    .where("Device.id = :deviceID").toString();

Because SQL verbs and clauses are represented as Java methods and nouns (tables, columns, and predicates) by strings, this version is much more readable and easier to maintain. It also scales better as query complexity increases.

A complete example using QueryBuilder along with HTTP-RPC’s Parameters and ResultSetAdapter classes is shown below. Parameters is used to prepare SQL statements for execution using named parameter values rather than indexed arguments. ResultSetAdapter provides access to the contents of a result set via the Iterable and Map interfaces, facilitating direct serialization of the result data to JSON:

String sql = QueryBuilder.select("Device.*", "Building.name as buildingName", "Site.name as siteName")
    .from("Device")
    .join("Building").on("buildingID = Building.id")
    .join("Site").on("siteID = Site.id")
    .where("Device.id = :deviceID").toString();

Parameters parameters = Parameters.parse(sql);

try (PreparedStatement statement = connection.prepareStatement(parameters.getSQL())) {
    parameters.apply(statement, mapOf(
        entry("deviceID", deviceID)
    ));

    try (ResultSetAdapter resultSetAdapter = new ResultSetAdapter(statement.executeQuery())) {
        JSONEncoder jsonEncoder = new JSONEncoder();

        jsonEncoder.write(resultSetAdapter, System.out);
    }
}

For more information, see the project README.

Transforming XML with Mustache Templates

Although XSLT is a powerful and flexible way to convert XML documents to alternate representations, it is often difficult and inconvenient to use in practice. The XSLT specification is large and complex, and platform support is inconsistent.

As of HTTP-RPC 7.4, the TemplateEncoder class can be used to transform XML using a lightweight syntax similar to Mustache. The new ElementAdapter class provides access to the contents of an XML DOM Element via the Map interface. The contents of this map can be easily transformed to another representation via a template document.

For example, this tutorial from W3 Schools uses XML to model a simple breakfast menu:

<?xml version="1.0" encoding="UTF-8"?>

<breakfast_menu>
    <food>
        <name>Belgian Waffles</name>
        <price>$5.95</price>
        <description>Two of our famous Belgian Waffles with plenty of real maple syrup</description>
        <calories>650</calories>
    </food>

    <food>
        <name>Strawberry Belgian Waffles</name>
        <price>$7.95</price>
        <description>Light Belgian waffles covered with strawberries and whipped cream</description>
        <calories>900</calories>
    </food>

    <food>
        <name>Berry-Berry Belgian Waffles</name>
        <price>$8.95</price>
        <description>Light Belgian waffles covered with an assortment of fresh berries and whipped cream</description>
        <calories>900</calories>
    </food>

    <food>
        <name>French Toast</name>
        <price>$4.50</price>
        <description>Thick slices made from our homemade sourdough bread</description>
        <calories>600</calories>
    </food>

    <food>
        <name>Homestyle Breakfast</name>
        <price>$6.95</price>
        <description>Two eggs, bacon or sausage, toast, and our ever-popular hash browns</description>
        <calories>950</calories>
    </food>
</breakfast_menu>

The example uses the following XSLT template to transform the markup to HTML. Even in this fairly simple example, the verbosity of XSLT is obvious, and has a negative impact on readability:

<?xml version="1.0" encoding="UTF-8"?>
<html xsl:version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <body style="font-family:Arial;font-size:12pt;background-color:#EEEEEE">
        <xsl:for-each select="breakfast_menu/food">
            <div style="background-color:teal;color:white;padding:4px">
                <span style="font-weight:bold"><xsl:value-of select="name"/> - </span>
                <xsl:value-of select="price"/>
            </div>
            <div style="margin-left:20px;margin-bottom:1em;font-size:10pt">
                <p>
                    <xsl:value-of select="description"/>
                    <span style="font-style:italic"> (<xsl:value-of select="calories"/> calories per serving)</span>
                </p>
            </div>
        </xsl:for-each>
    </body>
</html>

The javax.xml.transform.Transformer class can be used to apply the template to the XML document:

Source source = new StreamSource(getClass().getResourceAsStream("breakfast_menu.xslt"));

Transformer transformer = TransformerFactory.newInstance().newTransformer(source);

Source xmlSource = new StreamSource(getClass().getResourceAsStream("breakfast_menu.xml"));

File homeDirectory = new File(System.getProperty("user.home"));
File outputFile = new File(homeDirectory, "breakfast_menu_1.html");

try (FileOutputStream outputStream = new FileOutputStream(outputFile)) {
    transformer.transform(xmlSource, new StreamResult(outputStream));
}

The result of the transformation is shown below:

XSLT result

The following is an example of how a similar template can be defined using Mustache-style syntax. It is much simpler and easier to read than the previous version, and can be edited using any HTML-aware text editor:

<!-- Breakfast Menu -->
<html>
    <body style="font-family:Arial;font-size:12pt;background-color:#EEEEEE">
        <!-- {{#food*}} -->
            <div style="background-color:teal;color:white;padding:4px">
                <span style="font-weight:bold">{{name}} - </span>
                {{price}}
            </div>
            <div style="margin-left:20px;margin-bottom:1em;font-size:10pt">
                <p>
                    {{description}}
                    <span style="font-style:italic"> ({{calories}} calories per serving)</span>
                </p>
            </div>
        <!-- {{/food*}} -->
    </body>
</html>

This code applies the template to the XML document using TemplateEncoder:

DocumentBuilder documentBuilder = DocumentBuilderFactory.newInstance().newDocumentBuilder();

Document document;
try (InputStream inputStream = getClass().getResourceAsStream("breakfast_menu.xml")) {
    document = documentBuilder.parse(inputStream);
}

TemplateEncoder templateEncoder = new TemplateEncoder(getClass().getResource("breakfast_menu.html"));

File homeDirectory = new File(System.getProperty("user.home"));
File outputFile = new File(homeDirectory, "breakfast_menu_2.html");

try (FileOutputStream outputStream = new FileOutputStream(outputFile)) {
    templateEncoder.write(new ElementAdapter(document.getDocumentElement()), outputStream);
}

The result is identical to the XSLT version:

Mustache result

Simplicity isn’t the only advantage of the Mustache-style approach. On average, the XSLT Transformer version takes about 350ms to run, compared to about 30ms for TemplateEncoder – about a 10x difference.

For more information about HTTP-RPC, TemplateEncoder, and ElementAdapter, see the project README.

Efficiently Transforming JDBC Query Results to JSON

3/6/2021 Updated for HTTP-RPC 8.1

A lot of enterprise data is stored in relational databases and accessed via SQL queries. Many web services are little more than HTTP-based wrappers around such queries.

Unfortunately, transforming query results to JSON so it can be consumed by a client application often involves numerous inefficient steps, such as binding each row to a data object and loading the entire data set into memory before serializing it back to the caller. This type of approach has a negative impact on performance and scalabilty. Each row requires multiple heap allocations and constructor invocations, increasing latency and CPU load. Worse, the caller does not receive a response until the entire data set has been processed.

Further, since each response is loaded entirely into memory, high-volume applications require a large amount of RAM, and can only scale through the addition of more physical hardware. Eventually, the garbage collector has to run, slowing down the entire system.

A much more efficient approach is to stream response data. Instead of copying the query results into an in-memory data structure before sending the response, the web service can write a row of data to the output stream each time a row is read from the result set. This allows a client to begin receiving the data as soon as it is available, significantly reducing latency. Also, because no intermediate data structures are created, CPU and memory load is reduced, allowing each server to handle a higher number of concurrent requests. Finally, because fewer heap allocations are required, the garbage collector needs to run much less frequently, resulting in fewer system pauses.

Introducing HTTP-RPC

HTTP-RPC is an open-source framework for implementing REST services in Java. It is extremely lightweight and requires only a Java runtime environment and a servlet container. The entire framework is distributed as a single JAR file that is about 100KB in size, making it an ideal choice for applications where a minimal footprint is desired.

WebService

HTTP-RPC’s WebService type provides an abstract base class for REST-based web services. It extends the similarly abstract HttpServlet class provided by the servlet API.

Service operations are defined by adding public methods to a concrete service implementation. Methods are invoked by submitting an HTTP request for a path associated with a servlet instance. Arguments are provided either via the query string or in the request body, like an HTML form. WebService converts the request parameters to the expected argument types, invokes the method, and writes the return value to the output stream as JSON.

The RequestMethod annotation is used to associate a service method with an HTTP verb such as GET or POST. The optional ResourcePath annotation can be used to associate the method with a specific path relative to the servlet. For example, the following class might be used to implement a web service that performs a simple addition operation:

@WebServlet(urlPatterns={"/math/*"})
public class MathServlet extends WebService {
    @RequestMethod("GET")
    @ResourcePath("/sum")
    public double getSum(double a, double b) {
        return a + b;
    }
}

The following request would cause the method to be invoked, and the service would return the value 6 in response:

GET /math/sum?a=2&b=4

JSONEncoder

The JSONEncoder class, which is used internally by WebService to serialize response data, converts return values to their JSON equivalents as follows:

  • CharSequence: string
  • Number: number
  • Boolean: true/false
  • Enum: string
  • java.util.Date: number representing epoch time in milliseconds
  • java.time.TemporalAccessor: string
  • java.net.URL: string
  • Iterable: array
  • java.util.Map: object

Note that collection types are not required to support random access; iterability is sufficient. This is an important feature, as it allows service implementations to stream result data rather than buffering it in memory before it is written.

ResultSetAdapter

HTTP-RPC’s ResultSetAdapter class implements the Iterable interface and makes each row in a JDBC result set appear as an instance of Map, allowing query results to be efficiently serialized as an array of JSON objects.

For example, consider a web service that returns the result of a SQL query on this table, taken from the MySQL sample database:

CREATE TABLE pet (
    name VARCHAR(20),
    owner VARCHAR(20),
    species VARCHAR(20),
    sex CHAR(1),
    birth DATE,
    death DATE
);

A method to retrieve a list of all pets belonging to a given owner might be implemented as shown below. Note that the example uses HTTP-RPC’s Parameters class to simplify query execution using named parameters rather than positional values. Also note that the method uses JSONEncoder to explicitly write the results to the output stream rather than simply returning the adapter instance, to ensure that the underlying result set is closed and system resources are not leaked (see this article for more information on QueryBuilder):

@RequestMethod("GET")
public void getPets(String owner, String format) throws SQLException, IOException {
    Parameters parameters = Parameters.parse(QueryBuilder.select("name", "species", "sex", "birth")
        .from("pet")
        .where("owner = :owner").toString());

    try (Connection connection = dataSource.getConnection();
        PreparedStatement statement = connection.prepareStatement(parameters.getSQL())) {
        parameters.apply(statement, mapOf(
            entry("owner", owner)
        ));

        try (ResultSetAdapter resultSetAdapter = new ResultSetAdapter(statement.executeQuery())) {
            getResponse().setContentType("application/json");

            JSONEncoder jsonEncoder = new JSONEncoder();

            jsonEncoder.write(resultSetAdapter, getResponse().getOutputStream());
        }
    }
}

A response produced by the method might look something like this, where each object in the array represents a row from the result set:

[
  {
    "name": "Claws",
    "species": "cat",
    "sex": "m",
    "birth": 763880400000
  },
  {
    "name": "Chirpy",
    "species": "bird",
    "sex": "f",
    "birth": 905486400000
  },
  {
    "name": "Whistler",
    "species": "bird",
    "sex": null,
    "birth": 881643600000
  }
]

With just a few lines of code, query results can be quickly and efficiently returned to the caller, with no intermediate buffering required.

More Information

Complete source code for this example can be found here. For more information, see the project README.

Named Parameters in JDBC Queries

2/15/2021 Updated for HTTP-RPC 7.6

Prepared statements are a common way to execute parameterized queries in JDBC. For example, the following SQL might be used to retrieve a list of all users whose first or last name matches a particular character sequence:

SELECT * FROM user 
WHERE first_name LIKE ? or last_name LIKE ?

Parameter values are supplied at runtime via indexed setter methods defined by the PreparedStatement class:

statement.setString(1, pattern);
statement.setString(2, pattern);

This works fine for simple queries, but it becomes increasingly difficult to manage as the number of parameters grows. It is also redundant – although this query only requires a single argument, two parameter values must be supplied.

The Java Persistence API (JPA) provides a more convenient alternative using named parameters. For example, the above query might be written as follows in JPQL:

SELECT u FROM User u 
WHERE u.firstName LIKE :pattern or u.lastName LIKE :pattern

This is more readable and less verbose, as the caller only needs to provide the value of the “pattern” parameter once. It is also more resilient to changes, as the arguments are not dependent on ordinal position. Unfortunately, it requires a JPA-compliant object-relational mapping (ORM) framework such as Hibernate, a dependency that may not be satisfiable in some situations.

The Parameters class provided by the HTTP-RPC framework brings named parameter support to JDBC. The parse() method of this class is used to create a Parameters instance from a JPA-like SQL query; for example:

SELECT * FROM user 
WHERE first_name LIKE :pattern or last_name LIKE :pattern

It takes a string or reader containing the query text as an argument:

Parameters parameters = Parameters.parse(query);

The getSQL() method of the Parameters class returns the processed query in standard JDBC syntax. This value can be used in a call to the connection’s prepareStatement() method:

PreparedStatement statement = connection.prepareStatement(parameters.getSQL());

Argument values are specified via the apply() method, as shown below (mapOf() is a convenience method provided by the framework for declaratively instantiating maps values, similar to Map.of() in Java 9):

parameters.apply(statement, mapOf(
entry("pattern", pattern)
));

Once applied, the query can be executed:

ResultSet resultSet = statement.executeQuery();

Note that the Parameters class is not limited to queries; it can also be used for updates.

A complete example can be found here. It is a simple REST service that allows a caller to search a database of pets by owner name.

See the project README for more information.

Regular Expression to Validate a Comma-Separated List of Email Addresses

Recently, I needed to create a regular expression to validate the format of a comma-separated list of email addresses. Just thought I’d share the result in case it is of use to anyone. The pattern I came up with for matching a single address is as follows:

[\w-]+(\.[\w-]+)*@[\w-]+(\.[\w-]+)+

Note that I didn’t attempt to handle every possible combination of valid characters – only what I’d consider the most common ones.

The pattern is simply duplicated for the comma-separated list. Here’s an example of applying the pattern in Java:

Pattern addressListPattern = Pattern.compile(String.format("%1$s(,\\s*%1$s)*",
    "[\\w-]+(\\.[\\w-]+)*@[\\w-]+(\\.[\\w-]+)+"),
    Pattern.UNICODE_CHARACTER_CLASS);

System.out.println(addressListPattern.matcher("xyz").matches()); // false
System.out.println(addressListPattern.matcher("foo@bar.com").matches()); // true
System.out.println(addressListPattern.matcher("foo@bar.com, xyz").matches()); // false
System.out.println(addressListPattern.matcher("foo@b-ar.com, f.oo@b-ar.co.uk").matches()); // true
System.out.println(addressListPattern.matcher("føo@b-ar.com, f-oo@b-ar.co.uk").matches()); // true

Note that Pattern.UNICODE_CHARACTER_CLASS is not necessary (or supported) in Android, which supports Unicode character matching by default.

Hope it helps!