Introduction
In an earlier article I mentioned briefly some possibilities for bringing python-trained machine learning models into production. Specifically, a java environment. In this article I will take a closer look at how this can done and what pitfalls we should avoid.
Keras
In the words of the Keras website, Keras offers:
“consistent and simple APIs; it minimizes the number of user actions required for common use cases.”
For that reason it is very popular and is often a front-end for a Tensorflow backend. What can we do to deploy Keras-trained models in a java-environment? We can use Deeplearning4j (dl4j), which is:
the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala.
You can use dl4j for writing and training machine learning models. You can also import those that you have already prepared in python. Let’s look at a simple unit test and walk through the steps.
Unit tests
@Test
void testStandardActivation()
throws IOException,
InvalidKerasConfigurationException,
UnsupportedKerasConfigurationException {
File kerasModel = new File("src/test/resources/model/model_standard.h5");
MultiLayerNetwork model = KerasModelImport.importKerasSequentialModelAndWeights(
kerasModel.getAbsolutePath(), false);
for (Layer layer : model.getLayers()) {
log.info("Layer [{}]", layer);
}
log.info("summary [{}]", model.summary());
log.info("conf [{}]", model.conf());
Assertions.assertEquals(7, model.getLayers().length);
}
The model in Keras looks like this (adapted from the distribution article):
model_standard = Sequential()
model_standard.add(Dense(12, input_dim=X.shape[1], kernel_initializer='glorot_uniform', kernel_regularizer=regularizers.l2(0.001), activation='relu'))
model_standard.add(Dropout(rate=0.01))
model_standard.add(Dense(8, kernel_initializer='glorot_uniform', activation='relu'))
model_standard.add(Dropout(rate=0.01))
model_standard.add(Dense(8, kernel_initializer='glorot_uniform', activation='relu'))
model_standard.add(Dropout(rate=0.01))
model_standard.add(Dense(1))
model_standard.compile(loss='mse', optimizer=RMSprop(lr=0.01), metrics=['mae'])
We have seven layers (Dense, Dropout, Dense, Dropout, Dense, Dropout, Dense). The test checks this for us, once we have imported the model. Dl4j creates a Keras model object by importing the model and weights separately. Or, we can import both together in the form of an .h5 model:
MultiLayerNetwork model = KerasModelImport.importKerasSequentialModelAndWeights(
kerasModel.getAbsolutePath(), false);On import we can state if we want to apply the training configuration. This is the Boolean parameter above. It’s not relevant if we are importing a model that we have already compiled. We get back an instance of a MultiLayerNetwork which is just a standard representation of a layered Keras neural network. We can examine the layers and check that we have the expected number.
This model representation is limited to those layers that dl4j supports. There will be an unavoidable feature lag between the Keras API and features that Dl4j supports, as the libraries are maintained independently of each other. In an earlier article we discussed using a custom activation layer to fit a probability distribution using our predicted values. If we export and import this model into Dl4j, then our custom activation function cannot be recognized. We confirm this in a second unit test:
@Test
void testNegativeBinomial()
throws IOException,
InvalidKerasConfigurationException,
UnsupportedKerasConfigurationException {
File binomial = new File("src/test/resources/model/keras_binom.h5");
Assertions.assertThrows(UnsupportedKerasConfigurationException.class, () -> {
KerasModelImport.importKerasSequentialModelAndWeights(binomial.getAbsolutePath(), false);
});
}Python results
We can then use the first model – with a standard activation function – to make a prediction by passing in an input with the right shape. Let’s compare with our python code:
element = 9
type(X[element:element+1]), X[element:element+1].shape, X.shape, X[element:element+1]
prediction_standard = model_standard.predict(X[element:element+1])
y_actual_standard = y[element:element+1]
print(f'y_actual is {y_actual_standard[0]}, prediction is {prediction_standard}')which yields:
(numpy.ndarray,
(1, 13),
(506, 13),
array([[-0.40072931, 0.04877224, -0.47665354, -0.27259857, -0.26515405,
-0.39980821, 0.61609042, 1.32963473, -0.52300145, -0.57751897,
-1.50523663, 0.32932512, 0.62334395]])
)
y_actual is 18, prediction is [[16.903563]]Java results
Let’s now use this same tuple, shaped in the same way, with our java code:
INDArray input = Nd4j.create(tuple, new int[] { 1, 13 });
INDArray prediction = model.output(input);
log.info("result [{}]", prediction);
2021-04-26 17:09:06.266 _[34mINFO _[0;39m [main] _[KerasTest _[0;39m - result [[[17.4087]]]There are a couple of things to note here. Firstly, Dl4j uses the Nd4j library for array manipulation. Dl4j reshapes a java array into an Nd4j array using the supplied shape. The shape of the input can be up to 4 dimensions, depending on the type of model. A summary is here:
- NCHW: number of samples, channels, height, width
- NHWC: number of samples, height, width, height
- NWC: number of samples, width (or sequence length), channels
For non-RNN/CNN models:
- NW: number of samples, width (or number of features)
So for our model we need the shape [1, 13], as we have thirteen features and we are predicting on a single tuple.
Secondly, as of the time of writing the current version of Dl4j is 1.0.0.-beta7. 1.0.0.-beta7 introduced a breaking change that affects the ordering – or shape – of our input array for RNN models. The blog explains how this now streamlines Dl4j behaviour with that which we have most likely used when preparing our model with Keras:
As DL4J is frequently used as the deployment platform for pretrained Keras models, the importer will now apply the appropriate data format configuration, so imported models should expect data to be provided in the same format as it was in training.
Additionally, DL4J now also supports the NWC (channels last; shape:
https://blog.konduit.ai/2020/05/14/deeplearning4j-1-0-0-beta7-released/[numExamples, sequenceLength, channels]) format for all RNN and 1D CNN layers. Again, this makes it easier to use imported models with data formatted in the same way that was used to train them.
H2O
Let’s follow the same steps for H2O. We’ll create a model in python, export it to file and then load using the h2o java library. We’ll keep things simple and do the bare minimum just to get a trained model to export. Using the airlines-data dataset we create two different models:
airlines_data = h2o.import_file("https://s3.amazonaws.com/h2o-airlines-unpacked/allyears2k.csv")
from h2o.estimators import H2OGradientBoostingEstimator
from h2o.estimators import H2OIsolationForestEstimator
gbe = H2OGradientBoostingEstimator(ntrees = 1)
gbe.train(x = ["Origin", "Dest"], y = "IsDepDelayed", training_frame=airlines_data)
ife = H2OIsolationForestEstimator()
ife.train(training_frame=airlines_data)
There are two ways to export these models. Either as java files (download as POJO: Plain Old Java Object), or as a resource (download as MOJO: Model Object, Optimised), though not all models allow both options:
ife.download_pojo('h2o_ife_pojo', get_genmodel_jar=True)
…
=> H2OValueError: Export to POJO not supported
gbe.download_pojo('h2o_gbe_pojo', get_genmodel_jar=True)
=> ‘...\\h2o_gbe_pojo\\GBM_model_python_1619536617929_5.java'
gbe.download_pojo('h2o_gbe_pojo_nojar', get_genmodel_jar=False)
=> '...\\h2o_gbe_pojo_nojar\\GBM_model_python_1619536617929_5.java'
Looking at the outputs, we can see that the parameter get_genmodel_jar ensures that our export folder includes the h2o-genmodel.jar file. We need this in our java application. If we had already declared this as a maven/gradle dependency then we can omit it on export.
However, we would rather have our model declared as a resource that is independent of our application code. To achieve this we export the model as a MOJO folder instead:
gbe.download_mojo('h2o_gbe', get_genmodel_jar=True)
=> ...\\h2o_gbe\\GBM_model_python_1619536617929_5.zip
ife.download_mojo('h2o_ife', get_genmodel_jar=True)
=> …\\ h2o_ife\\IsolationForest_model_python_1619536617929_9.zip
The output folder include a zip file of the model elements (e.g. trees) and the optional .jar file.
Unit tests
Now, let’s create two simple unit tests with these models loaded as resources to illustrate a few things:
@Test
void testModgenImport() throws IOException, PredictException {
URL mojoUrl = this.getClass().getClassLoader()
.getResource("model/IsolationForest_model_python_1619536617929_9.zip");
log.info("url [{}]", mojoUrl);
MojoReaderBackend reader = MojoReaderBackendFactory.createReaderBackend(mojoUrl,
MojoReaderBackendFactory.CachingStrategy.MEMORY);
MojoModel model = ModelMojoReader.readFrom(reader);
EasyPredictModelWrapper modelWrapper = new EasyPredictModelWrapper(model);
log.info("modelWrapper [{}]", modelWrapper);
RowData testRow = new RowData();
testRow.put("thisColumnDoesNotExistInTheModel", "blah");
testRow.put("UniqueCarrier", "PS");
AnomalyDetectionPrediction prediction =
(AnomalyDetectionPrediction) modelWrapper.predict(testRow);
log.info("normalizedScore [{}]", prediction.normalizedScore);
log.info("score [{}]", prediction.score);
Assertions.assertTrue(prediction.score > 5 && prediction.score < 6);
/* cell contents should match the expected type */
testRow.put("DayOfWeek", "blah");
Assertions.assertThrows(PredictNumberFormatException.class, () -> {
modelWrapper.predict(testRow);
});
testRow.put("DayOfWeek", "1");
testRow.put("UniqueCarrier", "XX");
Assertions.assertThrows(PredictUnknownCategoricalLevelException.class, () -> {
modelWrapper.predict(testRow);
});
}
@Test
void unmatchedModelTypeInputTest() throws Exception {
URL mojoUrl = this.getClass().getClassLoader()
.getResource("model/GBM_model_python_1619536617929_5.zip");
MojoReaderBackend reader = MojoReaderBackendFactory.createReaderBackend(mojoUrl,
MojoReaderBackendFactory.CachingStrategy.MEMORY);
EasyPredictModelWrapper modelWrapper =
new EasyPredictModelWrapper(ModelMojoReader.readFrom(reader));
/*
* ensure we are calling the expected type of prediction
*/
Assertions.assertThrows(PredictException.class, () -> {
modelWrapper.predictAnomalyDetection(new RowData());
});
AbstractPrediction p = modelWrapper.predict(new RowData());
Assertions.assertTrue(p instanceof BinomialModelPrediction);
}
In the first test we loaded the Isolation Forest model. The data we loaded for training looks like this:
i.e. the column DayOfWeek is an integer column. If we provide a tuple where this does not match, then we get an expected exception:
testRow.put("DayOfWeek", "blah");
Assertions.assertThrows(PredictNumberFormatException.class, () -> {
modelWrapper.predict(testRow);
});
However, this does not happen when we call a prediction with an empty test row, or with columns not expected by the model:
testRow.put("thisColumnDoesNotExistInTheModel", "blah");Encoding considerations
Internally, the library initializes the RowData object with NaN (not-a-number) values. Any key/value that we supply the RowData object with, that the model does not expect, is simply ignored. Empty values are allowed (we even get a predicted value with a input tuple that is nothing other than NaNs). In other words, the java library does not enforce the column-type constraint explicitly. The columns we define have to match model meta-data, but not all columns must be supplied. Furthermore, with non-numerical columns – such as “UniqueCarrier” – the model has encoded them “on-the-fly”. Looking at the model parameters, we see that the default behaviour is to convert any non-numerical value to an integer coding:
But beware! If our test tuple contains a to-be-encoded value that the model has not yet seen during training, then – depending on the algorithm – this may cause an exception if it cannot be mapped:
testRow.put("UniqueCarrier", "XX");
Assertions.assertThrows(PredictUnknownCategoricalLevelException.class, () -> {
modelWrapper.predict(testRow);
});For more details see the note here. A final point is that we have to know what kind of prediction we are making. For instance, in the second unit test we should expect a (a BinomialModelPrediction) instead of an anomaly detection prediction:
Assertions.assertThrows(PredictException.class, () -> {
modelWrapper.predictAnomalyDetection(new RowData());
});
AbstractPrediction p = modelWrapper.predict(new RowData());
Assertions.assertTrue(p instanceof BinomialModelPrediction);
This information is implicit in the model itself. When instantiating a model wrapper the library parses the internal model category:
EasyPredictModelWrapper modelWrapper = new EasyPredictModelWrapper(model);and we get the corresponding type of prediction as an instance of AbstractPrediction when we call predict(). From EasyPredictModelWrapper.class:
public AbstractPrediction predict(RowData data, ModelCategory mc) throws PredictException {
switch (mc) {
case AutoEncoder:
return predictAutoEncoder(data);
case Binomial:
return predictBinomial(data);
...
case Unknown:
throw new PredictException("Unknown model category");
default:
throw new PredictException("Unhandled model category (" + m.getModelCategory() + ") in switch statement");
}
}
Weka
Our final candidate is Weka. This is a data mining tool that is part of the Pentaho Data Platform, but is a very effective machine learning tool in its own right. Weka allows us to load training data and to easily compare results from a range of different models. It requires the input data to use its own data format, combining meta-header with columns of data e.g.
%% Monthly totals of international airline passengers (in thousands) for
%% 1949-1960.
@relation airline_passengers
@attribute passenger_numbers numeric
@attribute Date date 'yyyy-MM-dd'
@data
112,1949-01-01
118,1949-02-01
132,1949-03-01
...We have a comment at the top, followed by a name (@relation) displayed internally by weka. Following that we have a list of attributes (@attribute), followed columns of data (@data) matching the attributes in type and order.
Weka is implemented in java, so, just like h2o, we can choose between exporting a model either as a java class or as a resource. Once we have exported the model as a binary file we can import into a java application. Import the model like this:
RandomForest rf = (RandomForest) SerializationHelper.read("src/test/resources/model/xv_rf.model");Of the three libraries we have looked at in this article, Weka is the least intuitive. The meta data is read (internally) from the model artefact, but setting up a tuple for prediction is not that elegant. The basic idea is to build a mini model instance. This mirrors the format used when training the model, although the feature names don’t have to match:
@relation modelName
@attribute c1 numeric
@attribute c2 numeric
@attribute c3 numeric
@attribute c4 numeric
@attribute c5 numeric
@data
...Unit tests
Let’s define a simple unit test to illustrate this:
@Test
void wekaTest() throws Exception {
RandomForest rf =
(RandomForest) SerializationHelper.read("src/test/resources/model/xv_rf.model");
ArrayList<Attribute> attributes = Lists.newArrayList();
attributes.add(new Attribute("c1"));
attributes.add(new Attribute("c2"));
attributes.add(new Attribute("c3"));
attributes.add(new Attribute("c4"));
attributes.add(new Attribute("c5"));
var modelInstances = new Instances("modelName", attributes, 0);
modelInstances.setClassIndex(4);
Instance instance = new DenseInstance(5);
instance.setValue(attributes.get(0), 1);
instance.setValue(attributes.get(1), 2);
instance.setValue(attributes.get(2), 3);
instance.setValue(attributes.get(3), 4);
instance.setDataset(modelInstances);
Double prediction = (Double) rf.classifyInstance(instance);
Assertions.assertNotNull(prediction);
}
The instance object is passed to the model to get a prediction:
Double prediction = (Double) rf.classifyInstance(instance);There is a very particular order needed here. The Instances object has to be initialized with the Attributes collection *before* the setValue methods are called on the Instance object. This is because the Instances constructor sets the indices silently, and these are needed for setValue. The attribute names are not used, just the index positions.
Summary
We have looked at three java libraries, each of which is suitable for carrying out all steps in model preparation and callout. As the integration of python-generated models within a java environment is what interests us here, we have just looked at how to import and call our models. Weka offers flexibility in building and comparing models, but is not so elegant when it comes to importing and calling ready-made models. Keras and Deeplearning4j is a powerful combination as – apart from customized functions – we can combine the power of Tensorflow models with the ease of the Keras API. H2O offers this, plus the fact that there is no feature-lag between the Python and Java APIs as they come from the same distributor.
