Testing the feature engineering pipeline

Let's create a Pipeline out of our transformers and estimators:

import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.PipelineModel

//Create an array out of individual pipeline stages
var transformers = Array(indexer,encoder,assembled)


var pipeline = new Pipeline().setStages(transformers).fit(df_notnull)

var transformed = pipeline.transform(df_notnull)

Note that the setStages method of Pipeline just expects an array of transformers and estimators, which we had created earlier. As parts of the Pipeline contain estimators, we have to run fit on our DataFrame first. The obtained Pipeline object takes a DataFrame in the transform method and returns the results of the transformations:

As expected, we obtain the very same DataFrame as we had while running the stages individually in a sequence.