Eclipse Deeplearning4j: Data pipeline, DataVec Examples

This project contains a set of examples that demonstrate how raw data in various formats can be loaded, split and preprocessed to build serializable (and hence reproducible) ETL pipelines using the DataVec library.

Go back to the main repository page to explore other features/functionality of the Eclipse Deeplearning4J ecosystem. File an issue here to request new features.

The examples in this project and what they demonstrate are briefly described below. This is also the recommended order to explore them in.

Loading Data

InputSplit and its implementations are utility classes for defining and managing a catalog of loadable locations (paths/files), in memory, that can later be exposed through an Iterator. In simple terms, they define where your data comes from or should be saved to, when building a data pipeline with DataVec.

Ex01_FileSplitExample.java Using FileSplit which loads files in a given location. Constructor overloading allows for varying functionality like filtering files to load, loading recursively etc
Ex02_CollectionSplitExample.java Create a split from a collection of URIs
Ex03_NumberedFileInputSplitExample.java Create a split from numbered files, following a common pattern like file1.txt, file2.txt ... file100.txt
Ex04_TransformSplitExample.java Maps URIs of a given split to new URIs. Useful when features and labels are in different files sharing a common naming scheme, and the name of the output file can be determined given the name of the input file. Eg. a-in.csv and a-out.csv
Ex05_SamplingBaseInputSplitExample.java Generate several splits from the main split say for training, validation and testing.
Ex06_KFoldIteratorFromDataSet.java Generate a K-Fold iterator from a dataset

Cleaning, Transforming and Analysing Data

IrisCSVTransform.java A basic example that introduces users to important concepts like Schema and TransformProcess with categoricalToInteger.
CSVMixedDataTypesLocal.java Common preprocessing steps like removing unnecessary columns, filtering based on column value, replacing invalid values, parsing date time etc
CSVMixedDataTypes.java Same as the above but with Apache Spark
PrintSchemasAtEachStep.java How to print schema at each step which would be useful for debugging transform scripts in a complicated pipeline
IrisAnalysis.java Basic Analysis of the dataset saved and presented as an html file
IrisNormalizer.java Proper useage of preprocessors with min max scaler
JoinExample.java Perform joins on datasets
PivotExample.java Combine multiple independent records by key.
WebLogDataExample.java Preprocessing/aggregation operations on some web log data
CustomReduceExample.java Custom Reduction example for operations on some simple CSV data that involve a custom reduction.
MultiOpReduceExample.java Reduce example with multiple ops on one column

Formats

CSVtoMapFileConversion.java A simple example on how to convert a CSV text file to a Hadoop MapFile format for better performance and the convenience of randomization supported by the MapFileRecordReader
SVMLightExample.java MNIST SVMLight example
ImagePipelineExample.java An imagepipeline that also demonstrates using transforms to augment a small dataset