snapmlspark API¶

class
snap_ml_spark.LinearRegression.
LinearRegression
(max_iter=1000, dual=True, regularizer=1.0, verbose=False, use_gpu=False, class_weights=None, gpu_mem_limit=0, n_threads=1, penalty='l2', tol=0.001, return_training_history=None, labelColIndex=1)¶ Linear Regression classifier
This class implements Regularized Linear regression using the IBM Snap ML distributed solver. It can handle sparse and dense dataset formats. Please use libsvm, snap or csv format for the Dual algorithm, or snap.t (transposed) format for the primal algorithm.
Parameters:  max_iter (int, default : 1000) – Maximum number of iterations used by the solver to converge.
 dual (bool, default : True) – Dual or primal formulation. Recommendation: if n_samples > n_features use dual=True, else dual=False.
 regularizer (float, default : 1.0) – Regularization strength. It must be a positive float. Larger regularization values imply stronger regularization.
 verbose (boolean, default : False) – Flag for indicating if the training loss will be printed at each epoch.
 use_gpu (bool, default : False) – Flag for indicating the hardware platform used for training. If True, the training is performed using the GPU. If False, the training is performed using the CPU.
 class_weights ('balanced'/True or None/False, optional) – If set to ‘None’, all classes will have weight = 1.
 gpu_mem_limit (int, default : 0) – Limit of the GPU memory. If set to the default value 0, the maximum possible memory is used.
 n_threads (int, default : 1 meaning that n_threads=256 if GPU is enabled, else 1) – Number of threads to be used.
 penalty (str, default : "l2") – The regularization / penalty type. Possible values are “l2” for L2 regularization (RidgeRegression) or “l1” for L1 regularization (LassoRegression). L1 regularization is possible only for the primal optimization problem (dual=False).
 tol (float, default : 0.001) – The tolerance parameter. Training will finish when maximum change in model coefficients is less than tol.
 return_training_history (str or None, default : None) – How much information about the training should be collected and returned by the fit function. By default no information is returned (None), but this parameter can be set to “summary”, to obtain summary statistics at the end of training, or “full” to obtain a complete set of statistics for the entire training procedure. Note, enabling either option will result in slower training.
 labelColIndex (int, default : 1) – Denotes which column to be marked as label. It’s applicable only when Dataframe is passed in as input and its holding features in individual columns rather than in DenseVector or SparseVector
Variables:  coef (ndarray, shape (n_features,)) – Coefficients of the features in the trained model.
 pred_array (ndarray, shape(number_of_test_examples,)) – linear predictions written by the predict() function of this class

fit
(data)¶ learn model
Parameters: data (pyspark.sql.DataFrame/py4j.java_gateway.JavaObject which points to a memory address where the actual data is stored and it cannot be accessed by python as a python array.) – data to fit model Returns: double – final training loss of the last epoch

get_params
()¶ Returns: all the initialized parameters of the Linear Regression model as a python dictionary

predict
(data, num_threads=0)¶ Predict regression values
Parameters:  data (pyspark.sql.DataFrame/py4j.java_gateway.JavaObject which points to a memory address where the actual data is stored and cannot be accessed by python as an array but only can passed as a parameter to this function in order to get the predictions) – data to make predictions
 num_threads – the number of threads to use for inference (default 0 means use all avaliable threads)
Returns: a pointer which points to a com.ibm.snap.ml.DatasetWithPredictions java object. This pointer cannot be accessed by python but the user can access the predictions from the pred_array_ field which is a python array.

class
snap_ml_spark.LogisticRegression.
LogisticRegression
(max_iter=1000, dual=True, regularizer=1.0, verbose=False, use_gpu=False, class_weights=None, gpu_mem_limit=0, n_threads=1, penalty='l2', tol=0.001, return_training_history=None, labelColIndex=1)¶ Logistic Regression classifier
This class implements regularized Logistic Regression using the IBM Snap ML solver. It can handle sparse and dense dataset formats. Use libsvm, snap or csv format for the Dual algorithm, or snap.t (transposed) format for the primal algorithm.
Parameters:  max_iter (int, default : 1000) – Maximum number of iterations used by the solver to converge.
 dual (bool, default : True) – Dual or Primal formulation. Recommendation: if n_samples > n_features use dual=True.
 regularizer (float, default : 1.0) – Regularization strength. It must be a positive float. Larger regularization values imply stronger regularization.
 verbose (boolean, default : False) – Flag for indicating if the training loss will be printed at each epoch.
 use_gpu (bool, default : False) – Flag for indicating the hardware platform used for training. If True, the training is performed using the GPU. If False, the training is performed using the CPU.
 class_weights ('balanced'/True or None/False, optional) – If set to ‘None’, all classes will have weight 1.
 gpu_mem_limit (int, default : 0) – Limit of the GPU memory. If set to the default value 0, the maximum possible memory is used.
 n_threads (int, default : 1 meaning that n_threads=256 if GPU is enabled, else 1) – Number of threads to be used.
 penalty (str, default : "l2") – The regularization / penalty type. Possible values are “l2” for L2 regularization or “l1” for L1 regularization. L1 regularization is possible only for the primal optimization problem (dual=False).
 tol (float, default : 0.001) – The tolerance parameter. Training will finish when maximum change in model coefficients is less than tol.
 return_training_history (str or None, default : None) – How much information about the training should be collected and returned by the fit function. By default no information is returned (None), but this parameter can be set to “summary”, to obtain summary statistics at the end of training, or “full” to obtain a complete set of statistics for the entire training procedure. Note, enabling either option will result in slower training.
 labelColIndex (int, default : 1) – Denotes which column to be marked as label. It’s applicable only when Dataframe is passed in as input and its holding features in individual columns rather than in DenseVector or SparseVector
Variables:  coef (ndarray, shape (n_features,)) – Coefficients of the features in the trained model.
 pred_array (ndarray, shape(number_of_test_examples,)) – binary predictions written by the predict() function
 proba_array (ndarray, shape(number_of_test_examples,)) – predicted probabilities written by the predict_proba() function

fit
(data)¶ learn model
Parameters: data (pyspark.sql.DataFrame/py4j.java_gateway.JavaObject which points to a memory address where the actual data is stored and it cannot be accessed by python as a python array.) – data to fit model Returns: double – final training loss of the last epoch

get_params
()¶ Returns: all the initialized parameters of the Logistic Regression model as a python dictionary

predict
(data, num_threads=0)¶ Predict label
Parameters:  data (pyspark.sql.DataFrame/py4j.java_gateway.JavaObject which points to a memory address where the actual data is stored and cannot be accessed by python as an array but only can passed as a parameter to this function in order to get the predictions) – data to make predictions
 num_threads – the number of threads to use for inference (default 0 means use all avaliable threads)
Returns: a pointer which points to a com.ibm.snap.ml.DatasetWithPredictions java object. This pointer cannot be accessed by python but the user can access the predictions from the pred_array_ field which is a python array.

predict_proba
(data, num_threads=0)¶ Predict probabilities
Parameters:  data (pyspark.sql.DataFrame/py4j.java_gateway.JavaObject which points to a memory address where the actual data is stored and cannot be accessed by python as an array but only can passed as a parameter to this function in order to get the predictions) – data to make predictions
 num_threads – the number of threads to use for inference (default 0 means use all avaliable threads)
Returns: a pointer which points to a com.ibm.snap.ml.DatasetWithPredictions java object. This pointer cannot be accessed by python but the user can access the predictions from the proba_array_ field which is a python array.

class
snap_ml_spark.SupportVectorMachine.
SupportVectorMachine
(max_iter=1000, regularizer=1.0, verbose=False, use_gpu=False, class_weights=None, gpu_mem_limit=0, n_threads=1, tol=0.001, return_training_history=None, labelColIndex=1)¶ Support Vector Machine classifier
This class implements regularized support vector machine using the IBM Snap ML solver. It can handle sparse and dense dataset formats. Use libsvm, snap or csv format for the Dual algorithm, or snap.t (transposed) format for the primal algorithm.
Parameters:  max_iter (int, default : 1000) – Maximum number of iterations used by the solver to converge.
 regularizer (float, default : 1.0) – Regularization strength. It must be a positive float. Larger regularization values imply stronger regularization.
 verbose (boolean, default : False) – Flag for indicating if the training loss will be printed at each epoch.
 use_gpu (bool, default : False) – Flag for indicating the hardware platform used for training. If True, the training is performed using the GPU. If False, the training is performed using the CPU.
 class_weights ('balanced'/True or None/False, optional) – If set to ‘None’, all classes will have weight 1.
 gpu_mem_limit (int, default : 0) – Limit of the GPU memory. If set to the default value 0, the maximum possible memory is used.
 n_threads (int, default : 1 meaning that n_threads=256 if GPU is enabled, else 1) – Number of threads to be used.
 tol (float, default : 0.001) – The tolerance parameter. Training will finish when maximum change in model coefficients is less than tol.
 return_training_history (str or None, default : None) – How much information about the training should be collected and returned by the fit function. By default no information is returned (None), but this parameter can be set to “summary”, to obtain summary statistics at the end of training, or “full” to obtain a complete set of statistics for the entire training procedure. Note, enabling either option will result in slower training.
 labelColIndex (int, default : 1) – Denotes which column to be marked as label. It’s applicable only when Dataframe is passed in as input and its holding features in individual columns rather than in DenseVector or SparseVector
Variables:  coef (ndarray, shape (n_features,)) – Coefficients of the features in the trained model.
 pred_array (ndarray, shape(number_of_test_examples,)) – binary predictions written by the predict() function

fit
(data)¶ learn model
Parameters: data (pyspark.sql.DataFrame/py4j.java_gateway.JavaObject which points to a memory address where the actual data is stored and it cannot be accessed by python as a python array.) – data to fit model Returns: double – final training loss of the last epoch

get_params
()¶ Returns: all the initialized parameters of the Support Vector Machine Model as a python dictionary

predict
(data, num_threads=0)¶ Predict label
Parameters:  data (pyspark.sql.DataFrame/py4j.java_gateway.JavaObject which points to a memory address where the actual data is stored and cannot be accessed by python as an array but only can passed as a parameter to this function in order to get the predictions) – data to make predictions
 num_threads – the number of threads to use for inference (default 0 means use all avaliable threads)
Returns: a pointer which points to a com.ibm.snap.ml.DatasetWithPredictions java object. This pointer cannot be accessed by python but the user can access the predictions from the pred_array_ field which is a python array.

class
snap_ml_spark.DatasetReader.
DatasetReader
¶ Load distributed dataset from file.

load
(file)¶ Load training data in memory
Parameters: file (string) – filename

setFormat
(format)¶ Specify the dataformat of the file. Format values: “snap” or “libsvm” or “csv”
Parameters: format (string) – data format


snap_ml_spark.Metrics.
accuracy
(dataWithPredictions)¶ Parameters: dataWithPredictions – binary predictions computed by the LogisticRegression or SupportVectorMachine predict() function Returns: accuracy computed based on the binary predictions of a classifier (LogisticRegression, SupportVectorMachines) Return type: double

snap_ml_spark.Metrics.
f1score
(dataWithPredictions)¶ Parameters: dataWithPredictions – binary predictions computed by the LogisticRegression or SupportVectorMachine predict() function Returns: f1score metric (2*(precision*recall)/(precision+recall)), computed based on the binary predictions of a classifier (LogisticRegression, SupportVectorMachines) Return type: double

snap_ml_spark.Metrics.
logisticLoss
(dataWithPredictions)¶ Parameters: dataWithPredictions – probabilities computed by the LogisticRegression predict_proba() function Returns: logistic loss computed by the logistic regression predicted probabilities Return type: double

snap_ml_spark.Metrics.
meanSquaredError
(dataWithPredictions)¶ Parameters: dataWithPredictions – linear regression predictions, predicted by the RidgeRegression predict() function Returns: mean squared error computed based on the provided dataWithPredictions parameter Return type: double

snap_ml_spark.Metrics.
precision
(dataWithPredictions)¶ Parameters: dataWithPredictions – binary predictions computed by the LogisticRegression or SupportVectorMachine predict() function Returns: precision metric (TP/(TP+FP)), computed based on the binary predictions of a classifier (LogisticRegression, SupportVectorMachines) Return type: double

snap_ml_spark.Metrics.
recall
(dataWithPredictions)¶ Parameters: dataWithPredictions – binary predictions computed by the LogisticRegression or SupportVectorMachine predict() function Returns: recall metric (TP/(TP+FN)), computed based on the binary predictions of a classifier (LogisticRegression, SupportVectorMachines) Return type: double

snap_ml_spark.Utils.
dump_to_snap_format
(X, y, filename, transpose=False, implicit_vals=False)¶ Nondistributed data writing to snap format
Parameters:

snap_ml_spark.Utils.
read_from_snap_format
(filename)¶ Nondistributed data loading from snap format
Parameters: filename (str) – The file where the data resides. Returns: X, y – Returns two datasets. X : the data used for training or inference y : the labels of the samples in X. Return type: numpy array or sparse matrix, numpy array