First Project - Example

Please use the following Google Colaboratory project for a complete example you can work through showing how you can integrate the Seclea Platform into your data science workflow. Please copy the Google Colaboratory project into your account and run it from there.

Google Colabcolab.research.google.com

Following is a step-by-step guide, followed by the Google Colaboratory project.

We will run through a sample project showing how to use Seclea's tools to record your data science work and explore the results in the Seclea Platform.

Set up the Project

You can head to platform.seclea.com and log in.

Please create a new project by clicking the "+ Create" button and giving it a name and description.

Creating a New Project on the Seclea Platform

After you have created your project, it will appear in the All Projects section, as shown below:

All Projects Section After Project Creation

Click the project name in the All Projects section to go to the project setting page on the Seclea Platform.

Project Dashboard on the Seclea Platform

Click the settings option in the left panel will take you to the respective project's settings.

Individual Project Setting Page on the Seclea Platform

In the project setting, you can select, modify or set new templates for compliance, risk management and any internal standard/policy you must comply with. Details on how to use compliance, risk management, and internal settings are detailed in their respective sections.

If you want to include additional team members in the project, you can include them using the Access section of the setting.

These are optional settings; you can set/modify them later in the project.

Integrating with Seclea-AI

You can get the seclea-ai package from pip

!pip install seclea_ai

When you initialise the SecleaAI object, you will be prompted to log in if you haven't already done so. You must use the same Project Name you used earlier and the Organization name provided with your credentials.

from seclea_ai import SecleaAI

# NOTE - use the organization name provided to you when issuing credentials.
seclea = SecleaAI(project_name="Car Insurance Fraud Detection", organization='')

Handling the Data

You can download the data for this tutorial if you are working on this in Colab or without reference to the repo - this is an Insurance Claims dataset with various features and 1000 samples.

Now we can upload the initial data to the Seclea Platform.

This should include whatever information we know about the dataset as metadata. There are only two keys to add in metadata for now - outputs and continuous_features.

You can leave out outputs if you haven't decided what you will be predicted yet, but you should know or be able to find the continuous features at this point.

You can also update these when uploading datasets during/after pre-processing.

import numpy as np
import pandas as pd

# load the data 
data = pd.read_csv('insurance_claims.csv', index_col="policy_number")

# define the metadata for the dataset.
dataset_metadata = {"outputs": ["fraud_reported"],
                    "favourable_outcome": "N",
                    "unfavourable_outcome": "Y",
                    "continuous_features": [
                                            "total_claim_amount",
                                            'policy_annual_premium',
                                            'capital-gains',
                                            'capital-loss',
                                            'injury_claim',
                                            'property_claim',
                                            'vehicle_claim',
                                            'incident_hour_of_the_day',
                                            ]}


# ⬆️ upload the dataset - pick a meaningful name here; you'll see it a lot on the platform!
seclea.upload_dataset(dataset=data, dataset_name="Auto Insurance Fraud", metadata=dataset_metadata)

Evaluating the Dataset

After running the above section, head back to the Seclea Platform so that we can take a closer look at our Dataset. To do so, you can navigate to the datasets section - under Prepare tab.

Youtube video showing how to access the dataset evaluation page on the Seclea Platform.

Personal Identifiable Information (PII) and Format Check

Youtube video showing how to access the dataset PII and format check functionality on the Seclea Platform.

Data Bias Check

Dataset bias evaluation functionality on the Seclea Platform.

Transformations

When using Seclea to record your Data Science work, you will have to take care with how you deal with transformations of the data.

We require that all transformations are encapsulated in a function that takes the data and returns the transformed data. There are a few things to be aware of, so please see the docs for more.

# Create a copy to isolate the original dataset
df1 = data.copy(deep=True)

def encode_nans(df):
    # convert the special characters to nans
    return df.replace('?', np.NaN)

df2 = encode_nans(df1)

Data Cleaning

We will carry out some pre-processing and generate a few different datasets to see how to track these on the platform. This also means we can train our models on different data and see how that affects performance.

## Drop the the column which are more than some proportion NaN values
def drop_nulls(df, threshold):
    cols = [x for x in df.columns if df[x].isnull().sum() / df.shape[0] > threshold]
    return df.drop(columns=cols)

# We choose 95% as our threshold
null_thresh = 0.95
df3 = drop_nulls(df2, threshold=null_thresh)

def drop_correlated(data, thresh):
    import numpy as np

    # calculate correlations
    corr_matrix = data.corr().abs()
    # get the upper part of correlation matrix
    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))

    # columns with correlation above threshold
    redundant = [column for column in upper.columns if any(upper[column] >= thresh)]
    print(f"Columns to drop with correlation > {thresh}: {redundant}")
    new_data = data.drop(columns=redundant)
    return new_data

# drop columns that are too closely correlated
correlation_threshold = 0.95
df4 = drop_correlated(df3, correlation_threshold)

Upload Intermediate Dataset

Before balancing the datasets we will upload them to the Seclea Platform.

• We define the metadata for the dataset - if there have been any changes since the original dataset we need to put that here, otherwise, we can reuse the original metadata. In this case, we have dropped some of the continuous feature columns so we will need to redefine

• We define the transformations that took place between the last state we uploaded and this dataset. This is a list of functions and arguments. Please have a look at docs.seclea.com for more details on the correct formatting.

from seclea_ai.transformations import DatasetTransformation

# define the updates to the metadata - only changes are updated - here a continuous feature has been dropped so now
# we remove it from the list of continuous features.
processed_metadata = {"continuous_features": [
                                            "total_claim_amount",
                                            'policy_annual_premium',
                                            'capital-gains',
                                            'capital-loss',
                                            'injury_claim',
                                            'property_claim',
                                            'incident_hour_of_the_day',
                                            ]}

# 🔀 define the transformations - note the arguments
cleaning_transformations = [
            DatasetTransformation(encode_nans, data_kwargs={"df": df1}, kwargs={}, outputs=["df"]),
            DatasetTransformation(
                drop_nulls, data_kwargs={"df": "inherit"}, kwargs={"threshold": null_thresh}, outputs=["data"]
            ),
            DatasetTransformation(
                drop_correlated, data_kwargs={"data": "inherit"}, kwargs={"thresh": correlation_threshold}, outputs=["df"]
            ),
        ]

# ⬆️ upload the cleaned datasets
seclea.upload_dataset(dataset=df4,
                      dataset_name="Auto Insurance Fraud - Cleaned",
                      metadata=processed_metadata,
                      transformations=cleaning_transformations)

def fill_nan_const(df, val):
    """Fill NaN values in the dataframe with a constant value"""
    return df.replace(['None', np.nan], val)


# Fill nans in 1st dataset with -1
const_val = -1
df_const = fill_nan_const(df4, const_val)

def fill_nan_mode(df, columns):
    """
    Fills nans in specified columns with the mode of that column
    Note that we want to make sure to not modify the dataset we passed in but to
    return a new copy.
    We do that by making a copy and specifying deep=True.
    """
    new_df = df.copy(deep=True)
    for col in df.columns:
        if col in columns:
            new_df[col] = df[col].fillna(df[col].mode()[0])
    return new_df


nan_cols = ['collision_type','property_damage', 'police_report_available']
df_mode = fill_nan_mode(df4, nan_cols)


# find columns with categorical data for both dataset
cat_cols = df_const.select_dtypes(include=['object']).columns.tolist()

def encode_categorical(df, cat_cols): 
  from sklearn.preprocessing import LabelEncoder

  new_df = df.copy(deep=True)
  for col in cat_cols:
    if col in df.columns:
        le = LabelEncoder()
        le.fit(list(df[col].astype(str).values))
        new_df[col] = le.transform(list(df[col].astype(str).values))
  return new_df

df_const = encode_categorical(df_const, cat_cols)
df_mode = encode_categorical(df_mode, cat_cols)

# Update metadata with new encoded values for the outcome column.
encoded_metadata = {"favourable_outcome": 0,
                    "unfavourable_outcome": 1,}


# 🔀 define the transformations - for the constant fill dataset
const_processed_transformations = [
    DatasetTransformation(fill_nan_const, data_kwargs={"df": df4}, kwargs={"val": const_val}, outputs=["df"]),
    DatasetTransformation(encode_categorical, data_kwargs={"df": "inherit"}, kwargs={"cat_cols":cat_cols}, outputs=["df"]),
]

# ⬆️ upload the constant fill dataset
seclea.upload_dataset(dataset=df_const, 
                      dataset_name="Auto Insurance Fraud - Const Fill", 
                      metadata=encoded_metadata,
                      transformations=const_processed_transformations)

# 🔀 define the transformations - for the mode fill dataset
mode_processed_transformations = [
    DatasetTransformation(fill_nan_mode, data_kwargs={"df": df4}, kwargs={"columns": nan_cols}, outputs=["df"]),
    DatasetTransformation(encode_categorical, data_kwargs={"df": "inherit"}, kwargs={"cat_cols": cat_cols}, outputs=["df"]),
]

# ⬆️ upload the mode fill dataset
seclea.upload_dataset(dataset=df_mode,
                      dataset_name="Auto Insurance Fraud - Mode Fill",
                      metadata=encoded_metadata,
                      transformations=mode_processed_transformations)

def get_samples_labels(df, output_col):
    X = df.drop(output_col, axis=1)
    y = df[output_col]

    return X, y

# split the datasets into samples and labels ready for modelling.
X_const, y_const = get_samples_labels(df_const, "fraud_reported")
X_mode, y_mode = get_samples_labels(df_mode, "fraud_reported")

def get_test_train_splits(X, y, test_size, random_state):
    from sklearn.model_selection import train_test_split

    return train_test_split(
        X, y, test_size=test_size, stratify=y, random_state=random_state
    )
    # returns X_train, X_test, y_train, y_test

# split into test and train sets
X_train_const, X_test_const, y_train_const, y_test_const = get_test_train_splits(X_const, y_const, test_size=0.2, random_state=42)
X_train_mode, X_test_mode, y_train_mode, y_test_mode = get_test_train_splits(X_mode, y_mode, test_size=0.2, random_state=42)

# 🔀 define the transformations - for the constant fill training set
const_train_transformations = [
    DatasetTransformation(
            get_test_train_splits,
            data_kwargs={"X": X_const, "y": y_const},
            kwargs={"test_size": 0.2, "random_state": 42},
            outputs=["X_train_const", None, "y_train_const", None],
            split="train",
            ),
]

# ⬆️ upload the const fill training set
seclea.upload_dataset_split(
                        X=X_train_const,
                        y=y_train_const,
                        dataset_name="Auto Insurance Fraud - Const Fill - Train",
                        metadata={},
                        transformations=const_train_transformations
)

# 🔀 define the transformations - for the constant fill test set
const_test_transformations = [
    DatasetTransformation(
            get_test_train_splits,
            data_kwargs={"X": X_const, "y": y_const},
            kwargs={"test_size": 0.2, "random_state": 42},
            outputs=[None, "X_test_const", None, "y_test_const"],
            split="test"
            ),
]

# ⬆️ upload the const fill test set
seclea.upload_dataset_split(X=X_test_const,
                      y=y_test_const,
                      dataset_name="Auto Insurance Fraud - Const Fill - Test",
                      metadata={},
                      transformations=const_test_transformations)

# 🔀 define the transformations - for the mode fill training set
mode_train_transformations = [
    DatasetTransformation(
            get_test_train_splits,
            data_kwargs={"X": X_mode, "y": y_mode},
            kwargs={"test_size": 0.2, "random_state": 42},
            outputs=["X_train_mode", None, "y_train_mode", None],
            split="train",
            ),
]

# ⬆️ upload the mode fill train set
seclea.upload_dataset_split(X=X_train_mode,
                      y=y_train_mode,
                      dataset_name="Auto Insurance Fraud - Mode Fill - Train",
                      metadata=processed_metadata,
                      transformations=mode_train_transformations)

# 🔀 define the transformations - for the mode fill test set
mode_test_transformations = [
    DatasetTransformation(
            get_test_train_splits,
            data_kwargs={"X": X_mode, "y": y_mode},
            kwargs={"test_size": 0.2, "random_state": 42},
            outputs=[None, "X_test_mode", None, "y_test_mode"],
            split="test",
            ),
]

# ⬆️ upload the mode fill test set
seclea.upload_dataset_split(X=X_test_mode,
                      y=y_test_mode,
                      dataset_name="Auto Insurance Fraud - Mode Fill - Test",
                      metadata={},
                      transformations=mode_test_transformations)



def smote_balance(X, y, random_state):
    from imblearn.over_sampling import SMOTE

    sm = SMOTE(random_state=random_state)

    X_sm, y_sm = sm.fit_resample(X, y)

    print(
        f"""Shape of X before SMOTE: {X.shape}
    Shape of X after SMOTE: {X_sm.shape}"""
    )
    print(
        f"""Shape of y before SMOTE: {y.shape}
    Shape of y after SMOTE: {y_sm.shape}"""
    )
    return X_sm, y_sm
    # returns X, y

# balance the training sets - creating new training sets for comparison
X_train_const_smote, y_train_const_smote = smote_balance(X_train_const, y_train_const, random_state=42)
X_train_mode_smote, y_train_mode_smote = smote_balance(X_train_mode, y_train_mode, random_state=42)

# 🔀 define the transformations - for the constant fill balanced train set
const_smote_transformations = [
    DatasetTransformation(
            smote_balance,
            data_kwargs={"X": X_train_const, "y": y_train_const},
            kwargs={"random_state": 42},
            outputs=["X", "y"]
            ),
]

# ⬆️ upload the constant fill balanced train set
seclea.upload_dataset_split(X=X_train_const_smote,
                      y=y_train_const_smote,
                      dataset_name="Auto Insurance Fraud - Const Fill - Smote Train",
                      metadata={},
                      transformations=const_smote_transformations)

# 🔀 define the transformations - for the mode fill balanced train set
mode_smote_transformations = [
    DatasetTransformation(
            smote_balance,
            data_kwargs={"X": X_train_mode, "y": y_train_mode},
            kwargs={"random_state": 42},
            outputs=["X", "y"]
            ),
]

# ⬆️ upload the mode fill balanced train set
seclea.upload_dataset_split(X=X_train_mode_smote,
                      y=y_train_mode_smote,
                      dataset_name="Auto Insurance Fraud - Mode Fill - Smote Train",
                      metadata={},
                      transformations=mode_smote_transformations)

Evaluating the Transformations

Now head to platform.seclea.com again to take another look at the Datasets section. You will see that there is much more to look at now.

You can see here how the transformations are used to show you the history of the data and how it arrived in its final state.

Evaluating the dataset transformations over the course of a project.

Modelling

Now we get started with the modelling. We will run the same models over each of our datasets to explore how the different processing of the data has affected our results.

We will use three models from sklearn for this, DecisionTree, RandomForest and GradientBoosting Classifers.

Training

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score
from sklearn.metrics import accuracy_score

classifiers = {
    "RandomForestClassifier": RandomForestClassifier(),
    "DecisionTreeClassifier": DecisionTreeClassifier(),
    "GradientBoostingClassifier": GradientBoostingClassifier()
}

datasets = [
    ("Const Fill", (X_train_const, X_test_const, y_train_const, y_test_const)),
    ("Mode Fill", (X_train_mode, X_test_mode, y_train_mode, y_test_mode)),
    ("Const Fill Smote", (X_train_const_smote, X_test_const, y_train_const_smote, y_test_const)),
    ("Mode Fill Smote", (X_train_mode_smote, X_test_mode, y_train_mode_smote, y_test_mode))
    ]

for name, (X_train, X_test, y_train, y_test) in datasets:

    for key, classifier in classifiers.items():
        # cross validate to get an idea of generalisation.
        training_score = cross_val_score(classifier, X_train, y_train, cv=5)

        # train on the full training set
        classifier.fit(X_train, y_train)

        # ⬆️ upload the fully trained model
        seclea.upload_training_run_split(model=classifier, X_train=X_train, y_train=y_train, X_test=X_test, y_test=y_test)

        # test accuracy
        y_preds = classifier.predict(X_test)
        test_score = accuracy_score(y_test, y_preds)
        print(f"Classifier: {classifier.__class__.__name__} has a training score of {round(training_score.mean(), 3) * 100}% accuracy score on {name}")
        print(f"Classifier: {classifier.__class__.__name__} has a test score of {round(test_score, 3) * 100}% accuracy score on {name}")

Analysis

Let's head back to platform.seclea.com and we can analyse our Models

Analysis of the trained models on the Seclea Platform.