headerLinks

"If 80 percent of our work is data preparation, then ensuring data quality is the important work of a machine learning team."

Using KNN, first find the k instances closer to the missing value instance, and then get the mean of that attribute related to the k-nearest neighbors (KNN).

Using regression, for each missing attribute, learn a regressor that can predict this missing value based on the other attributes.

Type 1: Missing Completely at Random (MCAR)

The Curse of Dimensionality in Your Dataset

Requiring less computational resources&nbsp;

Increasing the overall performance of the model

Preventing overfitting (when the model becomes too complex and the model memorizes the training data, instead of learning, so in the test data the performance decreases a lot)

Avoiding multicollinearity (high correlation of one or more independent variables). Also, applying this technique will reduce the noise data.&nbsp;

Correlation Between Features: This is the most common approach, which drops some features that have a high correlation with others.&nbsp;

Statistical Tests: Another alternative is to use statistical tests to select the features, checking the relationship of each feature individually with the output variable. There are many examples in the scikit-learn library like <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest" target="_blank" rel="noreferrer noopener">SelectKBest</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectPercentile.html#sklearn.feature_selection.SelectPercentile" target="_blank" rel="noreferrer noopener">SelectPercentile</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html#sklearn.feature_selection.chi2" target="_blank" rel="noreferrer noopener">chi2</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_classif.html#sklearn.feature_selection.f_classif" target="_blank" rel="noreferrer noopener">f_classif</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression" target="_blank" rel="noreferrer noopener">f_regression</a>.&nbsp;

Recursive Feature Elimination (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html" target="_blank" rel="noreferrer noopener">RFE</a>): The Recursive Feature Elimination, also known as Backward Elimination, where the algorithm trains the model with all features in the dataset, calculating the performance of the model, and then drops one feature at a time, stopping when the performance improvement becomes negligible.&nbsp;

Variance Threshold: Another feature selection method is the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html#sklearn.feature_selection.VarianceThresholdhttps://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html#sklearn.feature_selection.VarianceThreshold" target="_blank" rel="noreferrer noopener">variance threshold</a>, which detects features with high variability within the column, selecting those that go over the threshold. The premise of this approach is that features with low variability within themselves have little influence on the output variable.

Feature Engineering: Using Domain Knowledge to Create Features&nbsp;

Handling a Large Amount of Data (Sampling Data)

Sampling without replacement. This approach avoids having the same data repeated in the sample, so if the record is selected, it's removed from the population.

Sampling with replacement. With this approach, the object is not removed from the population and can be repeated multiple times for the sample data since it can be picked up more than once.

Stratified sampling. This method is more complex and refers to splitting the data into many partitions and getting random samples for each partition. In cases where the classes are disproportional, this approach keeps the proportional number of classes according to the original data.

Progressive sampling. This last technique starts with a small size and keeps increasing the dataset until a sufficient sample size is acquired.

Data Transformation: Converting the Data to the Same Structure&nbsp;

Transformation for categorical variables

Minmax scaler/ normalization

Others: The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MaxAbsScaler.html#sklearn.preprocessing.MaxAbsScaler" target="_blank" rel="noreferrer noopener">maxAbs scaler, </a><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html#sklearn.preprocessing.RobustScaler" target="_blank" rel="noreferrer noopener">robust scaler, </a><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html#sklearn.preprocessing.PowerTransformer" target="_blank" rel="noreferrer noopener">power transformer scaler</a>

The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MaxAbsScaler.html#sklearn.preprocessing.MaxAbsScaler" target="_blank" rel="noreferrer noopener">maxAbs scaler</a>: This technique takes the absolute maximum value of the feature and divides each record by this max value, scaling the data in the range of -1 and 1.

The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html#sklearn.preprocessing.RobustScaler" target="_blank" rel="noreferrer noopener">robust scaler</a>: This technique removes the median from the data and scales it using the interquartile range (IQR). As the name suggests, it's robust to outliers.

The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html#sklearn.preprocessing.PowerTransformer" target="_blank" rel="noreferrer noopener">power transformer scaler</a>: This technique changes the data distribution, making it more like a normal distribution. It's used most with heteroscedasticity data, which means that all variables don't have the same variance.

Handling Data with Unequal Distribution of Classes (Imbalanced Data)

Step 1: Start by analyzing and treating the correctness of attributes, like identifying noise data and any structural error in the dataset.

Step 2: Analyze missing data, along with the outliers, because filling missing values depends on the outliers analysis. After completing this step, go back to the first step if necessary, rechecking redundancy and other issues.

Step 3: The process of adding domain knowledge into new features for your dataset. If you don't get any useful new features for your project, don't worry and avoid creating useless features.

Step 4: Use this step for transforming the features into the same scale/unit. If you doubt that the model you will be using needs the data on the same scale, then apply it. It won't negatively affect the models that don't need data transformation.

Step 5: This stage avoids the curse of dimensionality, so if you think you're having this problem, you must apply this step in your pipeline. It comes after data transformation because some of the techniques (e.g., PCA) need transformed data.

Step 6: The last part before moving to the model phase is to handle the imbalanced data. Also, there are some specific metrics for calculating the model's performance when you have this issue in your data.

height

width

level

A man using a remote drone to clean a database

Puzzle of an artificial brain with circuits on the back

What is Bias in Machine Learning? Real-World Examples That Show the Impact of AI Bias

A laptop with charts, graphs and analysis of data

What is Exploratory Data Analysis? Steps & Examples

An example of a data preprocessing technique is data cleaning. It is the process of detecting and fixing bad and inaccurate observations from your dataset. 

What is an example of a data preprocessing technique?

If you skip the data preprocessing step, it will affect your work later on when applying this dataset to a machine learning model. Most of the models can’t handle missing values. By preprocessing the data, you’ll make the dataset more complete and accurate.

Why is data preprocessing important?

Data cleaning: cleaning out meaningless data, incorrect records or duplicate observations, adjusting or deleting observations that have missing data points, and fixing typos and inconsistencies in the dataset. Secondly, we need to reduce the amount of attributes/features so as not to affect the model’s performance when we feed it the dataset.

Back-end Developers

Front-end Developers

Mobile Developers

Data Preprocessing Techniques: 6 Steps to Clean Data in Machine Learning

What is an example of a data preprocessing technique?

Why is data preprocessing important?

What are the major steps of data preprocessing?

Looking to hire?

Join our newsletter

Read Next

What is Bias in Machine Learning? Real-World Examples That Show the Impact of AI Bias

What is Exploratory Data Analysis? Steps & Examples

Let's work together on your next software project