Fig1. Sportsbook Data

I have translated this data into a JSON wen can use to load the needed stats for our model. Let's use the following definition to hold out the data:

1 - Load the data into a JSON Object

pho_vs_okc = {
  "matchup": {
    "date": "2025-12-10",
    "time_est": "7:30 PM",
    "venue": "Paycom Center, Oklahoma City, OK, USA"
  },
  "betting_consensus": {
    "spread": {
      "phoenix": "+14.5",
      "oklahoma_city": "-14.5",
      "consensus": {
        "phoenix_pct": 61,
        "oklahoma_city_pct": 39
      }
    },
    "total": {
      "over_pct": 69,
      "under_pct": 31
    },
    "moneyline": {
      "phoenix": '',
      "oklahoma_city": ''
    }
  },
  "offensive_team_records": {
    "phoenix_suns": {
      "points_per_game": 115.88,
      "points_against_per_game": 113.46,
      "first_half_points": 57.00,
      "fg_pct": 46.74,
      "ft_pct": 79.03,
      "three_pt_pct": 36.76,
      "off_rebounds": 8.96,
      "fg_made": 42.17,
      "ft_made": 16.96,
      "three_pt_made": 14.58
    },
    "oklahoma_city_thunder": {
      "points_per_game": 123.04,
      "points_against_per_game": 106.88,
      "first_half_points": 60.38,
      "fg_pct": 49.72,
      "ft_pct": 82.98,
      "three_pt_pct": 37.37,
      "off_rebounds": 12.83,
      "fg_made": 44.13,
      "ft_made": 20.92,
      "three_pt_made": 13.88
    }
  },
  "defensive_opponent_stats": {
    "phoenix_suns": {
      "opp_points_per_game": 113.46,
      "opp_rebounds": 11.17,
      "opp_fg_pct": 47.61,
      "opp_ft_pct": 81.00,
      "opp_three_pt_pct": 35.87
    },
    "oklahoma_city_thunder": {
      "opp_points_per_game": 106.88,
      "opp_rebounds": 11.04,
      "opp_fg_pct": 42.80,
      "opp_ft_pct": 76.20,
      "opp_three_pt_pct": 36.93
    }
  },
  "best_available_odds": {
    "spread": {
      "phoenix": "+14.5 -110",
      "oklahoma_city": "-14.5 -105"
    },
    "total": {
      "over": "225.5 -110",
      "under": "225.5 -110"
    },
    "moneyline": {
      "phoenix": "+675",
      "oklahoma_city": "-1000"
    }
  }
}

From the json object we will be using a limited set of attributes for our model:

• points_per_game (ppg): defines how many points per game the team usually achieves.

• points_against_per_game (papg): average of points that the opposite team scores. This is a measure of how good their defense is.

• fg_pct (fgp): Field goal percentage is a term used to estimate how many points a team does at any part of the court (like 1 pts and 3 pts shots combined)

• ft_pct (ftp): This is similar to fg_pct , but only on free throws.

• three_pt_pct (tpp): This is the three-point percentage per game.

• off_rebounds (ofr): Offensive rebounds; this indicates an increase in possession. A team with better possession wins the game?

• odds_feature (oft): this is a new feature to be created as a difference of the Market probability between Pho and Okc with the vig removed (we will discuss this later in this post).

• home_feature (hmf): -1 if pho is the away team.

Lets load all the raw data into some team-named variables:

2 - Load initial team-named variables

import math

# extract stats

pho = pho_vs_okc["offensive_team_records"]["phoenix_suns"]
okc = pho_vs_okc["offensive_team_records"]["oklahoma_city_thunder"]

pho_def = pho_vs_okc["defensive_opponent_stats"]["phoenix_suns"]
okc_def = pho_vs_okc["defensive_opponent_stats"]["oklahoma_city_thunder"]

odds = pho_vs_okc["best_available_odds"]["moneyline"]

3 - Baseline Logistic Regression Model

$$\begin{aligned} \text{Let } & \mathbf{x} = \big( \mathrm{ppg},\ \mathrm{papg},\ \mathrm{fgp},\ \mathrm{ftp},\ \mathrm{tpp},\ \mathrm{ofr},\ \mathrm{oft},\ \mathrm{hmf} \big) \\[6pt] \text{and } & \mathbf{w} = \big( w_{\mathrm{ppg}},\ w_{\mathrm{papg}},\ w_{\mathrm{fgp}},\ w_{\mathrm{ftp}},\ w_{\mathrm{tpp}},\ w_{\mathrm{ofr}},\ w_{\mathrm{oft}},\ w_{\mathrm{hmf}} \big), \ \text{with intercept } w_{0}. \end{aligned}$$

$$z(\mathbf{x}) = w_{0} + w_{\mathrm{ppg}} \cdot \mathrm{ppg} + w_{\mathrm{papg}} \cdot \mathrm{papg} + w_{\mathrm{fgp}} \cdot \mathrm{fgp} + w_{\mathrm{ftp}} \cdot \mathrm{ftp} + w_{\mathrm{tpp}} \cdot \mathrm{tpp} + w_{\mathrm{ofr}} \cdot \mathrm{ofr} + w_{\mathrm{oft}} \cdot \mathrm{oft} + w_{\mathrm{hmf}} \cdot \mathrm{hmf}.$$

$$P(\text{Pho wins} \mid \mathbf{x}) = \sigma\!\left( z(\mathbf{x}) \right) = \frac{1}{1 + e^{-z(\mathbf{x})}}.$$

$$\hat{y} = \begin{cases} 1, & \text{if } P(\text{Pho wins} \mid \mathbf{x}) \ge 0.5, \\[6pt] 0, & \text{otherwise}. \end{cases}$$

Our model is composed of two main components: a linear relationship between the defined variables in x and their weights in w so that $$z(x) = w_0 + xw$$ where w0 is the intercept, x are our variables from the JSON and w are weights calibrated manually. The second component is the logistic function $$\sigma(z(x))$$ which estimates the probability that PHO wins the game. This means that the probability that OKC wins the game is $$1 - \sigma(z(x))$$.

Why this might work?

Well, this model combines all the selected stats into one single score.

Think of the stats from the JSON — things like field-goal percentage, free-throw percentage, points per game, rebounds, turnovers, and so on. Each of these stats gives us a clue about how strong each team is.

The model takes these numbers, multiplies each one by a weight that tells us how much that stat matters, and adds everything together. The result is a single score:

• A positive score means stats in favor of PHO, and a negative score means stats in favor of OKC

This score doesn’t mean “probability” yet — it’s just a raw measure of which team looks stronger on paper. To convert that raw score into something meaningful, we pass it through a “squashing” function. This function takes any number — whether extremely high or extremely low — and compresses it into a value between 0 and 1.

You can think of it like a dial:

• If the score strongly favors PHO → the dial turns closer to 1.0

• If the score is neutral → the dial settles around 0.5

• If the score favors OKC → the dial moves closer to 0.0

That final value is: The estimated probability that PHO wins the game, because the logictic function is a one-vs all classifier.

4 - Convert Moneyline into market probabilities with no vig

This algorithm takes betting moneyline odds and turns them into clean probabilities of each team winning. It begins by converting each team’s American moneyline into an “implied probability.” Positive moneylines mean the team is an underdog, so the formula uses 100 divided by the moneyline plus 100. Negative moneylines mean the team is a favorite, so the formula uses the absolute value of the line divided by that value plus 100. This gives a raw probability for each team, but sportsbooks include extra margin called the vig, so these raw probabilities add up to more than one. To correct this, the algorithm adds both raw probabilities to find the total vig, then divides each team’s raw probability by that total. After this division, the probabilities are properly scaled so they add up to one and represent the true implied chance of each team winning without the sportsbook’s built-in margin.

The following code makes the convertion and creates the odds_feature as a difference between the clean probabilities between pho and okc with no vig. This is a feature that establishes what the sportbook believes is going to win, but we will penalize this score with a weight so it does not remove importance of other variables.

# Convert moneyline odds to Market-implied probabilities

def implied_prob_from_moneyline(ml):
    """Convert American moneyline to raw implied probability."""
    if ml > 0:  # e.g. +675
        return 100 / (ml + 100)
    else:       # e.g. -1000
        return abs(ml) / (abs(ml) + 100)

pho_raw = implied_prob_from_moneyline(int(odds["phoenix"]))
okc_raw = implied_prob_from_moneyline(int(odds["oklahoma_city"]))

# Normalize (remove vig)
vig = pho_raw + okc_raw
pho_prob = pho_raw / vig
okc_prob = okc_raw / vig

# Create the odds feature for the model
# Is negative if OKC is favored Is positive if Phoenix is favored
odds_feature = pho_prob - okc_prob

5 - Load all variables X (X-Delta)

Now we load our features as the difference between all selected stats for each team this is the first step toward creating the feature vector X

delta_points = pho["points_per_game"] - okc["points_per_game"]
delta_points_against = pho["points_against_per_game"] - okc["points_against_per_game"]
delta_fg_pct = pho["fg_pct"] - okc["fg_pct"]
delta_ft_pct = pho["ft_pct"] - okc["ft_pct"]
delta_3pt_pct = pho["three_pt_pct"] - okc["three_pt_pct"]
delta_off_reb = pho["off_rebounds"] - okc["off_rebounds"]
home_feature = -1  # Phoenix is the away team

6 - Define some cherry-picked weights W for each value of Xi

Its important to mention that logistic regression is an algorithm that when exposed to multiple rows of data it can automatically estimate the values of W as an optimization problem. In this case we only have two sets of data from the sportsbook, one line of stats for PHO and one for OKC, so optimization is this case is not an option. This means that we need to place some values for each value of w so that they makes sense. This process is usually done by an expert in the NBA, so because I am not, I used some LLM to help me estimate these parameters. This is the resultant expression:

# Define the model weights (hand-built / interpretable) 

weights = {
    "intercept": 0.0,
    "points_per_game": 0.12, # weight defined to control best predictor (points)
    "points_against_per_game": -0.10, # how good team defense is
    "fg_pct": 0.08, # field goald percentage / almost as important as ppg
    "ft_pct": 0.04, # free throw percentage
    "three_pt_pct": 0.03, # three points percentage
    "off_rebounds": 0.06, # offensive rebounds (increase posession)
    "odds_feature": 1.50, # delta between pho_prob - okc_prob with no vig
    "home_feature": -0.60 # tells if team A (in this case Pho) is home or not
}

In a logistic regression model, each weight tells you how strongly a feature pushes the prediction toward a win or a loss. The farther a weight is from zero, the more influence that feature has on the final probability.

When a weight is close to 1, it means that increasing that feature pushes the model strongly toward predicting a higher probability of winning. A positive weight acts like a “boost.” Every time that feature increases by one unit, the model becomes noticeably more confident that the team will win. A weight near 1 means the model treats that feature as very important in signaling success.

When a weight is close to –1, it means that increasing that feature pushes the model strongly toward predicting a lower probability of winning. A negative weight acts like a “penalty.” As that feature increases, the model becomes more confident the team will lose. A weight near –1 means the model believes that feature strongly signals weakness.

Weights near zero barely affect the prediction at all, because they don’t push the probability up or down in a meaningful way.

So in simple terms, values near 1 are strong positive influencers toward winning, values near –1 are strong negative influencers toward losing, and values near zero don’t matter much to the model.

7 - Mix all ingredients and get the Final Probability

# Logistic function to convert score to win probability

def logistic(x):
    return 1 / (1 + math.exp(-x))

pho_win_prob = logistic(score)
okc_win_prob = 1 - pho_win_prob

This simple logictic function then estimates the probabilities that either one or the other team will win.

• pho_win_prob: 6.34%

• okc_win_prob: 93.65%

So the model concluded that based out on the defined weights and the selected stats, OKC is the clear winner. But, did OKC won? Yes it did! look at the game results from CBS Sports

Oklahoma Thunder was the winner as predicted by the model.

8 - Was the model lucky?

We dont know for sure. There is no certainty this model works unless we keep executing this code on more games and see if in the long run, we are always right, or wrong.

Behind the game of chess lies an interesting and extensive history dating approximately between the 10th and 11th centuries, when the game entered Europe as an evolution of chaturanga from the Gupta Empire (India, 6th century). During the Middle Ages and the Renaissance, reforms were made to the game, introducing the queen, the bishop, and castling. The 19th century marked the beginning of competitive chess, with the first international tournament held in London, where Adolf Anderssen was crowned champion. The 20th century served to professionalize chess; organizations such as FIDE (the International Chess Federation) were founded, and the concept of the “Grandmaster” was established.

An International Grandmaster (GM) is a title awarded by FIDE to players who demonstrate passion and achieve at least 2600 Elo points, are active as international competitors participating in tournaments and official matches against other high-profile players (including other GMs), and are capable of playing multiple games in tournaments.

The GM title is more than just an award; it has also been used as a symbol of intellectual superiority (and still is), especially during the Cold War, when the USSR used chess as a tool of discord to reinforce the country’s image, mainly against the United States and the rest of the world.

Although the rules of chess, as explained earlier, are simple, the implications of moves and strategies are highly complex. Numerically speaking, according to mathematician Claude Shannon, a chess game is estimated to have 10¹²⁰ possible games—more than the number of atoms in the observable universe. The sheer number of possibilities outlines a horizon of unimaginable complexities for humans, who cannot possibly know or understand all the alternatives. This has led to hundreds of books on openings and strategies developed throughout the history of chess.

Thus, the GM is, among all players in the world, the one with the greatest knowledge and practice—capable of challenging even the most difficult opponents in a fierce battle for victory. Historically, there have been great masters who left their mark, such as Wilhelm Steinitz, the first official world champion (1886); Emanuel Lasker, who held the title for 27 years (1894–1921); and the Cuban José Raúl Capablanca (1888–1942), whose magic on the board dazzled the world with his wonderful style.

During the 20th century, chess hegemony belonged to the Soviet Union, where numerous chess stars emerged, including Mikhail Tal (Latvia), world champion in 1960 and 1961; Anatoly Karpov, world champion from 1975 to 1985 and again from 1993 to 1999; and the greatest representative of Russian chess (and according to some experts, the world), Garry Kasparov, world champion from 1985 to 2000.

International chess also had its luminaries, such as Bobby Fischer (U.S.), world champion from 1972 to 1975, and Nigel Short (England), who challenged for the world title in 1993, losing to Kasparov.

The world of technology, especially video games, was not left behind and began developing programs capable of playing chess. In 1950, Claude Shannon published a paper on the possibility of programming a computer to play chess. In 1951, Alan Turing (the father of Artificial Intelligence) designed the first algorithm to play chess before it could even be programmed into a computer. In 1957, IBM developed the IBM 704 Chess Program, one of the first executable programs capable of playing chess, and in 1967, Mac Hack VI from MIT became the first program to defeat a human player in an official tournament.

From there, chess programs grew more complex and capable, but they were never strong enough to defeat a GM. The sheer number of moves combined with the processing and memory limitations of computers at the time prevented the development of algorithms able to analyze many moves ahead. One approach used was deep search, which considered multiple alternatives three to five moves deep for all available pieces. This caused slow responses, as the computer took too long to determine the next move that maximized winning chances. Lack of positional intuition, limited opening databases, poor evaluation of complex pawn structures, static piece value assessments, and almost no dynamic compensation (limited ability to sacrifice pieces for greater gains) were typical shortcomings of programs of the era. Intermediate and advanced players knew this, which created predictability in the computer’s moves.

Gradually, chess programs improved these limitations, but overall, they remained machines without real understanding of the problem.

In 1989, IBM invested in developing a machine capable of executing millions of moves per second, something unique for the time, with help from Carnegie Mellon University students. The first version was ready in 1995 and was officially presented as the supercomputer “Deep Blue.” Its VLSI processors were specifically created for chess, capable of processing and calculating many positions quickly. Minimax algorithms were used to reduce the number of positions to explore, and chess heuristics were designed with the help of grandmasters. In addition, the algorithm was reinforced with a database of thousands of openings and strategies, allowing Deep Blue to become the final frontier of computational chess.

In 1997, IBM made a bold move and challenged grandmaster Garry Kasparov to a set of chess games. The conditions set by IBM were: six classical games, forty moves in two hours, and then an additional hour for twenty more moves. The games were played under FIDE rules, and Kasparov could make claims or moves as in normal play. However, Kasparov had only limited access to Deep Blue; he could not touch or modify it. IBM controlled the input of moves into the computer exclusively.

The event took place from May 3 to 11, 1997, at the Equitable Center in New York City, where Garry Kasparov lost the series 3½–2½ in favor of Deep Blue. This was the first time in history that a GM lost to a computer.

After the event, Kasparov reacted explosively, pointing out that the moves—especially in the second game—were too creative to have come from a machine. This sparked a heated debate over whether IBM had other GMs manipulating Deep Blue’s moves. Adding to the mystery, IBM denied Kasparov a rematch, cementing this milestone in the company’s (and the world’s) history as the moment when Deep Blue defeated the grandmaster of grandmasters.

IBM’s supercomputer was perhaps the foundation of more advanced, general-purpose computational capabilities, reigniting investment and research into machines not only capable of solving specific problems like chess but also of going beyond games with finite rules and domains, to perhaps, someday, compete intellectually with humans.

The miniaturization of computers, the drop in memory costs, instant access to massive data storage, the exponential growth of processing power since the 1980s, and the relentless globalization of scientific research have created fertile ground for new technologies. In the first quarter of the 21st century alone, more progress has been made in developing intelligent technologies than in the hundred years before 2000.

In the blink of an eye, today we have self-driving cars, vacuum cleaners that sweep the house and return to their charging bases with minimal human intervention, smartphones capable of accessing all human knowledge and answering questions instantly in real time. Today, machines generate music, art, and poetry; process medical diagnoses; help scientists discover new proteins in seconds; and are used to develop medicines and disease treatments. All this happened in a breath and has generated many reactions among people—some filled with awe, others with fear at the evolution of technology, which they barely understand.

The fear of the population toward new technologies and advancements is something humanity has experienced before, from the invention of the steam engine by James Watt in 1769, to electricity and the light bulb by Edison and Swan in 1879, the development of computers with ENIAC in 1945, and the rise of intelligent machines and big data around 2010.

Are you scared?

The Matrix

If you have a basic understanding of algebra, you should already know that a Matrix is an object with a rectangular shape of values, such as rows and columns, to describe something. For example, I can create a matrix of 5 rows and 3 columns to represent movie preferences for five different people. In the following example, we will use a table representing a matrix M for movies. One means the person likes the movie, and zero otherwise.

Mathematically speaking, we can represent this matrix like:

$$M = \begin{pmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \\ a_{41} & a_{42} & a_{43} \\ a_{51} & a_{52} & a_{53} \end{pmatrix}$$

Where ( a_11 ) is the first cell where Juan likes Kung-fu panda. If we replace each cell with the table values, our matrix will look like this:

$$A = \begin{pmatrix} 1 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 1 & 1 \\ 1 & 1 & 0 \end{pmatrix}$$

With this, we get into a more concrete definition of a matrix: a rectangular array of numbers, symbols, or expressions arranged in rows and columns. It's an object of 1 or 2 dimensions that describes something.

The Tensor

It is like a matrix but with high dimensionality. Yes, we can now build a 3D or 4D matrix! The best way to explain a matrix is by describing an image in its three channels: Red, Green, and Blue.

Let's assume we have an image that is full color and is just 5 x 5 pixels in size. An image is explained as three different matrixes altogether. The first one describes the image as a set of pixels in Red color, the second one as a set of pixels in Green, and the third one as a set of pixels in Blue. The combination of all these matrixes is the final image.

A tensor is then an object that can hold three matrixes of size 5 x 5 pixels. We usually describe this tensor a T with the shape: T(5,5,3), where the 3 represents each color channel.

The mathematical description of this tensor looks like this:

$$\mathcal{T} = \begin{pmatrix} \text{Red Channel} & \begin{pmatrix} r_{11} & r_{12} & r_{13} & r_{14} & r_{15} \\ r_{21} & r_{22} & r_{23} & r_{24} & r_{25} \\ r_{31} & r_{32} & r_{33} & r_{34} & r_{35} \\ r_{41} & r_{42} & r_{43} & r_{44} & r_{45} \\ r_{51} & r_{52} & r_{53} & r_{54} & r_{55} \end{pmatrix}, \\ \text{Green Channel} & \begin{pmatrix} g_{11} & g_{12} & g_{13} & g_{14} & g_{15} \\ g_{21} & g_{22} & g_{23} & g_{24} & g_{25} \\ g_{31} & g_{32} & g_{33} & g_{34} & g_{35} \\ g_{41} & g_{42} & g_{43} & g_{44} & g_{45} \\ g_{51} & g_{52} & g_{53} & g_{54} & g_{55} \end{pmatrix}, \\ \text{Blue Channel} & \begin{pmatrix} b_{11} & b_{12} & b_{13} & b_{14} & b_{15} \\ b_{21} & b_{22} & b_{23} & b_{24} & b_{25} \\ b_{31} & b_{32} & b_{33} & b_{34} & b_{35} \\ b_{41} & b_{42} & b_{43} & b_{44} & b_{45} \\ b_{51} & b_{52} & b_{53} & b_{54} & b_{55} \end{pmatrix} \end{pmatrix}$$

The T tensor is the 5x5x3 object. We can add another dimension called Alpha for opacity if we like. You can keep adding as many dimensions as your heart desires.

Sample Tensor in Python with No Library

The following code creates a custom-made Tensor to display an image for the three color matrixes. (No Pytorch or Tensorflow)

import numpy as np
import matplotlib.pyplot as plt

# Create a 5x5 matrix for the red channel
red_channel = np.array([
    [0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0],
    [0, 0, 255, 0, 0],
    [0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0]
])

# Create a 5x5 matrix for the green channel
green_channel = np.array([
    [0, 0, 0, 0, 0],
    [0, 255, 255, 255, 0],
    [0, 255, 0, 255, 0],
    [0, 255, 255, 255, 0],
    [0, 0, 0, 0, 0]
])

# Create a 5x5 matrix for the blue channel
blue_channel = np.array([
    [255, 255, 255, 255, 255],
    [255, 0, 0, 0, 255],
    [255, 0, 0, 0, 255],
    [255, 0, 0, 0, 255],
    [255, 255, 255, 255, 255]
])

# Stack the color channels to create a 5x5x3 tensor
image_tensor = np.stack([red_channel, green_channel, blue_channel], axis=-1)

# Plot the image
plt.imshow(image_tensor)
plt.title('5x5 Image Pattern')
plt.axis('off')  # Hide axes
plt.show()

The code above generated the following image:

This shows how tensor T with pixels on and off for each channel was able to represent something in the image. The matplotlib uses all channels stacked together (as a sequence) to display the image.

New Tensor for Alpha Channel

We will slightly change the code to make the Blue tensor all blue, not just the pixels from the boundaries. This adds a new tensor, Alpha, who is responsible for showing opacity.

import numpy as np
import matplotlib.pyplot as plt

# Create the 5x5 matrices for each color channel
red_channel = np.array([
    [0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0],
    [0, 0, 255, 0, 0],
    [0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0]
])

green_channel = np.array([
    [0, 0, 0, 0, 0],
    [0, 255, 255, 255, 0],
    [0, 255, 0, 255, 0],
    [0, 255, 255, 255, 0],
    [0, 0, 0, 0, 0]
])

# Create a 5x5 matrix for the blue channel with all pixels set to blue (255)
blue_channel = np.full((5, 5), 255, dtype=np.uint8)

# Create a 5x5 matrix for the alpha channel
alpha_channel = np.array([
    [255, 255, 255, 255, 255],
    [255, 128, 128, 128, 255],
    [255, 128, 128, 128, 255],
    [255, 128, 128, 128, 255],
    [255, 255, 255, 255, 255]
])

# Normalize alpha values to range [0, 1]
alpha_normalized = alpha_channel / 255.0

# Blend the red channel with 50% opacity
red_blended = red_channel * alpha_normalized

# Stack the color channels to create a 5x5x3 tensor
image_tensor = np.stack([red_blended, green_channel, blue_channel], axis=-1)

# Plot the image
plt.imshow(image_tensor)
plt.title('5x5 Image with Blue Channel Fully Set to 255')
plt.axis('off')  # Hide axes
plt.show()

This code will display the color red at 50% opacity. Please note that the opacity tensor is used as a filter; this means it is multiplied by one or more channels to provide the desired effect.

Remember that there is a blue and green layer, so the colors with 50% opacity will be shown as follows:

Summary

Tensors are multi-dimensional arrays that can hold much more information than a simple 1D or 2D matrix. Tensors are used today in computer vision, deep learning, image and video representations, natural language processing, robotics, control systems, reinforcement learning, and recommender systems, among many others!

We usually need to do many calculations using tensors. Some libraries can help speed up the process and provide many functionalities, such as Tensorflow and PyTorch.

If you want to see tensors used with TensorFlow to build a movie recommender system, please review my post on Movie Recommendations.

In machine learning, Bias refers to the difference between the predicted and expected values. Bias can also be defined as the error that is introduced by approximating a real-life problem. In supervised learning, we usually have a training and a validation set. The training set produces a model later used to predict the validation set. For example, a regression model that predicts the probability of getting into a pre-diabetic state based on glucose levels will learn the relationship between the levels and the labeled probability from the training set. The validation set contains real-life assessments of patients' glucose and their diagnosis as a probability. We say a model has low bias if the predicted probabilities are close to those in the validation set. Our objective while producing ML models is to reduce Bias as much as possible.

What's Variance?

Variance measures how much a prediction changes if we change the dataset. In other words, variance targets how strong our model is in making good predictions, given that we are using different datasets. Let's use the Glucose example. Suppose we collect data from San Jose, California, for our training set and train the model. Can the model predict successfully if we use a validation set from Albuquerque, New Mexico? The answer is probably not, and the reason is that both places have different diets that contain more or less sugar. We need to produce ML models that have low variance. This means that regardless of where we are testing the model (people from New Mexico, Ohio, New York, or California), the model should be able to withstand the changing assumptions (and dietary conditions) of the populations under evaluation.

How are They Related?

Bias and Variance are independent, but they affect the models simultaneously. A low-bias model produces good predictions. A high bias produces predictions that are far away from the expected values. A model with low variance is able to produce good predictions regardless of the incoming data, and a high variance model will cause inconsistencies in the results no matter where we test out the model. We always aim for low bias and low variance.

Why is this important in Machine Learning?

The bias-variance trade-off describes the balance we should consider when building machine learning models. Bias manages the model error (predictive), and the variance measures model sensitivity to changing data. This tells us something very important about this trade-off and how to deal with it: to manage Bias, we look at the model characteristics (features used and their quality, types of machine learning algorithms, hyperparameter tuning, etc.), and with Variance, we look for the quality of the data, sampling techniques, dimensional reduction, regularization, trustability of the data sources and data generalization. Let's look at each one separately.

What to do to reduce Bias?

Reducing model bias is crucial for improving the performance of a machine-learning model. Here are the main techniques to reduce model bias:

1. Increase Model Complexity:

   • Use more complex algorithms or models (e.g., switching from linear regression to polynomial regression or from a simple decision tree to a random forest).

   • Increase the number of features or use feature engineering to create more informative features.

2. Ensemble Methods:

   • Use ensemble techniques like bagging, boosting, or stacking to combine the predictions of multiple models, which can reduce bias.

   • Examples include Random Forests (bagging) and Gradient Boosting Machines (boosting).

3. Use Better Features:

   • Improve feature selection and engineering to include more relevant features that capture the underlying patterns in the data.

   • Use domain knowledge to create new features that might better explain the variance in the target variable.

4. Reduce Underfitting:

   • Ensure that the model is not too simple for the data. This might involve using more sophisticated algorithms or adding more parameters.

   • Avoid overly aggressive regularization, which can excessively penalize the complexity of the model, leading to high bias.

5. Data Augmentation:

   • For problems like image recognition, data augmentation techniques (e.g., rotations, translations, flips) can create additional training examples, helping the model learn better.

6. Increase Training Data:

   • Collect more or better-quality data to provide the model with more information about the underlying patterns.

   • More data can help the model generalize better, reducing bias.

7. Hyperparameter Tuning:

   • Carefully tune hyperparameters to find the best settings that reduce bias without introducing too much variance.

   • Techniques like grid search, random search, or Bayesian optimization can be used for effective hyperparameter tuning.

8. Addressing Data Imbalance:

   • For classification problems, ensure that the classes are balanced. Techniques like oversampling, undersampling, or synthetic data generation (e.g., SMOTE) can help.

9. Transfer Learning:

   • Use pre-trained models (especially in deep learning) that have been trained on large datasets. Fine-tuning these models on your specific dataset can help reduce bias.

10. Regularization with Care:

    • Use regularization techniques like L1 or L2 regularization judiciously. While they help in reducing variance, overly strong regularization can increase bias.

By applying these techniques, you can reduce model bias, helping the model to capture the underlying patterns in the data more accurately, which leads to better performance and generalization.

What to do to reduce Variance?

Reducing model variance is crucial for improving the generalizability of a machine-learning model. Here are the main techniques to reduce variance:

1. Simplify the Model:

   • Use simpler models with fewer parameters, such as linear regression, instead of polynomial regression or smaller decision trees.

   • This helps avoid overfitting by reducing the model’s capacity to fit the noise in the training data.

2. Regularization:

   • Regularization techniques like L1 (Lasso) or L2 (Ridge) regularization should be applied to penalize the complexity of the model.

   • Regularization discourages the model from fitting the noise in the training data, leading to a more generalizable model.

3. Ensemble Methods:

   • Ensemble techniques, such as bagging (e.g., Random Forests), can reduce variance by averaging the predictions of multiple models.

   • Boosting methods (e.g., Gradient-Boosting Machines) also help, but they primarily focus on reducing bias and can still improve generalization.

4. Cross-Validation:

   • Use cross-validation techniques to ensure that the model’s performance is consistent across different subsets of the data.

   • K-fold cross-validation helps understand how the model generalizes to unseen data and reduces the risk of overfitting.

5. Increase Training Data:

   • Collect more training data to provide the model with a better representation of the underlying distribution.

   • More data helps in reducing the model’s sensitivity to specific training examples.

6. Data Augmentation:

   • For tasks like image recognition, data augmentation techniques (e.g., rotations, translations, and flips) can artificially increase the size of the training dataset.

   • This helps the model to learn invariant features and reduces overfitting.

7. Pruning (for Decision Trees):

   • Apply pruning techniques to decision trees to remove unimportant branches that do not generalize well to new data.

   • Pruning helps simplify the model and reduce variance.

8. Early Stopping (for Neural Networks):

   • Use early stopping during training to halt the training process when performance on a validation set starts to degrade.

   • This prevents the model from overfitting to the training data.

9. Dropout (for Neural Networks):

   • Use dropout regularization in neural networks, where randomly selected neurons are ignored during training.

   • This prevents the network from becoming overly reliant on specific neurons and helps in reducing overfitting.

10. Bagging (Bootstrap Aggregating):

    • Train multiple models on different subsets of the data created through bootstrapping and aggregate their predictions.

    • This helps in reducing the model’s variance by averaging out the noise.

11. Parameter Tuning:

    • Carefully tune hyperparameters using grid, random, or Bayesian optimization techniques.

    • Proper hyperparameter tuning can help find the optimal balance between bias and variance.

By applying these techniques, you can reduce your model's variance, which leads to better generalization and improved performance on unseen data.

Where is overfitting in the trade-off?

Overfitting is a phenomenon in machine learning where a model learns the details and noise in the training data to such an extent that it negatively impacts the model’s performance on new data. In the context of the bias-variance trade-off, overfitting is primarily associated with high variance.

So, suppose the model has high variance, and the algorithm used in the training set is learning from all that noise. In that case, you will find that the accuracy will drop significantly when predicting the validation set.

Solutions to Overfitting

To mitigate overfitting and find a better balance in the bias-variance trade-off, the following techniques can be used:

1. Simplify the Model: Use a simpler model with fewer parameters.

2. Regularization: Add a regularization term to the loss function to penalize complex models (e.g., L1 or L2 regularization).

3. Cross-Validation: Use techniques like k-fold cross-validation to ensure the model’s performance is consistent across different subsets of the data.

4. More Training Data: Collect more training data to help the model learn the underlying patterns more effectively.

5. Pruning: For decision trees, apply pruning techniques to remove branches that do not provide additional power in predicting the target variable.

6. Dropout: In neural networks, use dropout to randomly ignore a subset of neurons during training to prevent the network from becoming too reliant on specific neurons.

7. Early Stopping: Halt the training process once the performance on a validation set starts to degrade.

8. Data Augmentation: For tasks like image recognition, use data augmentation to artificially increase the size and variability of the training dataset.

Summary

The variance-bias tradeoff is crucial because it helps build accurate and generalizable models, ensuring they perform well on unseen data. Balancing bias and variance is key to effective machine learning.

In addition to the tools for custom labeling, a huge community also publishes datasets and pre-trained models for object detection, classification, segmentation, keypoint detection, and semantic segmentation, among many others.

Today, we will use a model to classify an image for Rock, Paper, or Scissors using the "rock-paper-scissors-sxsw/14" model.

This is the image we will use for our little experiment; feel free to download it and save it as rock.jpg

Open Google Colab

Go to google colab and create a new notebook. Put the rock.jpg at the root directory.

Loading Libraries

To use Roboflow's API, we need to create an account and get the API key. The following code will install the inference library.#!pip install inference-sdk

!pip install inference-sdk

Now, let's load the required libraries. We will use inference_sdk to call the pre-trained model from Roboflow. cv2 will be used to load the rock.jpg image and create the bounding box, and matplotlib will be used to display the image in the lab notebook.

import inference_sdk
import cv2
import matplotlib.pyplot as plt

Call Roboflows Inference API

This code is provided by Roboflow as part of the snippets they offer for each model. This one uses the InferenceHTTPClient to call the model hosted on Roboflow remotely. The client.infer method will stream the image to the Roboflow API and return a json with the bounding box and the classification and confidence score information.

from inference_sdk import InferenceHTTPClient

CLIENT = InferenceHTTPClient(
    api_url="https://detect.roboflow.com",
    api_key="PUT-YOUR-API-KEY-HERE"
)

result = CLIENT.infer('rock.jpg', model_id="rock-paper-scissors-sxsw/14")

result

The result object is a json that contains the following structure:

{'time': 0.04682349800009433,
 'image': {'width': 1024, 'height': 768},
 'predictions': [{'x': 552.5,
   'y': 383.5,
   'width': 385.0,
   'height': 371.0,
   'confidence': 0.5471271276473999,
   'class': 'Rock',
   'class_id': 1,
   'detection_id': '90730496-ac1c-4f26-af09-b03b33409047'}]}

Custom Annotation with CV2

With the JSON structure, we will use cv2 (OpenCV) to create a rectangle object as the bounding-box and a text object to show the class label and the confidence score.

# Load the image
image = cv2.imread("rock.jpg")

# Get image dimensions
image_width = image.shape[1]
image_height = image.shape[0]

# Get prediction data from JSON
x = result['predictions'][0]['x']
y = result['predictions'][0]['y']
width = result['predictions'][0]['width']
height = result['predictions'][0]['height']

# Calculate coordinates for the bounding box
x_min = int(x - width / 2)
y_min = int(y - height / 2)
x_max = int(x + width / 2)
y_max = int(y + height / 2)

# Draw bounding box on the image
color = (0, 255, 0)  # Green color for the bounding box
thickness = 4
image_with_box = cv2.rectangle(image, (x_min, y_min), (x_max, y_max), color, thickness)

# draw label
text = result['predictions'][0]['class'] + ' ' + str(round(result['predictions'][0]['confidence'],2))
position = (x_min, y_min-10)  # Bottom-left corner of the text (x, y)
font = cv2.FONT_HERSHEY_SIMPLEX
font_scale = 1
color = (255, 0, 0)  # Blue color in BGR
thickness = 2
line_type = cv2.LINE_AA

# Add text to the image
cv2.putText(image, text, position, font, font_scale, color, thickness, line_type)

# Display the annotated image
plt.imshow(image_with_box)
plt.show()

This code translates the coordinates provided by the Roboboflow model to something cv2 can understand. The resultant image is shown below.

Summary

The most complex part is showing the annotation. Roboflow's pre-trained models and tools make it easy to create custom datasets and models trained online with very little coding and very quickly. Roboflow offers a free account with limited functionalities (enough to run this example!). Other plans for computer vision tasks start at $250 USD per month.

So what If I want to do this myself with no Roboflow help? well, you will need to look for an annotation tool (like the COCO Annotator or the Ultralytics Annotator in case you are using one of the YOLO flavors) and an object detection framework such as YOLOv9 to train your models with GPUs from one of your favorite cloud providers such as AWS, Google Cloud or Azure).

Fortunately, companies like Facebook and OpenAI have already invested millions of dollars in producing these technologies, and they placed them on a website called "Hugging Face".

For this post, we will be using the pre-trained model "distil-whisper/distil-small.en," which is able to translate audio arrays into text. There are many different models and versions of the "distil-small.en" model based on the number of parameters. It's said that a model with a higher number of parameters might produce better transcriptions but will also convert audio into extremely long sequences, so the final transcription might take a lot of time, depending on how long the audio file is.

For our case, we will be using the smallest model, which has 166 million parameters to convert a sample audio file of 1:51 minutes.

Download the sample Audio File

Please download this sample audio file from this link. This is an mp3 file, but you are welcome to use your own file. Rename the audio file as "speech.mp3" and place the file at the root of your colab directory.

Nicely done! Now, it's time to code. Surprisingly, the code required for this is very simple but requires some tweaking. Let's get our hands dirty.

Import Libraries

To reproduce this code, you can use Google Colab, so we will need to install the following dependencies:

  !pip install transformers
  !pip install soundfile
  !pip install librosa

• Transformers: This is the main library that allows us to play with thousands of pre-trained models and download them directly from the model objects.

• Soundfile: this library will help us load the wav, flac or mp3 files

• Librosa: Its Librosa not LeviOsA. This library will help us with some pre-processing we need to do to the mp3 file such as converting stereo files to mono and downcasting an audio file to another sampling rate.

Great! Now let's load all the libraries we need:

from transformers.utils import logging
logging.set_verbosity_error()

import soundfile as sf
import io
import numpy as np
import librosa

from transformers import pipeline

Create the Transcription Pipeline

A pipeline is an object that loads a pre-trained model and sets up all the required processing so we don't have to do it ourselves. This is very simple, we just need to create the pipeline object and call the model we will be using for the transcription.

asr = pipeline("automatic-speech-recognition", 
model="distil-whisper/distil-small.en")

This single line of code will download the model and store it in the ASR model variable. We are almost ready to use it. The only thing we need to do is make sure the audio file is compatible with the model, and for that, we will need to transform the audio file with Librosa.

Transforming the Audio File

The "distil-small.en" has been trained in a specific sampling rate. Concretely this model works with audio that is in the 16Hz sampling rate. We can find this out by checking the model sampling rate:

print(asr.feature_extractor.sampling_rate)

output: 16000

Now, its time to check the sampling rate of the audio file "speech.mp3"

audio, sampling_rate = sf.read('speech.mp3')
print(sampling_rate)

output: 44100

No problem, we need to do two things with this audio file. The first one 1) is to make sure the file is mono, as these models work with single-channel audio files and 2) downcast the 44Hz audio file to 16Hz. Let's do exactly that in this single block of code:

audio_transposed = np.transpose(audio)
audio_mono = librosa.to_mono(audio_transposed)

audio_16KHz = librosa.resample(audio_mono,
                               orig_sr=sampling_rate,
                               target_sr=asr.feature_extractor.sampling_rate)

We will transpose our audio file to a single array to manage a united version of the audio file. The transposed array will now be converted to mono using the Librosa to_mono method. That's it; now the file is in a single channel.

The second part is also very simple. Librosa has a resampling method that allows users to change the sampling rate of a mono audio file. The audio_16KHz object now contains the pre-processed audio file that is compatible with the HuggingFace model for audio transcription.

Audio Transcription

This is the most simple part now. We will use the ASR object to translate the audio file into text. We will use some special parameters that allow us to process audio in 30-second chunks. This way, we can process longer audio without error, as the model only processes audio of less than 30 seconds.

asr(
    audio_16KHz,
    chunk_length_s=30, # 30 seconds
    batch_size=4,
    return_timestamps=True,
)["chunks"]

Thats it! just wait some time; you will get a JSON array with the timestamp and text recognized. This is the output of this transcription:

[{'timestamp': (0.0, 7.0),
  'text': ' Community Update December 2019 posted on December 27th 2019 by Frederick Font.'},
 {'timestamp': (7.0, 11.0),
  'text': ' Hi everyone, welcome to a new community update.'},
 {'timestamp': (11.0, 15.0),
  'text': ' Yeah, we know we have not been updating you very regularly lately,'},
 {'timestamp': (15.0, 18.0),
  'text': ' but this does not mean we have not been working hard on free sound.'},
 {'timestamp': (18.0, 25.6),
  'text': ' As has been the case for the last year, we have not been very much concentrated on working on either under the hood improvements'},
 {'timestamp': (25.6, 30.56),
  'text': ' or research type of issues which do not have a clearly visible output in the Fresound'},
 {'timestamp': (30.56, 32.28), 'text': ' website yet.'},
 {'timestamp': (32.28, 37.12),
  'text': ' But we are indeed working on great things which will definitely end up in the platform.'},
 {'timestamp': (37.12, 41.6),
  'text': ' Here is a summary of our current main working threads.'},
 {'timestamp': (41.6, 44.44),
  'text': ' BUG Fixs, General Maintenance and Software updates.'},
 {'timestamp': (44.44, 45.44),
  'text': ' This is a big one as we are about to carry, and software updates.'},
 {'timestamp': (45.44, 49.72),
  'text': ' This is a big one as we are about to carry out necessary software updates.'},
 {'timestamp': (49.72, 55.76),
  'text': ' For nerds, Python, Jango updates, which affect all of our code base and are therefore quite'},
 {'timestamp': (55.76, 56.2), 'text': ' time-consuming.'},
 {'timestamp': (56.2, 57.2), 'text': ' New features.'},
 {'timestamp': (57.2, 63.44),
  'text': ' We are working on new features mostly related to the search page.'},
 {'timestamp': (63.44, 68.56),
  'text': " However, all these new features require a lot of previous research work. Don't forget we're a research"},
 {'timestamp': (68.56, 73.44),
  'text': " institution, so that's what we do best. And again, features need their time to"},
 {'timestamp': (73.44, 78.32),
  'text': " become a reality. The new search features we're working on will allow to cluster"},
 {'timestamp': (78.32, 86.64),
  'text': ' search results as well as adding new filtering options. New front end. Yes, we have not abandoned this one.'},
 {'timestamp': (86.64, 91.52),
  'text': ' It is going very slowly, much more than we thought, but it will eventually become a reality'},
 {'timestamp': (91.52, 93.68), 'text': ' and it is indeed in our roadmap.'},
 {'timestamp': (93.68, 100.16),
  'text': " Oh, and by the way, we've just published a tech-oriented post about Free Sound in the Creative Commons open-source"},
 {'timestamp': (100.16, 101.24), 'text': ' blog.'},
 {'timestamp': (101.24, 103.44), 'text': ' You might want to check that out.'},
 {'timestamp': (103.44, 105.6),
  'text': " And that's it for the short update. Thanks"},
 {'timestamp': (105.6, 109.92),
  'text': ' for a reading and stay tuned for the updates in the coming year. Big happy new'},
 {'timestamp': (109.92, 113.04), 'text': ' year to everyone.'}]

Great Success

Let's suppose we collected some data from some University students. We asked every student their gender (Male = M, Female = F) and their weight. We will build a model that can predict if you are a male or a female based only on your weight. I know this is not the ideal case for this, but this example will help to demonstrate a statistical trick to discriminate between two populations that show different behaviors in the same dataset.

Let's check the data

let's add some libraries we will use in our example:

from matplotlib import pyplot as plt
import numpy as np
import scipy.stats as stats

We will use some fake data to test this out and we will use sklearn to train-test split the data in a 70/30 proportion.

weight = [52, 46, 52, 47, 47, 49, 50, 47, 53, 55, 44, 52, 47, 48, 51, 54, 51, 51, 53, 52, 47, 52, 47, 50, 47, 53, 52, 44, 54, 52, 55, 55, 53, 42, 53, 50, 49, 49, 46, 52, 56, 48, 50, 56, 49, 44, 53, 46, 50, 51, 42, 37, 43, 53, 57, 50, 51, 51, 46, 48, 50, 47, 51, 47, 49, 49, 47, 44, 53, 51, 56, 58, 47, 44, 55, 53, 46, 53, 52, 47, 49, 52, 36, 49, 55, 49, 55, 55, 42, 50, 49, 37, 53, 45, 52, 56, 51, 55, 48, 58, 49, 56, 54, 48, 56, 49, 48, 57, 55, 58, 52, 51, 39, 43, 47, 45, 50, 54, 50, 49, 56, 43, 44, 50, 47, 36, 53, 47, 43, 51, 50, 50, 52, 54, 46, 49, 52, 52, 56, 52, 54, 43, 51, 52, 56, 48, 49, 55, 51, 64, 55, 46, 45, 52, 54, 46, 47, 56, 58, 50, 48, 46, 42, 54, 50, 53, 42, 55, 45, 54, 50, 51, 56, 48, 49, 47, 47, 52, 43, 47, 48, 47, 46, 53, 53, 51, 49, 47, 50, 57, 55, 51, 48, 55, 45, 49, 47, 47, 49, 54, 46, 49, 48, 50, 47, 48, 50, 52, 53, 51, 54, 34, 49, 53, 63, 49, 47, 50, 56, 51, 54, 46, 45, 50, 58, 43, 52, 49, 53, 50, 56, 48, 54, 45, 54, 46, 43, 51, 40, 57, 51, 45, 53, 50, 53, 52, 48, 56, 60, 48, 50, 51, 46, 47, 53, 52, 45, 47, 54, 49, 52, 53, 60, 58, 56, 53, 54, 44, 51, 54, 53, 51, 45, 53, 52, 52, 51, 42, 55, 54, 53, 45, 52, 49, 53, 58, 44, 47, 47, 45, 55, 53, 45, 53, 48, 50, 58, 52, 50, 45, 49, 55, 55, 54, 42, 47, 41, 46, 50, 55, 52, 47, 49, 48, 52, 51, 53, 50, 48, 45, 51, 53, 51, 61, 51, 41, 55, 43, 56, 53, 47, 53, 51, 51, 52, 56, 42, 42, 47, 48, 56, 46, 43, 43, 48, 50, 50, 50, 49, 51, 51, 47, 41, 49, 47, 44, 50, 52, 49, 49, 49, 45, 54, 54, 54, 45, 47, 49, 52, 51, 45, 49, 47, 54, 53, 55, 46, 53, 50, 51, 51, 53, 49, 52, 55, 47, 53, 51, 53, 49, 58, 53, 58, 53, 51, 49, 51, 42, 51, 46, 45, 61, 50, 40, 44, 40, 58, 46, 47, 50, 51, 45, 54, 45, 58, 43, 51, 48, 48, 57, 48, 50, 61, 47, 53, 56, 49, 40, 46, 53, 49, 50, 51, 58, 56, 61, 46, 55, 58, 49, 49, 49, 45, 50, 47, 53, 46, 53, 45, 52, 53, 47, 54, 54, 52, 55, 50, 45, 53, 42, 52, 52, 52, 52, 51, 49, 44, 61, 49, 47, 56, 48, 56, 41, 47, 50, 57, 57, 47, 53, 47, 51, 50, 49, 50, 47, 55, 50, 54, 60, 42, 53, 46, 58, 52, 54, 53, 46, 46, 55, 52, 50, 43, 48, 54, 44, 43, 45, 53, 58, 49, 48, 51, 48, 48, 39, 55, 56, 58, 52, 50, 49, 49, 54, 49, 49, 48, 55, 52, 47, 54, 48, 53, 50, 51, 49, 59, 47, 64, 51, 53, 54, 52, 48, 54, 56, 46, 52, 48, 41, 51, 55, 56, 49, 52, 48, 45, 50, 56, 49, 46, 51, 55, 43, 53, 47, 56, 64, 49, 46, 56, 45, 53, 50, 53, 50, 51, 48, 49, 53, 41, 44, 50, 50, 50, 49, 51, 53, 48, 51, 50, 50, 46, 59, 50, 46, 47, 44, 45, 48, 44, 49, 52, 52, 50, 51, 51, 49, 52, 49, 56, 51, 56, 50, 46, 43, 53, 61, 53, 44, 52, 57, 51, 56, 50, 53, 54, 46, 57, 52, 64, 52, 55, 54, 52, 52, 49, 53, 51, 57, 48, 50, 50, 52, 41, 51, 52, 50, 54, 51, 47, 51, 40, 54, 54, 52, 51, 54, 51, 46, 48, 53, 48, 49, 46, 47, 61, 49, 46, 39, 51, 48, 49, 53, 42, 52, 50, 55, 49, 46, 46, 51, 51, 56, 51, 52, 52, 54, 44, 46, 42, 36, 48, 50, 50, 50, 43, 50, 47, 55, 49, 50, 47, 55, 49, 53, 37, 50, 46, 53, 47, 52, 56, 46, 58, 52, 51, 51, 55, 51, 46, 50, 54, 53, 53, 46, 48, 49, 53, 51, 38, 48, 51, 49, 47, 54, 49, 52, 51, 43, 47, 48, 48, 48, 43, 53, 64, 56, 46, 55, 55, 47, 47, 58, 52, 51, 44, 47, 52, 61, 51, 54, 61, 50, 45, 43, 51, 50, 48, 52, 53, 51, 48, 50, 48, 47, 52, 51, 50, 47, 51, 58, 58, 62, 53, 46, 51, 56, 44, 49, 54, 55, 47, 54, 44, 54, 51, 46, 48, 53, 54, 50, 55, 48, 50, 53, 50, 54, 47, 46, 44, 52, 50, 53, 43, 46, 51, 48, 49, 48, 45, 47, 41, 55, 52, 48, 53, 45, 62, 45, 47, 48, 52, 53, 53, 56, 38, 44, 48, 49, 47, 46, 52, 50, 49, 50, 47, 49, 55, 53, 46, 48, 47, 55, 52, 48, 47, 53, 53, 43, 46, 56, 48, 51, 40, 51, 57, 50, 57, 51, 53, 42, 45, 52, 51, 44, 50, 50, 53, 46, 47, 54, 50, 50, 56, 56, 53, 51, 49, 52, 46, 49, 47, 47, 50, 50, 41, 49, 52, 42, 47, 54, 47, 47, 51, 56, 47, 49, 51, 47, 48, 56, 44, 60, 41, 50, 52, 44, 48, 53, 48, 54, 54, 52, 44, 50, 46, 55, 52, 50, 51, 44, 56, 49, 50, 53, 52, 54, 52, 51, 54, 45, 48, 47, 45, 53, 45, 52, 52, 49, 52, 49, 45, 51, 44, 51, 49, 54, 47, 49, 53, 49, 46, 45, 52, 47, 47, 47, 43, 48, 51, 53, 54, 56, 46, 50, 55, 59, 54, 44, 48, 59, 47, 41, 48, 50, 45, 53, 53, 48, 49, 50, 53, 43, 59, 47, 44, 47, 51, 50, 56, 57, 55, 49, 56, 56, 54, 42, 49, 56, 51, 58, 44, 47, 56, 45, 53, 54, 47, 45, 50, 48, 47, 57, 53, 55, 47, 53, 49, 50, 46, 50, 48, 51, 46, 44, 51, 56, 55, 51, 47, 50, 49, 49, 47, 46, 51, 47, 51, 50, 48, 43, 49, 46, 46, 56, 49, 51, 51, 54, 50, 52, 59, 53, 46, 47, 50, 46, 47, 46, 49, 48, 49, 42, 47, 52, 48, 61, 49, 50, 54, 49, 49, 55, 52, 47, 51, 42, 55, 41, 54, 55, 50, 51, 51, 50, 53, 52, 46, 54, 52, 51, 56, 51, 53, 53, 44, 44, 53, 56, 54, 50, 46, 49, 50, 59, 48, 50, 54, 48, 46, 51, 44, 47, 54, 51, 55, 54, 51, 39, 52, 52, 49, 44, 47, 60, 51, 34, 46, 45, 52, 55, 57, 55, 42, 50, 57, 52, 53, 52, 45, 44, 42, 47, 46, 55, 47, 51, 54, 51, 47, 50, 57, 46, 49, 47, 47, 44, 44, 55, 52, 47, 48, 41, 48, 53, 43, 51, 45, 51, 55, 49, 54, 48, 52, 47, 47, 55, 43, 54, 48, 52, 51, 49, 48, 46, 49, 50, 47, 50, 49, 60, 51, 52, 53, 48, 51, 50, 42, 51, 52, 50, 52, 47, 51, 54, 47, 44, 53, 48, 45, 54, 49, 47, 46, 51, 52, 49, 47, 50, 48, 59, 54, 45, 61, 53, 43, 52, 54, 53, 61, 54, 49, 51, 50, 57, 43, 44, 51, 52, 47, 46, 48, 48, 51, 56, 43, 56, 46, 50, 53, 48, 45, 43, 50, 56, 48, 52, 48, 46, 48, 47, 42, 52, 49, 51, 49, 54, 48, 47, 54, 53, 42, 45, 46, 52, 55, 50, 42, 52, 45, 44, 44, 54, 48, 46, 48, 51, 55, 52, 50, 50, 48, 56, 52, 41, 56, 55, 50, 42, 53, 46, 47, 50, 46, 53, 56, 48, 50, 52, 53, 46, 57, 49, 51, 52, 50, 53, 53, 47, 47, 46, 55, 54, 48, 49, 49, 49, 51, 45, 45, 50, 48, 55, 52, 48, 48, 41, 50, 50, 50, 50, 61, 43, 52, 48, 53, 54, 49, 44, 47, 55, 58, 48, 58, 46, 44, 54, 55, 47, 47, 45, 47, 52, 50, 46, 52, 49, 48, 47, 50, 56, 50, 56, 54, 49, 54, 50, 50, 45, 56, 48, 56, 60, 50, 47, 54, 51, 53, 50, 50, 57, 56, 41, 53, 49, 54, 51, 57, 49, 50, 52, 47, 49, 53, 55, 49, 53, 47, 51, 50, 37, 61, 56, 50, 49, 67, 49, 52, 57, 55, 46, 53, 51, 57, 50, 50, 49, 49, 48, 56, 60, 42, 48, 49, 52, 56, 47, 57, 40, 60, 50, 52, 51, 54, 51, 47, 48, 59, 47, 53, 52, 49, 49, 49, 50, 39, 52, 48, 42, 50, 49, 50, 46, 52, 48, 51, 51, 51, 46, 49, 46, 48, 45, 55, 47, 47, 60, 57, 50, 51, 46, 49, 48, 47, 38, 52, 55, 51, 44, 48, 54, 51, 59, 51, 56, 46, 48, 51, 45, 56, 47, 51, 53, 50, 45, 53, 59, 50, 56, 53, 48, 42, 50, 54, 49, 50, 60, 50, 65, 58, 54, 47, 45, 55, 44, 51, 48, 49, 45, 46, 51, 48, 55, 53, 50, 46, 58, 59, 50, 49, 51, 54, 53, 61, 55, 52, 51, 45, 41, 52, 48, 50, 51, 52, 49, 51, 50, 50, 55, 44, 50, 47, 49, 55, 52, 58, 51, 50, 48, 45, 46, 55, 56, 43, 49, 50, 59, 53, 45, 47, 49, 40, 45, 55, 46, 50, 53, 50, 45, 52, 48, 53, 53, 51, 49, 52, 49, 52, 56, 36, 50, 45, 45, 49, 49, 56, 51, 50, 50, 48, 46, 50, 48, 48, 47, 52, 39, 44, 51, 55, 48, 55, 46, 53, 45, 54, 55, 57, 53, 49, 46, 53, 49, 49, 53, 52, 52, 51, 52, 51, 48, 53, 44, 39, 46, 48, 51, 52, 44, 54, 44, 49, 54, 55, 48, 51, 50, 54, 43, 53, 47, 47, 45, 54, 46, 52, 51, 55, 59, 48, 49, 55, 54, 51, 51, 44, 56, 50, 54, 47, 53, 58, 51, 54, 47, 43, 47, 55, 49, 75, 72, 59, 73, 74, 54, 62, 64, 68, 66, 82, 77, 79, 76, 63, 83, 76, 72, 75, 71, 66, 69, 67, 65, 67, 67, 69, 57, 52, 61, 72, 64, 74, 59, 72, 70, 68, 80, 70, 70, 72, 74, 78, 73, 76, 65, 68, 70, 75, 73, 68, 81, 76, 93, 73, 69, 66, 65, 61, 74, 74, 69, 76, 79, 73, 73, 82, 61, 62, 71, 83, 76, 61, 66, 77, 64, 72, 67, 68, 66, 67, 53, 79, 75, 61, 66, 71, 67, 71, 60, 72, 72, 72, 71, 69, 71, 78, 61, 71, 80, 69, 59, 64, 72, 75, 74, 69, 69, 70, 62, 61, 66, 77, 66, 67, 62, 64, 77, 69, 73, 78, 79, 63, 70, 72, 70, 70, 68, 70, 77, 88, 69, 62, 71, 74, 71, 58, 78, 68, 70, 74, 81, 68, 64, 67, 74, 64, 66, 65, 67, 65, 78, 78, 70, 64, 68, 72, 81, 68, 68, 64, 69, 64, 66, 65, 70, 80, 71, 59, 70, 69, 64, 78, 68, 71, 69, 69, 75, 62, 65, 59, 77, 62, 70, 72, 76, 84, 78, 60, 69, 74, 76, 76, 73, 73, 86, 65, 72, 72, 64, 82, 81, 73, 62, 65, 68, 74, 65, 73, 66, 80, 64, 65, 69, 65, 74, 74, 71, 65, 63, 66, 72, 72, 77, 77, 68, 75, 68, 72, 62, 68, 59, 68, 59, 73, 71, 82, 73, 71, 74, 73, 73, 71, 71, 75, 81, 72, 76, 75, 75, 84, 68, 65, 76, 71, 64, 67, 65, 65, 76, 80, 66, 80, 63, 72, 61, 76, 73, 61, 70, 72, 75, 71, 53, 73, 84, 74, 67, 68, 64, 70, 74, 68, 75, 63, 71, 73, 63, 72, 61, 68, 67, 62, 79, 65, 72, 72, 66, 69, 71, 68, 65, 81, 69, 70, 70, 61, 77, 73, 80, 73, 74, 60, 68, 71, 69, 81, 87, 66, 66, 71, 67, 65, 77, 67, 75, 66, 74, 77, 62, 72, 71, 80, 73, 64, 75, 69, 62, 67, 71, 70, 68, 64, 61, 80, 68, 83, 62, 70, 66, 75, 70, 61, 71, 64, 56, 55, 75, 78, 73, 62, 71, 67, 73, 65, 71, 67, 69, 66, 71, 69, 72, 77, 70, 68, 74, 70, 69, 68, 71, 80, 63, 70, 64, 70, 70, 68, 71, 65, 67, 63, 64, 73, 71, 62, 85, 73, 75, 73, 78, 69, 69, 77, 69, 70, 66, 61, 70, 71, 72, 68, 64, 60, 58, 68, 67, 71, 82, 60, 65, 73, 70, 74, 71, 69, 81, 60, 76, 70, 62, 67, 66, 67, 63, 72, 71, 62, 63, 65, 70, 66, 68, 79, 76, 79, 70, 68, 62, 66, 66, 75, 77, 68, 67, 73, 80, 58, 64, 78, 69, 65, 70, 70, 70, 75, 64, 59, 66, 74, 66, 84, 57, 79, 80, 63, 83, 84, 77, 74, 64, 67, 79, 64, 73, 70, 74, 75, 63, 68, 63, 70, 70, 66, 77, 70, 71, 79, 68, 70, 69, 66, 69, 69, 59, 73, 63, 73, 70, 63, 80, 78, 70, 77, 71, 68, 66, 78, 72, 71, 63, 77, 68, 71, 66, 73, 80, 77, 73, 70, 63, 61, 71, 67, 62, 80, 79, 62, 72, 61, 66, 69, 72, 75, 69, 76, 80, 76, 80, 76, 66, 77, 62, 72, 75, 62, 62, 61, 74, 72, 64, 70, 63, 69, 72, 73, 78, 63, 73, 63, 65, 69, 66, 72, 70, 85, 67, 63, 69, 72, 70, 70, 75, 76, 63, 66, 70, 67, 76, 74, 70, 70, 75, 67, 61, 61, 74, 77, 61, 68, 69, 78, 73, 76, 70, 63, 76, 62, 79, 73, 66, 79, 71, 64, 61, 69, 72, 72, 71, 75, 66, 77, 62, 63, 63, 64, 60, 69, 67, 72, 82, 75, 81, 76, 65, 66, 71, 82, 78, 59, 70, 72, 74, 73, 80, 75, 70, 78, 86, 77, 66, 68, 71, 68, 67, 78, 64, 71, 65, 64, 73, 73, 60, 72, 70, 76, 75, 68, 70, 60, 71, 57, 78, 74, 71, 70, 71, 59, 73, 76, 64, 76, 76, 70, 75, 65, 62, 70, 62, 64, 69, 68, 78, 81, 72, 67, 84, 67, 79, 65, 72, 62, 65, 71, 69, 61, 65, 58, 68, 68, 75, 60, 73, 63, 68, 87, 71, 69, 70, 61, 77, 75, 75, 72, 66, 75, 82, 71, 73, 61, 66, 81, 79, 71, 67, 66, 77, 64, 70, 70, 76, 69, 76, 74, 77, 71, 78, 70, 67, 64, 75, 69, 84, 64, 74, 65, 69, 73, 66, 77, 63, 74, 67, 61, 68, 59, 64, 59, 66, 65, 55, 66, 71, 75, 65, 58, 68, 75, 79, 69, 69, 67, 75, 71, 78, 75, 68, 54, 72, 75, 75, 62, 81, 67, 66, 78, 75, 59, 64, 74, 65, 67, 72, 67, 79, 66, 51, 72, 65, 64, 66, 61, 71, 67, 67, 65, 63, 68, 82, 62, 69, 87, 66, 72, 65, 58, 70, 83, 76, 70, 69, 75, 72, 63, 65, 66, 80, 72, 76, 68, 66, 78, 64, 73, 68, 69, 66, 80, 80, 68, 56, 67, 78, 59, 71, 70, 64, 81, 77, 78, 74, 61, 69, 70, 79, 61, 63, 71, 66, 74, 70, 69, 77, 76, 77, 75, 69, 77, 69, 68, 71, 77, 64, 65, 72, 69, 79, 68, 79, 68, 78, 73, 62, 58, 55, 76, 85, 75, 72, 77, 67, 68, 66, 83, 63, 88, 67, 77, 75, 80, 66, 86, 65, 77, 67, 63, 72, 75, 66, 68, 65, 71, 73, 72, 68, 80, 68, 81, 74, 73, 73, 77, 74, 68, 77, 70, 66, 72, 71, 77, 62, 65, 72, 78, 77, 69, 67, 78, 72, 66, 70, 64, 74, 75, 69, 66, 67, 72, 78, 65, 57, 74, 67, 61, 72, 66, 63, 66, 67, 72, 75, 65, 66, 75, 69, 62, 68, 61, 67, 67, 77, 70, 71, 65, 66, 66, 77, 68, 67, 74, 68, 70, 78, 60, 78, 69, 75, 73, 66, 76, 69, 66, 67, 65, 72, 74, 61, 65, 73, 77, 80, 75, 72, 66, 66, 70, 73, 77, 72, 67, 75, 71, 75, 68, 70, 77, 72, 64, 69, 66, 71, 61, 81, 74, 63, 65, 71, 77, 56, 81, 66, 64, 79, 72, 74, 64, 66, 74, 75, 71, 77, 74, 63, 86, 78, 85, 74, 68, 85, 63, 73, 61, 66, 68, 80, 69, 74, 65, 67, 57, 71, 70, 71, 68, 61, 62, 58, 65, 79, 77, 68, 64, 66, 72, 67, 74, 59, 73, 76, 69, 65, 74, 72, 66, 79, 79, 67, 72, 67, 68, 70, 60, 78, 78, 73, 74, 70, 50, 75, 68, 83, 71, 77, 66, 77, 80, 81, 69, 65, 70, 65, 63, 67, 65, 83, 69, 54, 74, 73, 71, 66, 70, 82, 54, 66, 69, 66, 62, 69, 67, 71, 61, 73, 72, 71, 65, 78, 70, 77, 70, 68, 66, 69, 71, 60, 68, 73, 69, 74, 67, 69, 73, 72, 73, 69, 65, 55, 74, 71, 67, 73, 62, 74, 75, 65, 70, 72, 69, 64, 70, 72, 73, 75, 70, 68, 66, 71, 81, 68, 80, 78, 72, 66, 65, 66, 64, 63, 74, 72, 67, 76, 76, 69, 61, 66, 75, 69, 66, 75, 58, 78, 67, 68, 73, 70, 77, 70, 80, 80, 73, 70, 66, 71, 72, 73, 63, 75, 66, 65, 87, 82, 75, 60, 65, 71, 67, 78, 77, 74, 65, 74, 68, 74, 76, 72, 70, 73, 62, 63, 71, 70, 70, 68, 73, 80, 78, 59, 69, 73, 63, 73, 61, 67, 81, 76, 69, 68, 58, 66, 63, 68, 59, 68, 65, 79, 72, 64, 68, 75, 67, 68, 67, 71, 70, 73, 72, 74, 67, 66, 74, 59, 72, 60, 72, 74, 81, 64, 72, 79, 67, 78]
sex = ['F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'F', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M', 'M']

Let's split the data into 70/30 sets.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(weight, sex, test_size=0.3, random_state=0)

Now let's plot the data for X_train and X_test colored by gender:

plt.hist([x for (x,y) in list(zip(X_train,y_train)) if y=='M'], color='blue', alpha=0.5, bins=15)
plt.hist([x for (x,y) in list(zip(X_train,y_train)) if y=='F'], color='red', alpha=0.5, bins=15)
plt.show()

Now its time for the histogram for the test set:

plt.hist([x for (x,y) in list(zip(X_test,y_test)) if y=='M'], color='blue', alpha=0.5, bins=15)
plt.hist([x for (x,y) in list(zip(X_test,y_test)) if y=='F'], color='red', alpha=0.5, bins=15)
plt.show()

As you can observe, it's a multimodal distribution. This is something very typical in many scenarios, such as when there are patients with certain diseases (cancer/no cancer) or maybe in the banking industry, to label if a customer will be labeled as declined for a loan (declined/approved).

We can use a Gaussian Mixture, a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. SKlearn already has this implemented, and we only need to tell the Gaussian mixture class how many labels or classes exist in the dataset. In this case, we know there are only two genders (male and female), so we will set the parameter n_components=2. Sklearn will automatically identify both distributions and estimate each distribution's means. The trick is simple: if new data comes for a prediction, the Gaussian Mixture Model will predict with a value Xi is close to one mean or two the other and will assign the label accordingly:

import numpy as np
from sklearn.mixture import GaussianMixture

X = np.array(X_train).reshape(np.array(X_train).shape[0], 1)
gm = GaussianMixture(n_components=2, random_state=0).fit(X)

print(gm.means_)

This will print gm.means_[0] = 70.18 and gm.means_[1] = 50.12. We can plot the X_train and y_train distribution again with some vertical lines to check if this works correctly.

plt.hist([x for (x,y) in list(zip(X_train,y_train)) if y=='M'], color='blue', alpha=0.5, bins=15)
plt.hist([x for (x,y) in list(zip(X_train,y_train)) if y=='F'], color='red', alpha=0.5, bins=15)

plt.axvline(gm.means_[0], color='k', linestyle='dashed', linewidth=1)
plt.axvline(gm.means_[1], color='k', linestyle='dashed', linewidth=1)
plt.show()

The gm model was trained on the train set data, and the two means were added to the histogram. All looks good. Now, the only thing left is to predict the test set and get the accuracy. The following snippet will do the trick:

y_pred = gm.predict(np.array(X_test).reshape(np.array(X_test).shape[0], 1))
y_pred = ['M' if x == 0 else 'F' for x in y_pred]

from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, y_pred)

print("Accuracy:", accuracy)

Accuracy: 0.97

Summary

The Gaussian Mixture is a tool for segregating data with multiple distributions. In this case, two normal distributions represented males and females, which SKlearn effortlessly separated. In some cases, the data might be a little more complex with more dimensions; this might require another type of Mixture Model, such as a Variational Bayesian Mixture Model that can help cluster across multiple classes. You can check an example of such a scenario with sklearn mixtures.

Acknowledgments

This code was inspired by an old post from Luis Perez, PhD, about when we worked at the university building AI content for students.

That sounds good, right? well, let's test it out by building the embeddings out from the description of some superheroes I got in a csv file.

Pre-requisites

1 - Ok to build this, you will need to get an OpenAI key. you can get on for free at https://openai.com/api/. Just sign-in, click on Personal and then View API Keys and create a new one. Make sure to copy-paste it in a secure place.

2 - Download the csv file from My Github Datasets Repo

3 - Open a jupyter notebook and install the following libraries

• openai

• pandas

• numpy

Coding Time!

Alright, enough chat, let's get into the business here. The first step is to load the required libraries.

Load Libraries

import pandas as pd
import numpy as np
import openai

from openai.embeddings_utils import get_embedding, cosine_similarity

openai.api_key = 'put-your-api-key-here'

In this snippet, we are loading all the utils we need as well as setting up the API Key.

Create the "Combined" Field for Search

We load the supers.csv into a pandas dataframe. We will search over the "history_text" and "powers_text" descriptions. To make this easier, we will merge both columns into a new one called "Combined". The following code achieves exactly this. We will be using only the first 50 super heroes for our testing.

df = pd.read_csv("data/supers.csv")
df = df.fillna('')

# lets use the first 50 superheros. (because of the rate limit)
df = df.head(50)

df["combined"] = df[["history_text", "powers_text"]].apply(" ".join, axis=1)

Create the Embeddings

The "Combined" column plays an important role. We need to convert this text into an Embedding. The embedding is a vector representation of the tokens in the text according to the "text-embedding-ada-002" model. This means we will create a new pandas column with the name "Embedding" to store this representation. The code below loops over the data frame and creates an embedding out of each combined text row. Then each embedding is converted as numpy.array and added to the dataframe. Note that the model embeddings are created by making a remote call to OpenAI through the get_embedding(...) function, which has a rate limit of 60 calls/second unless you upgrade your subscription.

for item in df["combined"]:
    embedding = get_embedding(item, engine="text-embedding-ada-002")
    embeddings.append(np.array(embedding))

df["embedding"] = embeddings

Now we have a new column called "embeddings" that will hold the model representation for each row. the only thing left to do here is to build a Search function that can calculate the cosine similarity between the search query and the existent embeddings. The most similar results should be the ones returned first.

Search Function

This is where the magic happens. The search function converts a query such as "helps batman" to a vector embedding and then compares it to the rest of the dataset. The results are sorted by the similarity column calculated from the cosine function. Take a look.

def search(df, description, n=3):
    text_embedding = get_embedding(
        description,
        engine="text-embedding-ada-002"
    )
    df["similarity"] = df.embedding.apply(lambda x: cosine_similarity(x, text_embedding))

    results = (
        df.sort_values("similarity", ascending=False)
        .head(n)
    )

    return results

Searching

Here comes the fun and final part. We now just look for information and we let the search(...) function make the magic. The n=3 means to show the top 3 results.

results = search(df, "helps batman", n=3)

Now we loop over the results with this code to see what came back.

for index, row in results.iterrows():
    print("---------------------- ")
    print(row["real_name"], '[Similarity Score:', round(row["similarity"]*100),"%]")
    print(" ")
    print("History:", row["history_text"][:200])
    print(" ")
    print("Powers:", row["powers_text"][:200])

These are the results from the search:

---------------------- 
Alfred Pennyworth [Similarity Score: 84 %]

History: Alfred CranePennyworth is the butler, mentor, surrogate father, and close friend of Bruce Wayne. Alfred has served the Wayne family since before Bruce was born. After Bruce was left orphaned from the 

Powers: Alfred is exceptionally intelligent, which extends to his considerable investigative, analytical, communications, computer operating, engineering, and medical skills, as well as those in ordinary hous
---------------------- 
Aaron Cash [Similarity Score: 82 %]

History: Aaron Cash is the head of security at Arkham Asylum. He has a hook for a hand after his real hand was eaten by Killer Croc.

Powers: 
---------------------- 
Bruce Wayne [Similarity Score: 82 %]

History: He was one of the many prisoners of Indian Hill to be transferred to another facility upstate on the orders of The Court. However, Fish Mooney hijacks the bus and drives it into Gotham City, where the

Powers:

As you can see, with a Similarity Score of 84%, Alfred Pennyworth, was the first result. For those who don't know, Alfred is Batman's butler.

Cheers!

Greetings, friends; in this post, I will explain my solution for a variation of the Stable Matching problem, particularly for the case of couple matching. The idea is that given a set of Men (M) with, let's say, ten dudes and a set of Women (F) with 100 ladies, we will use a distance-based method (Euclidean Distance) to match every woman to every dude so that every dude has the same amount of ladies assigned to each one after a quota has been reached.

This exercise aims to replicate a little of Tinder's or Match's way of assigning you a person you might want to meet. We will make some assumptions. The first one is that there is at least one female for every male. The other one is that there might be more women in the data than men, so this will overflow. The algorithm in this post will assign many ladies to a single dude based on their matching score. Once a male has reached the assignment quota, the best matches will go to another male. The last assumption here is the direction of the assignments; we are assigning many females to a single male. Feel free to revert this if your heart feels this is wrong.

Ok, let's start. The first thing to do here is to create random data about the parties of interest. I have created a pandas data frame for each set with the following attributes: Age, Education, State, Children, and Income. You can add more if you want.

The following code will create 10 random Males and 100 random females. Each variable is encoded. This means that it uses a numeric representation of a particular category.

import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from random import randrange
import math

Create the Fake DataSet

# 1: age 18 - 25
# 2: age 26 - 30
# 3: age 31 - 35
# 4: age 36 - 40
# 5: age 41 - 45
# 6: age 46 - 50
# 7: age 50+

# 1: edu none
# 2: edu elementary
# 3: edu highschool
# 4: edu college

# 1: state AL
# 2: state CO
# 3: state CT
# 4: state GA

# 1: income 20-50k
# 2: income 50-100k
# 3: income 100k-150k
# 4: income 150-200k
# 5: income 200k+

males = pd.DataFrame()

for i in range(0,10):
    males = males.append({'Name': 'Male_' + str(i+1) , 'Age' : randrange(7)+1, 'Education' : randrange(4)+1, 'State' : randrange(4)+1, 'Children': randrange(3)+1, 'income' : randrange(5)+1}, ignore_index=True)

male_names = males.Name
males = males.drop(['Name'], axis = 1)

females = pd.DataFrame()
for i in range(0,100):
    females = females.append({'Name': 'Female_' + str(i+1) , 'Age' : randrange(7)+1, 'Education' : randrange(4)+1, 'State' : randrange(4)+1, 'Children': randrange(3)+1, 'income' : randrange(5)+1}, ignore_index=True)

female_names = females.Name
females = females.drop(['Name'], axis = 1)

The code above will generate the following grid of data. You can observe that each attribute is represented by a number from the encoding defined as comments. The same attributes are used for both Males and Females, as this is the way, later, to compare how similar a couple is. This is the list of the ten males with their features.

The Main Algorithm

Part 1 - Create a Grid with Euclidean Distances

The first part of our journey to find a match is to compare each male to each female feature array and estimate how "similar" they are with a distance-based method. In our case, I chose to use the Euclidean Distance. If you want, select another technique like Cosine, Minkowski, or Chebychev.

# Estimation of Euclidean Distance between Males and Females...
male_matchings = []
for index, m_row in males.iterrows():
    male = m_row.to_numpy()
    matchings = []
    for index, f_row in females.iterrows():
        female = f_row.to_numpy()
        similarity = np.linalg.norm(male-female) # euclidean distace
        matchings.append(similarity)
    male_matchings.append(matchings)

The code snippet shows an O(N^2) nested-loop approach to compare each male with each woman with the "np.linalg.norm(male-female)" method from Numpy, which tells the Euclidean Distance between two arrays. The closer they are, the closer to zero the value is. The male_matchings is a list of lists that contains such distances.

Part 2 - Sort Similarity Arrays

Each row of the male_matchings list represents a man, and the sublist attached represents the distance obtained for each woman. Let's sort each sublist so that the best match is on top. The following code obtains each sublist, sorts it, and appends it to the matches_values list of lists. Now the matches_values contain the sorted representation of the male_matchings list of lists.

# re-sorting of matchings & matrix of values and names
matches = [] # sorted collections
matches_values = []
for fem in male_matchings:

    x = {}
    for i in range(0, len(fem)):
        x[female_names[i]] = fem[i]

    sorted_x = sorted(x.items(), key=lambda kv: kv[1])
    matches.append([k[0] for k in sorted_x])
    matches_values.append([k[1] for k in sorted_x])

Part 3 - Assignments based on Quota

We did some pre-work to start making some assignments. We estimated the euclidean distance between each pair and sorted them so that the most similar other was on top of their stack. Now, we must start assigning one to another but with certain conditions. Here is the algorithm:

While there are Females available {

1. We will assign one lady to one man at a time. This means that "Anna" will be assigned to "Johnny," assuming "Anna" is his first option.

2. Once a lady has been assigned, her name is added to the taken_females array. This means that even if another dude has a good match with Anna, she is no longer available.

3. After Johnny gets a lady assigned, we must move down to the second male in the list and assign their first match if she is not assigned. We will continue like this until all males can get given. If the first lady in the list is unavailable, then the male gets nothing until the second iteration.

4. Quota validation: every male can only be matched with math.ceil((1/len(male_names)) * len(female_names)). This means that if there are 100 women and 10 men, only 10 ladies can be assigned per male. If the quota is reached for that dude, he will be skipped and will not get any more assignments.

}

The following code shows how this is done:

# main algorithm.   
assigments = pd.DataFrame()
taken_females = []
min_score = 0

quota = math.ceil((1/len(male_names)) * len(female_names))
quotas = {}
for name in male_names:
    quotas[name] = 0
print("Max Quota per Male", quota)

j_max = len(matches[0]) # width of matrix (number of females)
i_max = len(matches) # height of matrix (number of males)

for j in range(0,j_max):

    #print("Iteration:", j+1)

    current = pd.DataFrame({
        'males' : male_names,
        'females' : np.array(matches)[:,j].tolist(),
        'scores' : np.array(matches_values)[:,j].tolist()
    }).sort_values(by='scores', ascending=True)

    # MAKE ASSIGMENTS based on best score for each column.
    for index, row in current.iterrows():

        female_name = row['females']
        male_name = row['males']
        score = row['scores']

        # check quota before assigment...
        if quotas[male_name] >= quota:
            continue

        # if the female is taken, then continue con next male...
        if female_name in taken_females:
            continue
        else:
            assigments = assigments.append({'Male': male_name , 'Female' : female_name}, ignore_index=True)
            #print(" --- ",male_name, "assigned to", female_name)
            taken_females.append(female_name)
            min_score += score
            quotas[male_name] += 1

    # convergence, all females taken
    if len(taken_females) == len(female_names):
        print("All Females Assigned - Algorithm Converged at", round(min_score,2))
        break

The code above shows that each male gets ten ladies each. This is to emulate the app functionality where you get a match, and if you don't like it, then you swipe left and continue with the next one.

The following image shows the assigments.groupby(["Male"]).sum() command to show which females were assigned to each male.

Additional Comments

The quota concept was introduced to allow all males to get assigned. If the quota validation is removed from the code, then you can guarantee the best matches will happen, but you might leave one bro without a lady, and here, we don't do that.

A content-based recommender is a type of system that makes recommendations to users based on their preferences and considers product attributes. This sounds very much like the Amazon "if you liked that, maybe you might also like this..." type of recommendation.

The idea is to recommend something the user has never rated before. When I say "rated" means it probably hasn't visited, rated, or clicked it before.

In a real setting, you can change rates for visited, clicked, or watched. You might also find some other metric to describe the interest of the user in a certain product.

The example below is a hello-world type of example where we are going to recommend movies to users. We will only have 4 users and 5 movies:'Star Wars', 'The Dark Knight', 'Shrek', 'The Incredibles', 'Bleu', and 'Memento'. Each movie is described by its genre. however we will use one-hot encoding for the genres: 'Action', 'Sci-Fi', 'Comedy', 'Cartoon', 'Drama'.

Let's start coding. The first thing to do is to represent User ratings and Movie features as tensors (matrices).

Let's start by loading Tensorflow and Numpy.

import numpy as np
import tensorflow as tf

Load Data

We have the following data from 4 users that rated some movies. We also have the metadata from 6 movies described by the genre. Let's keep some arrays with the user names and movie names for later use.

users = ['Juan', 'Daniel',  'Ana', 'Christian']

movies = [
    'Star Wars', 'The Dark Knight', 'Shrek',
    'The Incredibles', 'Bleu', 'Memento'
]

features = ['Action', 'Sci-Fi', 'Comedy', 'Cartoon', 'Drama']

num_users = len(users)
num_movies = len(movies)
num_feats = len(features)
num_recommendations = 2

The following table shows the preferences shown by users for some of the movies. There are blank spaces as not all the users have watched all the movies.

The users_movies tensor contains the information from the table. Let's load this information manually.

users_movies = tf.constant([
                [4,  6,  8,  0, 0, 0],
                [0,  0, 10,  0, 8, 3],
                [0,  6,  0,  0, 3, 7],
                [10, 9,  0,  5, 0, 2]], dtype=tf.float32)

Let's do the same thing with the movies_feats. The following tensor will contain the genre metadata for each movie as seen in the table above.

movies_feats = tf.constant([
                [1, 1, 0, 0, 1],
                [1, 1, 0, 0, 0],
                [0, 0, 1, 1, 0],
                [1, 0, 1, 1, 0],
                [0, 0, 0, 0, 1],
                [1, 0, 0, 0, 1]], dtype=tf.float32)

User Embeddings

Now that we have those tensors with user and movie data, we need to create the user embeddings. The user embeddings show the relationship between each user and the movies by performing matrix multiplication. We can easily achieve this with the tf.matmul (yes, for matrix multiplication) in TensorFlow.

users_feats = tf.matmul(users_movies, movies_feats)
users_feats

This will print the following tensor:

As we can see, this is basically telling us how important is each feature for each user. We must normalize each row so that each value adds to 1. The following code normalizes each row:

users_feats = users_feats / tf.reduce_sum(users_feats, axis=1, keepdims=True)
users_feats

The user_feats tensor now has normalized values for each user (row) and for each feature (column).

Ranking Features for Each User

The user_feats tell us how important is each feature for each user. Let's print this so we can understand the preferences of each user by the calculated weights.

The top_users_features is a tensor that returns the sorted indices of the most important attributes for each row. We are basically sorting each row and returning the index for each attribute. We will later loop and print the name of each feature name.

We find the top-k values of each row with the "tf.nn.top_k" function form tensorflow.

top_users_features = tf.nn.top_k(users_feats, num_feats)[1]
top_users_features

The tensor above shows the indices of the features that are more relevant for each user. The following code prints, for every user, the name of what seems to be more relevant to them:

for i in range(num_users):
    feature_names = [features[int(index)] for index in top_users_features[i]]
    print('{}: {}'.format(users[i], feature_names))

• Juan: ['Action', 'Sci-Fi', 'Comedy', 'Cartoon', 'Drama']
• Daniel: ['Drama', 'Comedy', 'Cartoon', 'Action', 'Sci-Fi']
• Ana: ['Action', 'Drama', 'Sci-Fi', 'Comedy', 'Cartoon']
• Christian: ['Action', 'Sci-Fi', 'Drama', 'Comedy', 'Cartoon']

So Juan and Christian prefer Action and Sci-Fi, and Daniel (and Ana) prefer Drama and Comedy. Please note that they all have different preferences and in different order as defined in the users_feats tensor.

Find Similarity Between Users and Movies

Now, we need to calculate the similarity between the user ratings and the movie features (already calculated in the users_feats tensor). The idea is to create the projected ratings for each movie and store them in a new tensor called users_ratings.

users_ratings =  [tf.map_fn(lambda x: tf.tensordot(users_feats[i], x, axes = 1), 
                           tf.cast(movies_feats, tf.float32))
                  for i in range(num_users)]

In this code, we are using dot-product as a similarity measure between the tensors to calculate the projected ratings. This tensor will now serve as the prediction for the entire dataset. The higher the dot-product, the more we will recommend the movie. The only problem with this is that we don't want to recommend movies the user has already seen, so we will remove the movie the user has already seen by applying a mask to the users_ratings tensor.

The first thing we will do is to create the tensor users_unseen_movies which will have a True if the movie has not been rated by the user and False otherwise.

users_unseen_movies = tf.equal(users_movies, tf.zeros_like(users_movies))
users_unseen_movies

The ignore_matrix is just a tensor the same size as the users_movies filled with zeros. This will serve as a base tensor to fill with only the probabilities of the unseen movies.

ignore_matrix = tf.zeros_like(tf.cast(users_movies, tf.float32))
ignore_matrix

Now, the magic is created with the tf.where function from tensorflow, which is able to apply the mask to the users_ratings so that only the unseen movies are kept in the tensor.

users_ratings_new = tf.where(
    users_unseen_movies,
    users_ratings,
    ignore_matrix)

users_ratings_new

As you can observe, now only the unsee movie probabilities are shown. These will be our movie recommendations!

Recommendation: Top 2 movies (from the users_ratings_new tensor)

We can now calculate all of the users to 2 movies with the tf.nn.top_k function. The following code will return the top-k indices for each row of the users_ratings_new matrix. Remember that the users_ratings_new rows represent the users and the columns the movies. The following code obtains the tensor with the top movies for each user:

top_movies = tf.nn.top_k(users_ratings_new, num_recommendations)[1]
top_movies

As we did previously, we can print the name of the user and the top-2 movies that are recommended for the user that it has never seen before!

for i in range(num_users):
    movie_names = [movies[index] for index in top_movies[i]]
    print('{}: {}'.format(users[i], movie_names))

Acknowledgments

This post has been a reproduction of the code provided by the Google Cloud Platform Data Analyst Recommender Systems program.

When we research something, we look to gain knowledge and answers about things (if any). The idea is to follow a process (the scientific method) that will lead us to an outcome, but that process might be affected by the desired results and the nature of the problem itself.

While there is sufficient documentation, books, and articles about research methods for some well-depicted problems, these often point to research in medicine, psychology, economics, and natural and political sciences (among others). During my doctoral studies, I found that the books I had to review as part of the program had a strong focus on psychology. I found it challenging as I was trying to apply research on information systems, particularly for machine learning, which is a quantitative discipline. Although the books in the program covered practically everything in terms of methodologies, they still let me spin a bit on what to use if I want to create research on how to build a self-driving car or train a convolutional neural network to identify Covid in X-rays.

This post reviews the Exploratory Research previously used in machine learning research to understand how problems are framed and solved. I will also briefly introduce the method I used for my dissertation at the end of this post for sign language recognition.

Exploratory Research

Exploratory research looks to investigate questions that have not been previously studied in depth or are different from other problems in literature. For example, research titles such as "Generative adversarial nets" (Goodfellow, 2014) sound like a candidate topic to be assessed as exploratory. This ground-breaking research reveals how to train generative models using an adversarial process by training two neural networks simultaneously. If we look at the 2014 paper, it goes straight to the action, with virtually no clarification on methodology, research techniques, or anything in this space. This type of research does not mean there is no research question and methods. Still, the way it was written shows only the novelty, not the failed experiments and other original theories for the adversarial GANS apart from what has been included in the literature review.

Tom Dietterich (2019) from Oregon State University recommends the following process to drive the exploratory research successfully.

Exploratory research -> Initial Solutions -> Refinement & Evaluation -> Competing Solutions & Comparative Evaluation -> Mapping The Solution to Space -> Engineering & Technology Transfer.

This process states an "initial solution" to a problem that can be refined, reduced, and evaluated to be compared to other solutions. This will lead the research to determine if the outcome is important enough for publication or if it needs additional refinement, or should it be stashed.

Initial Solutions

The idea here is to provide the kick-start solution to the problem. It does not have to be pretty, but it will serve as the base model for improvement. This stage is essential as this helps to define a precedent for a base solution to a complex problem. A paper might be created at this stage even if the solution does not generalize well. For example, Bayesian networks (Pearl 1985) described simple message passing for tree-structured networks.

Nothing stimulates good research like a bad paper about an interesting problem - Dietterich

Refinement and Evaluation

This is the process where the initial solution is evaluated for improvement. The refinement process can affect the initial solution by proposing new metrics, algorithms, hyper parameters, optimization techniques, or more data. The idea is to see if this could be done better even if the new solution takes a 180-degree turn.

Competing Solutions & Comparative Evaluation

In the previous step, the model or algorithm is compared against itself. We are looking to understand how this is compared against other methods or techniques available for similar purposes in this phase.

It is now always easy to perform the comparative evaluation when the research is proposing something completely new, as in the case of adversarial networks. But the analysis can also be put in perspective against other research efforts that look to solve the same problem, but not in the same way.

After Goodfellow's paper on adversarial gans, now it's easier to do this as many researchers are looking for progress, and improvements against each other can be used for benchmarks.

Mappings Solutions to Space

This looks to identify the design space for a particular problem. What are the bounds, critical design decisions, and how is the algorithm compared to others? For example, the concept of learning and the training time changes drastically between KNN and logistic regression on foundational machine learning algorithms. The training time becomes a problem for KNN as the dataset size increases. But why? How does this affect the usage of the algorithm for particular issues? How can the issue be fixed? These are questions to revisit in this stage as time and space complexity are under evaluation.

Engineering & Technology Transfer

At least in machine learning research, applied research is recommended, and a proof-by-construction will help demonstrate that the investigation is sound. Many research papers test over well-known datasets such as ImageNet, Fake News Detection Dataset, Boston Housing, Atari RL, or MNIST. Study replicability is essential as we often read articles to solve other problems and require code or parts of the approach used to solve the research problem. Writing a paper with publicly accessible data and a code-repository is one of the best things we can do for the scientific community that uses ML for their research efforts.

Design Science

Design science research focuses on the development and performance of (designed) artifacts with the explicit intention of improving the functional performance of the artifact. Design science research is typically applied to categories of artifacts, including algorithms, human/computer interfaces, design methodologies (including process models), and languages. Its application is most notable in the Engineering and Computer Science disciplines, though it is not restricted to these and can be found in many disciplines and fields (Vaishnavi et al., 2019).

Design science is a research methodology that focuses on an artifact and looks for improvement iteratively. So, a machine learning model can be seen as an artifact, subject to the seven guidelines for the research:

• Design as an artifact: Design-science research must produce a viable artifact in the form of a construct, a model, a method, or an instantiation.
• Problem relevance: The objective of design-science research is to develop technology-based solutions to important and relevant business problems.
• Design evaluation: The utility, quality, and efficacy of a design artifact must be rigorously demonstrated via well-executed evaluation methods.
• Research contributions: Effective design-science research must provide clear and verifiable contributions in the areas of the design artifact, design foundations, and/or design methodologies.
• Research rigor: Design-science research relies upon the application of rigorous methods in both the construction and evaluation of the design artifact.
• Design as a search process: The search for an effective artifact requires utilizing available means to reach desired ends while satisfying laws in the problem environment.
• Communication of research: Design-science research must be presented effectively both to technology-oriented as well as management-oriented audiences.

Dietterich's exploratory research process can be merged with the Design Science methodology (they are highly compatible but not the same). Design science puts mathematical rigor and artifact validation as a key to improving the solution (research results), methods, constructs, and other design theories to construct new knowledge. The idea is that the exploratory analysis is performed in iterations. We can evaluate the effect of a particular change in the model or methods used on each cycle.

Design Science for Sign Language Recognition

I used design science for my dissertation "Video-Based Costa Rican Sign Language Recognition for Emergency Services" which proposes the process required to transform a video into text when a person is communicating in sign language (LESCO).

The entire project was envisioned as a collection of artifacts: from the raw video data, the algorithms to transform the data, to the machine learning models used to classify each sign into its textual meaning (label).

Everything is measurable, therefore it can be improved.

The valuable thing about design science is that we can plan the experiment and collect the information on each iteration. After every cycle, we observe what changed and rerun the experiment to see if there are any improvements (pretty much trial and error, which in synthesis is at the core of the experimental research paradigm). This can take us to a path for improvement where we are storytelling how we got into the eureka moment.

References

Dietterich, T. (2019). Research Methods in Machine Learning. 46.

Hevner, A. R.; March, S. T.; Park, J. & Ram, S. Design Science in Information Systems Research. MIS Quarterly, 2004, 28, 75-106. URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.1725&rep=rep1&type=pdf

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (NIPS'14). MIT Press, Cambridge, MA, USA, 2672–2680.

Pearl, J. (1985) A model of self-activated memory for evidential reasoning, in Proceedings of the 7th Conference of the Cognitive Science Society, University of California, Irvine, CA, pp. 329–334.

Vaishnavi, V., Kuechler, W., and Petter, S. (2004/19). "Design Science Research in Information Systems" January 20, 2004; last updated June 30, 2019. URL: http://desrist.org/design-research-in-information-systems

One of the challenges we face while designing machine learning solutions is picking the right prediction features. Deciding which elements are essential and which ones genuinely contribute to model performance becomes complicated as the number of features increases.

The infinite monkey theorem states that a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will almost surely type any given text, such as the complete works of William Shakespeare.

As absurd as the theorem sounds (which has been proved btw), has an important implication in machine learning. When there are too many features, there is the probability that the model can predict the response variable by pure luck, just like the monkey typing random stuff, because some variables result in being perfect predictors.

In statistics, a spurious correlation (or spuriousness) refers to a connection between two variables that appears to be causal but is not. With spurious correlation, any observed dependencies between variables are merely due to chance or related to some unseen confounder (Kenton, 2021).

Spuriousness then implicates the existence of variables in machine learning models that might be nearly-perfect predictors but by pure luck. To understand better the predictive capabilities of spurious relationships, let's look at the following example from Tyler Vigen:

Let's seriously consider the relationship from the chart above. If this is true, the US government should reduce the spending on science and technology to reduce the suicide rate. But that relationship is casual; it is just a coincidence. And this phenomenon can also happen within your machine learning model. There might be variables that perfectly predict the response (y) but by luck. This situation can also appear unnoticed as the dimensional space increases.

Covariance & Spurious Correlations

Covariance is a measure of the joint variability of two random variables. This is, the changes in the X variable are aligned to changes in the Y variable. The issue with covariance is that the relationship is judged based on the linear relationship (as in the Pearson Correlation Coefficient). Two random variables might be perfectly correlated but with a non-linear relationship. This can happen in neural networks when an image classifier learns to identify a "moose" from images because it realizes that the moose images are always in snowy conditions. The classifier learns about the snow and forgets the moose. And because the dataset images with snow are the only ones with a moose, then the classifier predicts the moose perfectly every time, making us think that the classifier is doing a great job.

If you want to know how to influence neural networks to perform the wrong predictions, please read:

• Unsupervised Adversarial Defense through Tandem Deep Image Priors (https://www.researchgate.net/publication/347798137_Unsupervised_Adversarial_Defense_through_Tandem_Deep_Image_Priors/figures?lo=1)
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (https://arxiv.org/pdf/1610.02136.pdf)
• Adversarial Examples in the Physical World (https://www.taylorfrancis.com/chapters/edit/10.1201/9781351251389-8/adversarial-examples-physical-world-alexey-kurakin-ian-goodfellow-samy-bengio)

What can I do?

• First, today's machine learning solutions are based on correlations, not causations. Many solutions work because of the correlations learned, even if they are full of nonsense. Keep this in mind.
• Looking for the best features is an art more than science. When the model has way too many elements, they seem to contribute to the model's accuracy in the same way. Check for the curse of dimensionality.
• Focus on data first. Make sure you collect good data (moose pictures in many shapes, angles, and seasons) and use cross-validation to validate the model performance. Good data is the critical ingredient for a good model.
• Split the data into train/valid/test sets. The train set is used to make the model learn. The valid set is used to validate the model accuracy with cross validation. The test set is a randomly picked sample before the train and valid set were created. This set is important as it represents real-world data the model has never seen before.
• Using the covariance matrix only helps spot linear relationships. Don't take this for granted.
• Test manually. Collect images or new samples and use them to validate the model. Spurious relationships usually overfit the model. This means that if the classifier is making lucky predictions, it should not work with other external images.
• Always start with a simple architecture and data model. The simpler it is, the easiest to spot the issues in the classifier. Neural networks that are too tall and wide are difficult to debug.
• Don't hack the data to make the classifier work. A penguin detector based on penguin images from cartoon books might be convenient because of the reduced amount of colors. Still, it will certainly not generalize to authentic penguin images from the wild.
• Test several algorithms. There might be a chance that a machine learning algorithm might overfit the data giving us the sensation that the classifier can learn beyond the training set. For example, don't just use decision trees; test random forests and compare the results.

We will start our exploration by building a binary classifier for Cat and Dog pictures. The images were downloaded from the Kaggle Dogs vs Cats Redux Edition competition. There are 25,000 images of dogs and cats we will use to train our convolutional neural network.

If you are wondering how to get PyTorch installed, I used miniconda with the following commands to get the environment started.

# install conda environment with pytorch support
# - conda create -n torch python=3.7
# - conda activate torch
# - conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

Import Libraries

import numpy as np
import pandas as pd
import os
import random
import time

import torch
import torchvision
import torch.nn as nn
import torchvision.datasets as datasets
from torchvision import datasets, transforms
from torch.utils.data import Dataset, DataLoader
import torch.nn.functional as F

from sklearn.model_selection import train_test_split

from PIL import Image
import matplotlib.pyplot as plt

Download Data and Preprocess Images

Once you downloaded the data, put the training images in the data/train/ folder on your local computer. The following code will parse the train folder and will collect the path for each image and will save it into the img_files list:

img_files = os.listdir('data/train/')
img_files = list(filter(lambda x: x != 'train', img_files))
def train_path(p): return f"data/train/{p}"
img_files = list(map(train_path, img_files))

print("total training images", len(img_files))
print("First item", img_files[0])

output:

• total training images 25000
• First item data/train/cat.0.jpg

Before we start transforming our data, let's split the dataset into train-test sets with the following code.

# create train-test split
random.shuffle(img_files)

train = img_files[:20000]
test = img_files[20000:]

print("train size", len(train))
print("test size", len(test))

output:

• train size 20000
• test size 5000

Now, we have to use each path to load the image, convert it to RGB, and also label the image based on the name. If the word "cat" is in the path, then the label will be 0, otherwise 1 for a dog. Additional transformation is needed for the image, so a transform object will be created to resize the image to 244x244 and normalize its values. The following class will receive a Dataset (from torch.utils.data) and will apply the transformation.

# image normalization
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

# preprocessing of images
class CatDogDataset(Dataset):
    def __init__(self, image_paths, transform):
        super().__init__()
        self.paths = image_paths
        self.len = len(self.paths)
        self.transform = transform

    def __len__(self): return self.len

    def __getitem__(self, index): 
        path = self.paths[index]
        image = Image.open(path).convert('RGB')
        image = self.transform(image)
        label = 0 if 'cat' in path else 1
        return (image, label)