About

In the previous post, understanding spectral reflectance, we saw that objects could be differentiated by their surface reflectance. The surface reflectance can be sensed as multi-spectral images from satellites. In the following post, we will examine the Sentinel-2 mission and the resulting data. Afterwards, we will review an example remote sensing dataset, BigEarthNet.

Sentinel-2 mission

Sentinel-2 is an earth-observation mission and consists of two satellites Sentinel-2A and Sentinel-2B. Both of which are operated by the European Space Agency (ESA). The task is to gather multi-spectral data for climate change, agriculture monitoring, and emergency management. The data is published under a free and open data policy, making it valuable for academic purposes.

Fig 1: A Sentinel-2 satellite(Image from Satellite Imaging Corporation)

With both satellites and their large field of view (290km), they can sense most of the earth's land cover every 5 days. The revisit frequency is also called the temporal resolution. The spatial resolution is reported as $XX\,\text{m}$, which refers to the length and height of a pixel. So a resolution of 10m would correspond to a single pixel capturing an area of 10m x 10m, or 100m². An example remote sensing image can be seen in the following figure. The Sentinel-2 satellites have a spatial resolution of 10m (four visible and near-infrared), 20m (six red edge and short-wave infrared), and 60m (three atmospheric correction) bands. [1]

Example remote sensing image with a 30m spatial resolution (Image from GSP Humboldt)

In total, thirteen bands are sensed, ranging from the visible/near-infrared (VNIR) to the short-wave infrared (SWIR) spectrum. Each band has effectively 16-bits per channel, or as it is commonly referred to in the remote image sensing community, a radiometric resolution of 16-bits. The term radiometric resolution highlights the domain in which the images are used but is no different from bits per channel¹.

The following figure shows all thirteen bands grouped by their spatial resolution. ESA introduced Band 8A in the Sentinel-2 mission as band 08 was too contaminated by water vapor and insensitive to other parameters for some applications. But, the original sensor for band 08 remained in the sensory equipment. The narrowness of band 8A should make the results less noisy towards water vapor but still be wide enough for most applications [2].

alt.Chart(sentinel_band_data).mark_rect().encode(
    x=alt.X("start:Q", title="Wavelength in nm"),
    x2="end:Q",
#     color=alt.Color("Band", scale=alt.Scale(scheme="category20")),
    color=alt.Color("Color", scale=None),
    tooltip=[
        alt.Tooltip("Band:O", title="Band"),
        alt.Tooltip("Usage"), 
        alt.Tooltip("Central Wavelength"),
        alt.Tooltip("Spatial Resolution"), 
    ]
).transform_calculate(
    start="datum['Central Wavelength'] - 1/2 * datum['Bandwidth']",
    end="datum['Central Wavelength'] + 1/2 * datum['Bandwidth']"
).properties(
    height=100,
    width=600
).facet(
    row="Spatial Resolution"
)

Everyone can register on scihub.copernicus.eu and search for remote sensing imagery. The images, also called tiles or granules, from the Sentinel-2 mission sense an area of 100km² and are ~600MB in size. [3] The Copernicus program provides two types of data for public usage:

L2A (Level 2A with atmospheric correction)
L1C (Level 1C without atmospheric correction)

Applying atmospheric correction algorithms transform a so-called TOA (Top Of Atmosphere) to a BOA (Bottom Of Atmosphere) image. If one is interested in the surface reflectance values, or more generally on the objects on the ground, the L2A data should be preferred. In the case of missing L2A data, the Sentinel-2 toolbox can be used to generate L2A from L1C images.

a) True-color image

b) False-color composition

Fig 2: Sentinel-2 example image

Fig. 2a shows the visible bands of a randomly selected image with low cloud coverage. Remote sensing images that show the visible bands, like classic RGB images are called true-color images (TCI). To visualize the data from the other bands one can:

Show each band independently as a grayscale image
Map three bands to the classic RGB channels of an image (called false-color image/composite)

Fig. 2b shows a popular false-color composite, using the bands 08, 04, and 03. With the band in the near-infrared spectrum (band 08) as the red channel, the healthy green vegetation will light up in bright red. As the bare soil has a low reflectance in the near-infrared spectrum, it will range from tan to turquoise. With the EO Browser, you can interact with satellite imagery and false-color composites without requiring you to download any images or manually applying transformations on different spectral bands.

Note: I highly recommend playing around with the EO Browser as it is the easiest way to interact with the various bands.

To get most of the high-volume data from remote sensing images, one can employ deep-learning. Deep-learning has become the state-of-the-art solution to complex computer vision applications. Usually, deep-learning models are used on classic RGB images, but they also seem to be promising for these multi-spectral images. To train and test these models, researchers need large, high-quality datasets. The data assembly was not a problem, thanks to the open data policy of the Sentinel-2 imagery. Sumbul et. al [4] were able to assemble and published such a dataset, BigEarthNet.

BigEarthNet

The BigEarthNet archive uses Sentinel-2 tiles that are distributed over 10 countries from Europe ². Only tiles with a cloud cover percentage under 1% containing no missing/faulty pixels were considered. The tiles were then split into smaller non-overlapping patches for further processing and publication. In total, the dataset consists of 590,326 patches, each of which covers a region of 1.200m x 1.200m. Due to the different spatial resolution of the various bands, the patches have different sizes. 120 x 120 pixels for 10m bands, 60 x 60 pixels for 20m bands, and 20 x 20 pixels for 60m bands.

As the archive is based on Sentinel-2 images, the radiometric resolution is 16-bits. The time-frame for the acquisition dates were between June 2017 $-$ June 2018. Due to the winter months generally having higher cloud coverages, the winter season has the fewest samples, as seen in the following chart.

season_data = pd.DataFrame([
    {"Season": "Autumn", "# Images": 154_943},
    {"Season": "Winter", "# Images": 117_156},
    {"Season": "Spring", "# Images": 189_276},
    {"Season": "Summer", "# Images": 128_951},
])

alt.Chart(season_data).mark_bar().encode(
    x="Season",
    y="# Images",
    tooltip=[
        alt.Tooltip("Season"),
        alt.Tooltip("# Images"), 
    ],
).properties(
    width=300
)

The authors identified 70,987 patches that are fully covered by clouds, cloud shadows, and seasonal snow. They provide CSV files to exclude these patches if desired and recommend to do so if neural networks are only trained on the BigEarthNet dataset.

The last point to note is that not all bands are used in this dataset; band 10 is omitted. Band 10 does not include any surface-level information, as it is mainly used to detect cirrus clouds [5]. These clouds form at high altitudes and are transparent or semi-transparent in the optical bands but have a high impact on the original spectral reflectance [6]. For the data preprocessing step, the 10th band is a significant indicator of the data quality but does not hold any information for down-stream processes. To use the correct terminology, band 10 is important for TOA analysis and the conversion from TOA to BOA images. However, the other 60m bands used for atmospheric correction were not removed.

With that said, we have covered all the essential details of the BigEarthNet archive. But, before we move on and use the dataset, let's take a short recap.

Summary

The most important features of the Sentinel-2 earth-observation mission are:

2 satellites (Sentinel-2A/Sentinel-2B)
Temporal resolution of 5 days
Spatial resolution of 10m, 20m, and 30m (depending on the specific band)
Radiometric resolution of 16-bits
Senses 13 bands, from the visible/near-infrared to the short-wave infrared spectrum
Open data policy

Thanks to the open data policy, researchers were able to create BigEarthNet, a large, freely available multi-spectral dataset for deep-learning. The significant properties are:

Provides ~590,000 patches
- Each patch covers a region of 1.200m x 1.200m
Various resolutions defined by Sentinel-2 specs
Does not include band 10, as it does not contain surface-level information
Provides CSV with all uninformative patches (patch with only snow/clouds)

References

[1]European Space Agency, “Sentinel-2 spatial resolution.” 12-Oct-2020 [Online]. Available at: https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi/resolutions/spatial
[2]European Space Agency, “Sentinel-2 Heritage.” 12-Oct-2020 [Online]. Available at: https://sentinel.esa.int/web/sentinel/missions/sentinel-2/heritage
[3]European Space Agency, “Sentinel-2 Data Products.” 12-Oct-2020 [Online]. Available at: https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi/product-types
[4]G. Sumbul, M. Charfuelan, B. Demir, and V. Markl, “Bigearthnet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding,” in IEEE International Geoscience and Remote Sensing Symposium, 2019 [Online]. Available at: http://bigearth.net/static/documents/BigEarthNet_IGARSS_2019.pdf
[5]European Space Agency, “Sentinel-2 cloud masks.” 12-Oct-2020 [Online]. Available at: https://sentinel.esa.int/web/sentinel/technical-guides/sentinel-2-msi/level-1c/cloud-masks
[6]S. Qiu, Z. Zhu, and C. E. Woodcock, “Cirrus clouds that adversely affect Landsat 8 images: What are they and how to detect them?,” Remote Sensing of Environment, no. Q, Sep. 2020 [Online]. Available at: https://www.sciencedirect.com/science/article/pii/S0034425720302546

1. There is some ambiguity of the actual radiometric resolution. The official website writes a radiometric resolution of 12-bits per channel, but the information is out of date. Inspection of images starting around 2016 reveals a resolution of at least 15-bits. But the full 16-bit range is used to encode special values. As the data will always end-up using 2 Bytes for storage (16-bits), it is commonly treated as a 16-bit image. I tried my best to comprehend the data range fully, but there are still some open questions. I am trying to answer these questions and have an open discussion on gis.stackexchange.com. Feel free to join the discussion!↩

2. These 10 countries are: Austria, Belgium, Finland, Ireland, Kosovo, Lithuania, Luxembourg, Portugal, Serbia, Switzerland↩

About

In the Introduction to multi-channel images post, we defined a multi-spectral image as an image that combines various electromagnetic bands. As an example, we looked at the combination of a classic RGB image with the data from a long-infrared image. But, before we jump into approaches to visualize and process multi-spectral images, we should deepen our knowledge of spectral bands.

Spectral reflectance

In the post mentioned above, we saw that our LCD monitors radiate light in the visible electromagnetic (EM) spectrum, which we perceive as colors. Instead of an object emitting light from specific spectral bands, a surface reflects radiation. In this reflection process, some parts of the EM spectrum are also absorbed by the material. We then observe and process the reflected light.

LCD emitting light

Flowers reflecting light

Emitting vs. reflecting objects

This spectral reflectance is why we perceive a healthy leaf as green, even though it is not emitting light like our monitor. It merely reflects the electromagnetic radiation of the sun. Please take a moment and try to guess which part of the visible light a green leaf reflects and which it absorbs.

We perceive the primary color of healthy leaves as green. With our knowledge of spectral reflectance, we can deduce that the main band being reflected is green, while the blue and red bands are mainly absorbed. To verify our educated guess, we can look at published reflectance curves. These reflectance curves tell us what parts of EM radiation are absorbed and reflected. The USGS Spectral Library is free of charge and available for anyone on the USGS website. We can search for spectrums of objects with their fantastic query tool without downloading the complete 5GB dataset. As an example leaf, I chose an Aspen leaf with the following spectrum title "Aspen Aspen-1 green-top ASDFRa AREF". Now is the chance to use some pandas to read the spectrum and visualize the result with your favorite plotting library.

import numpy as np
import pandas as pd
import altair as alt

# data from https://en.wikipedia.org/wiki/Visible_spectrum#Spectral_colors
# with small modification for red, explained in text
visible_light = [
    {"color": "violet", "start": .38, "end": .45},
    {"color": "blue", "start": .45, "end": .485},
    {"color": "cyan", "start": .485, "end": .5},
    {"color": "green", "start": .5, "end": .565},
    {"color": "yellow", "start": .565, "end": .59},
    {"color": "orange", "start": .59, "end": .625},
    {"color": "red", "start": .625, "end": .7},
]

visible_light_df = pd.DataFrame(visible_light)

# After the README -1.23e+034 is a NaN value from defective bands
# And that the ASDFR name string means that we have 2151 channels
# ranging from 0.35 - 2.5 microns
asdfr_start = 0.35
asdfr_end = 2.5
asdfr_channels = 2_151
asdfr_spectrum = pd.DataFrame(np.linspace(asdfr_start, asdfr_end, asdfr_channels), columns=["EM spectrum in microns"])

aspen_df = pd.read_csv(
    "2020-10-14/splib07a_Aspen_Aspen-1_green-top_ASDFRa_AREF/splib07a_Aspen_Aspen-1_green-top_ASDFRa_AREF.txt", 
    na_values="-1.2300000e+034",
    header=0,
    skiprows=1,
    names=["Reflectance in %"]
)

aspen_df = aspen_df.dropna()
aspen_spectrum = asdfr_spectrum.merge(aspen_df, left_index=True, right_index=True)

base = alt.Chart(aspen_spectrum)
interval = alt.selection_interval(bind="scales")

zoomed = base.mark_line().encode(
    x=alt.X("EM spectrum in microns", title="EM spectrum in microns", scale=alt.Scale(domain=[0.4, 0.7], zero=False)),
    y=alt.Y("Reflectance in %", title="Reflectance in %"),
    color=alt.value("black"),
)

vis_light_chart = alt.Chart(visible_light_df).mark_rect(opacity=0.65).encode(
    alt.X("start"),
    alt.X2("end"),
    color=alt.Color("color", scale=None)
)

alt.layer(
    vis_light_chart, 
    zoomed, 
    title="Zoomed in on visible light"
).add_selection(interval).properties(width=600, height=350)

Just as we thought, the leaf mainly reflects the green color! :tada:

But we also see that the reflection starts to increase sharply at the end of the red spectrum. The code draws the red range until 700nm, but after the spectral color map from Wikipedia, we can perceive wavelengths of up to 780nm. So why did we purposefully leave out the range from 700$-$780nm? Because we humans are less sensitive to wavelengths in that range. We only notice these wavelengths at a high intensity. ¹ Due to the limited effect on our color perception, the visualization only includes a wavelength of up to 700nm. So even if our leaf reflects more radiation starting at around 700nm, we perceive it as green.

As this already indicates, in reality, it is not that easy to map from one spectral plot to the color we perceive. The color we see also depends on the hue, intensity, and how our brain processes the surrounding colors/lighting conditions.

WIRED uploaded an interesting video if you want to learn more about how our brain influences our color perception. They show how we can create images with specific lighting conditions such that people see different colors, similar to the dress seen in the following figure.

Black and blue or gold and white dress (Image from Wikipedia)

Video about color constancy by Wired

But let's go back to our previous graph. Suppose we look at the complete spectrogram curve. In that case, we see a new property of our leaf: It heavily reflects the radiation in the non-visible spectrum, specifically in the near-infrared spectrum between 0.7 and 1.3 microns.

full = base.mark_line().encode(
    x=alt.X("EM spectrum in microns", title="EM spectrum in microns"),
    y=alt.Y("Reflectance in %", title="Reflectance in %", scale=alt.Scale(domain=[0, 0.5])),
    color=alt.value("black"),
)


alt.layer(
    vis_light_chart, 
    full, 
    title="Full spectrum curve"
).add_selection(interval).properties(width=600, height=350)

Important: Differenciating objects by comparing their spectral reflectance is central for multi-spectral remote sensing applications!

While the curve represents green vegetation in general, the shape of the curve in the near-infrared range can be used to discriminate between different plant species. Most healthy vegetation has a very high reflectance in the near-infrared range, while fake vegetation, and many other materials don't. In stark contrast to healthy vegetation, liquid water greatly absorbs electromagnetic radiation in the near-infrared spectrum. By dissolving substances, we would also affect the spectral reflectivity of the water.

# After the README -1.23e+034 is a NaN value from defective bands
# And that the BECK name string means that we have 480 channels
# ranging from 0.2 - 3 microns
beck_start = 0.2
beck_end = 3
beck_channels = 480
beck_spectrum = pd.DataFrame(np.linspace(beck_start, beck_end, beck_channels), columns=["EM spectrum in microns"])

water_df = pd.read_csv(
    "2020-10-14/splib07a_Seawater_Open_Ocean_SW2_lwch_BECKa_AREF/splib07a_Seawater_Open_Ocean_SW2_lwch_BECKa_AREF.txt", 
    na_values="-1.2300000e+034",
    header=0,
    skiprows=1,
    names=["Reflectance in %"]
)

water_df = water_df.dropna()
water_spectrum = beck_spectrum.merge(water_df, left_index=True, right_index=True)

water_base = alt.Chart(water_spectrum)
interval = alt.selection_interval(bind="scales")

water = water_base.mark_line().encode(
    x=alt.X("EM spectrum in microns", title="EM spectrum in microns"),
    y=alt.Y("Reflectance in %", title="Reflectance in %", scale=alt.Scale(domain=[0, 0.5])),
    color=alt.value("black"),
)

alt.layer(vis_light_chart, water, title="Seawater").add_selection(interval).properties(width=600, height=350)

As we can see, materials have unique spectral reflectance curves. If we do not limit ourselves to the visible light we could more easily differentiate between objects, even if they seem identical to us humans. If we could see the near-infrared band, green grass would light up, but fake grass wouldn't.

With modern satellites, we can sense multi-spectral images and get even more information about the area than from simple RGB images. These images focus on specific bands from the spectral curve. To fully utilize the data from all spectral bands, deep-learning is gaining popularity. In the next posts, we will take a closer look at a dataset, which can be used to train neural networks. But, before we move on, let's summarize the main points of this post.

Summary

Objects either emit or reflect/absorb electromagnetic radiation
Spectral reflectance curves show us what parts of electromagnetic radiation is reflected and absorbed
Spectral reflectance curves are unique to materials/objects and can be used to differentiate them
- Real green grass has a high reflectance in the near-infrared spectrum
- Fake grass doesn't reflect electromagnetic radiation in the near infrared spectrum very well
Multi-spectral images focus on specific bands of the spectral reflectance curves
Deep-learning is gaining popularity for combining and processing the information of all bands

See you in the next post; until then, have a productive time! :+1:

1. FYI, some call this range the near-infrared range, but this is not standardized.↩

About

In the previous two posts, we looked at grayscale images, which used a single channel, RGB images with three channels, and at multi-spectral images with various channels. I provided Python image generating code to be copied and executed locally or ran as notebooks via the badges on binder or Colab. But, it may not be as useful for people without a programming background. In my opinion, it is crucial to visualize and interact with technical topics to get a better feel for it or to be able to look at it from a different angle.

Therefore, I will try to introduce some widgets from time to time. Due to hosting limitations, these will be slow and may not look as polished but they are free of charge for me and allow you to interact with the written widgets. :)

It took me some time to find a viable solution, so the widgets are posted separately. I will show both widgets here for quick access, but I will also add them to the previous posts, where they belong.

Note: Sometimes the initialization of the widgets hang. The best solution I found was reopening the same page in Incognito Mode and reloading the page. It can take up to a minute until the widgets are loaded.

About

In the previous post, we took a closer look at grayscale images and how they are encoded on the computer. But these types of images are boring... Images with color are nicer to look at and carry more information for us. In the following, we will examine RGB images and what the difference is between those images and multi-spectral images.

Short recap: Grayscale images

To summarize our findings from the previous post: We have seen that every image consists of pixels. These pixels are nothing more than numbers stored as binary values on the disk. Depending on how many bits we use per pixel, we can tune the number of distinct colors (color-depth). Besides adjusting the color-depth, we also tuned the resolution. With a higher resolution, we can show more details. Remember, with a single pixel, we can only encode a single color, but with many pixels, we can capture scenes and objects!

RGB images

So how do we add color information to our images? If we think back to art class, we may remember primary colors. Red, green, and blue can be combined to create any color. Because we work with displays and light emitting from them, we additively mix the primary colors.¹ Each of these three colors has its own channel. In the previous grayscale examples, there was only a single channel for our gray values. Now we use three channels instead of one. These three channels are combined and presented to us in an additive manner. A close-up image of an LCD screen helps us to understand how we create colors on our screen.

Subpixels of an LCD monitor (Image from Robnil01, Wikimedia)

Our color values are once again only binary values on our disk. But if we show them on our LCD screen, each pixel has a red, green, and blue subpixel, which are combined to show us the color we want. Now let's start to program and see if we can mix some colors with Python!

Note: Now there are different places where we could put the color channel dimension. Sadly, almost every library has a different definition of where the color channel, height, and width dimension should be. So always check the library you are using to see how you should lay out the data. Otherwise, you will not get the results you expect! PIL assumes the following order: W x H x C

import numpy as np
from PIL import Image, ImageOps

def to_rgb_image(x):
    return Image.fromarray(x, mode="RGB")

def upscale_image(img, img_width=224, img_height=224):
    return img.resize((img_width, img_height), resample=Image.NEAREST)

# Let's start with a single pixel but with three channels!
# Btw do not forget to set dtype, otherwise the colors will be wrong ;)
img_values = {
    "pixel_red": np.array([255, 0, 0], dtype=np.uint8).reshape(1, 1, 3),
    "pixel_green": np.array([0, 255, 0], dtype=np.uint8).reshape(1, 1, 3),
    "pixel_blue": np.array([0, 0, 255], dtype=np.uint8).reshape(1, 1, 3),
}

for name, value in img_values.items():
    img = to_rgb_image(value)
    img = upscale_image(img)
    bordered_img = ImageOps.expand(img, border=1, fill="black")

Channel 0 -- Red

Channel 1 -- Green

Channel 2 -- Blue

Visualization of primary color pixels

Now that we have the three primary colors, we can mix them to get almost any color! Here are some examples:

pixel_values = {
    "black": [0, 0, 0],
    "white": [255, 255, 255],
    "red_and_green": [255, 255, 0],
    "green_and_blue": [0, 255, 255],
    "all_150": [150, 150, 150],
}

for name, p_val in pixel_values.items():
    img = to_rgb_image(
        np.array(p_val, dtype=np.uint8).reshape(1, 1, 3)
    )
    img = upscale_image(img)
    bordered_img = ImageOps.expand(img, border=1, fill="black")

All values 0

All values 255

All values 150

Red & Green 255

Green & Blue 255

Visualization of different color values

You can also play around with the following widget to combine the red (r), green (g), and blue (b) color values to a single RGB color²:

If we don't limit ourselves to a single pixel, we can visualize vibrant images, showing us many different objects and scenes. With the extra color information, we can more easily differentiate objects, like flowers or fruits.

Grayscale image

RGB image

Image with RGB colors vs. grayscale image

In the computer vision field, most architectures also work with RGB images. These are the types of images we usually use for everything. The extra color information helps machine learning researches to increase the accuracy of the predictions further. It seems reasonable for us humans to assume that color information improves the prediction performance because it is easier to identify objects if we add color to the image. But for the computer, these are once again nothing more than 0s and 1s. So what would happen if we add more channels?

Before we move on, let's summarize what we have learned so far.

Summary

To summarize the previous section: The LCD screens we are looking at combine red, green, and blue subpixels in each pixel to transform the binary values into colors. These three colors are used because they are primary colors and can be additively combined to create any color. On disk, these values are still nothing more than binary numbers. But now we have three channels and, therefore, three times as many bytes per image compared to a grayscale image. For a 28 x 28 pixels image, we now have 28 x 28 x 3 x color-depth bytes. The extra color information helps us (and neural networks) to identify and differentiate objects.

Introduction to remote sensing images

In the previous section, we saw how different a grayscale image looks from an RGB image and that it is easier for us to identify and differentiate colored objects. The same holds for neural networks! In the computer vision setting, images are used as input, and the network takes some action based on it. For example, we could use it to predict a specific class (dog or cat) or transform the picture (remove people from the scenery). But what would happen if we add more channels? Would the prediction performance still increase?

In a field called remote sensing we sometimes use multi-spectral images as input images. After Wikipedia, remote sensing is:

In current usage, the term "remote sensing" generally refers to the use of satellite or aircraft-based sensor technologies to detect and classify objects on Earth.

So the images used in remote sensing could be classic RGB images from drones used to classify different objects. Here the classes could be building, car, forest, water, fields, etc. With the introduction of Deep Learning and neural networks in the field, the processing of a different image type is gaining popularity:multi-spectral images. To answer what multi-spectral images are, let's take a step back and think about how our RGB images are displayed on an LCD screen. We know that each pixel uses subpixels to add them together to a color. The subpixel shines in a single color. More accurately, the subpixel emits electromagnetic waves in a specific wavelength in the spectrum of visible light. Here, visible refers to a spectrum we humans can perceive.

Electromagnetic spectrum (Image from Wikipedia)

A blue subpixel mostly emits an electromagnetic wave with a wavelength of around 470nm, which we perceive as blue. That means that the LCD screen does not add the different wavelengths together in some way, it only drives the subpixels differently, and for our eyes, it seems like the light of the subpixels have been combined to a specific color. As we have seen previously, if we zoom in on an LCD screen, we can differentiate the subpixels' primary colors again.

If our LCD screen would only emit electromagnetic waves outside of the visible spectrum, it would be of little benefit to us humans. But visualizing bands that we aren't able to see is quite helpful! A well-known use case is thermal imaging. Here the long-infrared band is detected or sensed and visualized. The long-infrared band shows us the temperature variations, even if the objects aren't visible to us. The following figure shows us an example.

Normal RGB image (Image from Wikipedia)

Infrared image (Image from Wikipedia)

RGB vs. infrared image

Note: Note that the infrared image only focusses on a single band and visualizes it as a normal RGB image with an intensity scale next to it.

If we look at both the RGB and infrared images, we can combine them to get even more information! The combination would count as a multi-spectral image. We do not limit ourselves to three visible light bands and can gain even more insights. What bands are sensed depends on the given sensor and the desired use-case. For example, the Sentinel-2 satellite takes multi-spectral images with 13 bands in the visible, near-infrared, and short wave infrared part of the spectrum. The next post will take a closer look at how we can load and visualize these remote images. But, before we move on, let's summarize what we have learned so far.

Summary

We went from grayscale images with a single channel, to RGB images with three channels, to multi-spectral images with more than three channels. These multi-spectral images do not only focus on the visual-spectrum of electromagnetic waves but use even more bands. The main idea is that the information from different bands allows us to learn more about the object or scene. In our previous example, we were able to verify that the person had five fingers on his left hand, even if we weren't able to see it from the visual-spectrum of the light. For my master thesis, I hope that this additional information can be used in multi-spectral remote sensing images to increase neural networks' accuracy and robustness further.

But before we dive deep into neural networks, we first need to understand how we can visualize and work with these multi-spectral images, which will be the goal of the next post.

Until then, have a productive time! :+1:

1. For a more detailed comparison of additive vs. subtractive colors, see the blog post from thepapermillstore.com ↩

2. Sometimes the initialization of the widgets hang. The best solution I found was reopening the same page in Incognito Mode and reloading the page. It can take up to a minute until the widgets are loaded.↩

About

Before we can dive into multi-spectral satellite images, I think a quick refresher on how images are encoded and represented in memory is a good starting point.

Binary encoding

Let's take a short recap of how classical computer vision images are encoded in memory. Internally a computer (ignoring quantum-computing) only works with binary numbers. A binary number is either a 0 or a 1, on or off. The value of such a binary number is called a bit. The smallest data-element is called a byte. A byte consists of 8 bits. There are different ways how we could use these 8 bits/1 byte to encode our data. The data we are trying to store/load defines how we interpret the data. If we want to only work with positive integers, we use an unsigned integer type. An unsigned integer with 8 bits can encode all numbers from 0$-$255. If all bits are 1, also called set, the value is 255. If all bits are 0 the corresponding value is 0.¹

Grayscale images

Images, like everything in a computer, are also only encoded in binary values. The most straightforward images are grayscale images. The possible colors of each pixel of a grayscale image can only range from black to gray to white, with all different gray shades in-between. Pixels are the basic elements of a picture. The word itself, pixel, is a combination of the words picture and element/cell. So an image consists of pixels, similar to how a brick wall consists of bricks.

Pixel

Complete Image

My weird analogy

We can understand how simple 8-bit grayscale images are encoded with the knowledge of our previous simple encoding scheme. The 8-bit refers to the color-depth. It indicates how many bits are used per channel. We only have a single channel for a grayscale image, the colors range from black to white. (We will take a closer look at different channels in the next post.) For now, we note that our grayscale channel is encoded with 8-bits. Or, put differently, we use 8-bits for every pixel to show different shades of gray. With 8-bits, we can color each pixel in 256 (2⁸) different ways.

With the numpy and PIL library, we can easily create our own 8-bit grayscale image by merely changing the value of a byte.

import numpy as np
from PIL import Image, ImageOps

def to_grayscale_image(x):
    grayscale_8_bit_mode = "L"
    return Image.fromarray(x, mode=grayscale_8_bit_mode)

def upscale_image(img, img_width=224, img_height=224):
    return img.resize((img_width, img_height), resample=Image.NEAREST)

# PIL requires np arrays as input
# Datatype is uint8, our unsigned int consisting of 8-bits
# zero is our single byte/value with value 0
# -> Array has a width and height of 1
zero = np.zeros((1, 1), dtype=np.uint8)

img_values = {
    "pixel_0": zero, 
    "pixel_64": zero + 64, 
    "pixel_192": zero + 192, 
    "pixel_255": zero + 255
}

for name, value in img_values.items():
    img = to_grayscale_image(value)
    img = upscale_image(img)
    bordered_img = ImageOps.expand(img, border=1, fill="black")
    # display(bordered_img) # To display in jupyter
    bordered_img.save(f"2020-09-02/{name}.png")

0

64

192

255

Visualization of different 8-bit grayscale pixel values

You can also play around with the following widget to see the grayscale values of your own choice²:

Until now, we did not care about the resolution of our images. The resolution defines how many pixels we use to visualize the object. A resolution of 1 corresponds to a single pixel. But, with a single-pixel picture, we cannot retain a lot of information. As shown above, we could only create a single shade of gray. Let's increase our resolution for the following images to a size of 224 pixels x 224 pixels. With more pixels, we can show more levels of detail.

Now we can extend our previous code to draw gradients!

zeros = np.zeros((224, 224), dtype=np.uint8)
x_gradient = np.arange(0, 224, dtype=np.uint8).reshape(1, 224)
y_gradient = np.arange(0, 224, dtype=np.uint8).reshape(224, 1)

# Using numpy's broadcasting
x_grad_2d = zeros + x_gradient
y_grad_2d = zeros + y_gradient
sum_grad_2d = x_gradient + y_gradient
diff_grad_2d = x_gradient - y_gradient

# Convert to a grayscale image as before
# and save or show files

Visualization of different 8-bit grayscale images

If we don't limit ourselves to simple mathematical operations, we can show images with great detail.

Typical test image

The pepper's image consists of the following values. Each value is saved as a single byte on disk.

array([[ 71,  96,  92, ..., 136, 132, 127],
       [ 87, 119, 113, ..., 181, 176, 170],
       [ 83, 114, 111, ..., 178, 174, 166],
       ...,
       [ 92, 120, 108, ..., 191, 198, 198],
       [ 88, 124, 104, ..., 202, 198, 194],
       [ 78, 126, 106, ..., 200, 193, 188]], dtype=uint8)

Important: Even if these images reveal a lot of information to us humans, in the end, they are only stored as 0s and 1s on the computer.

Before we move with the next topic, let's review what we have learned so far.

Summary

Increasing the number of pixels (resolution) allows us to encode more details. The color-depth shows us how many bits we use per channel to encode a color. Our previous 8-bit grayscale pixel can, therefore, encode 256 different shades of gray. With a higher color-depth, we have access to more shades. But what is when we want to enrich our image with colors?

How to add colors to our image and how remote sensing images are different will be the topic of the next blog post!

Until then, have a productive time! :+1:

1. A quick refresher on how to translate binary numbers to unsigned integers can be found on ryanstutorials ↩

2. Sometimes the initialization of the widgets hang. The best solution I found was reopening the same page in Incognito Mode and reloading the page. It can take up to a minute until the widgets are loaded.↩

Kai’s Master Thesis Blog

Introduction to the BigEarthNet dataset

About

Sentinel-2 mission

BigEarthNet

Summary

References

Understanding spectral reflectance

About

Spectral reflectance

Summary

Color widgets

About

Introduction to multi-channel images

About

Short recap: Grayscale images

RGB images

Summary

Introduction to remote sensing images

Summary

Introduction to grayscale images

About

Binary encoding

Grayscale images

Summary