5 Analysing film colour

Like sound design, colour shapes our experience of the cinema in many different ways. Different genres are associated with different colour schemes and with different levels of brightness and saturation (I.-P. Chen et al., 2012; Redfern, 2021a) so that we can identify a genre the its use of warm and cool colours, the level of the lightness of the image, and the level of saturation (Bellantoni, 2005). Colours within a film carry meanings determined by the local economy of the individual text, associated with narratively significant objects, characters, locations, and themes, that nonetheless exist within the culturally codified of meanings colour shared by audiences belonging to a particular culture. Colour acts as a unifying factor, fusing together different aspects of film style including production design, costume design, lighting, cinematography, and post-production colour processing to create a coherent world for the viewer (Rotem, 2003). Changes in the dominant colour of a film often mark changes in mood, location, and the evolution of characters. Colour organises our attention as viewer, separating the foreground and background and drawing our attention to what is most important within a frame. Colour sets the mood of a film, conveying emotional information about the internal lives of characters and the state of the on-screen world while also eliciting emotional repsonses from the viewer, influencing both what we feel and the intensity of those emotions (Detenber et al., 2021).

In this chapter we will cover how to use R to analyse colour in motion pictures. We will learn how to sample frames from a film using the av package to pass commands to FFmpeg; how to calculate the average colour and colourfulness of a frame; how to create a colour palette for a film using cluster analysis; and how to analyse at the temporal structure of colours in a film using movie barcodes.

5.1 Colour models and spaces

A colour model is a mathematical model describing the ways a colour can be represented as an ordered, immutable list of numbers or tuple.

A colour space is a geometrical model that maps a colour model onto to actual colours to define the range of colours capable of being represented as determined by its gamut (the range of a colour space’s chromaticity), gamma (the non-linear distribution of luminance values), and whitepoint (a standard illuminant that defines the colour temperature of white within a colour space).

In this chapter we will work with three colour models and colour spaces: RGB, L*a*b*, and LCH(ab).

5.1.1 RGB

The RGB colour model (Figure 5.1) is an additive colour model in which colours are defined by the chromaticity of red, green, and blue primaries. Colours are specified as a tuple of RGB values with one number for each colour channel, with each channel represented on a scale of $[0, 1]$ or $[0, 255]$. For example, red has the maximum amount of red light but no green or blue light is given by the tuple $(1, 0, 0)$ or $(255, 0, 0)$, whereas magenta has the maximum amount of red and blue light but no green and is specified as $(1, 0, 1)$ or $(255, 0, 255)$. When all the values in an RGB tuple are equal then the specified colour is achromatic, ranging from black $(0, 0, 0)$ to white ($(1, 1, 1)$ or $(255, 255, 255)$), with greys lying between these extremes. For example, middle grey is $(0.5, 0.5, 0.5)$ or $(128, 128, 128)$.

The different scales used to represent RGB colours are equivalent and it is easy to convert between them by multiplying or dividing by 255. There is no standard way to represent RGB colours in R. Some packages use the $[0, 1]$ scale while others us $[0, 255]$. For consistency, we will use the latter here because this is how RGB colours are expressed in image and video processing software.

RGB colours can also be specified using hex codes by converting the tuple of decimal numbers to hexadecimal format. This is typically how numbers are represented in R when plotting. Each channel is described by a two character hex code in the range $[00, FF]$ and respects the ordering of the RGB tuple to form a hex triplet: $\#RRGGBB$. Thus, the hex code for red is $\#FF0000$ and the hex code for mid-gray is $\#808080$.

Figure 5.1: The RGB colour space (18-bit) (CC-BY-SA 4.0: Kjerish)

RGB is the most common colour model used for devices that project light, and so is the default colour space for televisions and computer monitors. Colour spaces using the RGB colour model include sRGB, which is the colour space of the world wide web; Rec.709, the colour space used by high definition television; and Rec.2020, which is the colour space used by ultra-high definition television.

RGB is not a perceptually uniform colour model – that is, the distances between two colours is not proportional to the Euclidean distance between them and so differences in the RGB values do not reflect the differences that we perceive between colours. We will use RGB colour values in our project to display colours on screen, but when performing calculations or plotting the distribution of colours we will use perceptually uniform colour spaces.

5.1.2 CIELAB

The CIELAB colour model or L*a*b* is a perceptually uniform colour space defined by three attributes:

Lightness (L*): the perceptual lightness of a colour in the range $[0\%, 100\%]$
a*: the chromaticity of a colour on the green (-a*) – red(+a*) axis
b*: the chromaticity of a colour on the blue (-b*) – yellow (+b*) axis

Achromaticity occurs when a* = b* = 0, ranging from black (L* = 0%) to white (L* = 100%), with greys lying between these limits. Figure 5.2 plots swatches from the CIELAB colour space for a range of lightness values.

Figure 5.2: Swatches from the CIELAB colour space for a range of lightness values.

The range of a* and b* is theoretically unlimited, though typically is clamped to the range $[-128, 127]$. Note that this is not true for the colorspace package, which clamps the range to $[-100, 100]$. Converting between colour spaces should be done with the same package in a project to avoid errors resulting from different ways of representing colours. For example, converting RGB colours to CIELAB using the colorspace package and then converting back to RGB with the farver package will not reproduce the original RGB colours because these packages express the ranges of a* and b* differently. Either package can be used for the purposes of analysis, but it is best to use only one package for colour conversion in a project. In this chapter we will use the farver package.

5.1.3 LCH(ab)

The L*a*b* colour space is perceptually uniform but it is not intuitive. LCH(ab) is a polar representation of the L*a*b* colour space that is much easier to understand. LCH(ab) has three colour attributes:

Lightness (L): this is identical to the L* value defined for the L*a*b* colour space above
Chroma (C): the saturation of a colour measured as the distance from the achromatic axis in the range $[0\%, 100\%]$
Hue (H): the basic colour represented as the angle on a colour wheel, where red = 0°, yellow = 90°, green = 180°, and blue = 270°.

Achromatic colours occur when C = 0%, and vary with lightness, from black (L = 0%) to white (L = 100%), with arbitrary values for hues. Figure 5.3 plots slices of the LCH(ab) colour space for a range of lightness values.

Figure 5.3: The LCH(ab) colour space sliced at a range of lightness values

We can calculate the saturation of a colour from its lightness and its chroma:

\[ S_{ab} = 100 \times \frac{C_{ab}}{\sqrt{C_{ab}^{2} + L_{ab}^{2}}} , \]

where saturation represents the proportion of pure chromatic colour in the total colour experience.

5.2 Set up the project

5.2.1 Create the project folder

The first tasks are to create a project in RStudio with a new folder called Colour that will be the working directory for this chapter and will contain the associated .rproj file. Once this step is completed, we can run the script projects_folders.R we created in Chapter 3 to create the folder structure required for the project.

We will create some additional folders during the course of the project.

5.2.2 Packages

In this chapter we will use the packages listed in Table 5.1.

Table 5.1: Packages for analysing colour data.
Package	Application	Reference
av	Perform operations on video files	Ooms (2022)
devtools	Load functions directly from GitHub	Wickham et al. (2021)
farver	Convert between different colour spaces	Pedersen et al. (2021)
fpc	Perform cluster analysis	Hennig (2020)
ggpubr	Combine plots into a single figure	Kassambara (2020)
here	Use relative paths to access data and save outputs	K. Müller (2020)
imager	Image loading and data extraction	Barthelme (2022)
pacman	Installs and loads R packages	T. Rinker & Kurkiewicz (2019)
tidyverse	Data wrangling and visualisation	Wickham et al. (2019)
treemapify	Data visualisation using treemaps	Wilkins (2021)
viridis	Accessible colour palettes	Garnier (2021)

5.2.3 Fuelled

In this chapter our data set will be Fuelled (Video 5.1), a short animated film created by students at Sheridan College, Ontario, that premiered on YouTube in December 2021.

The film tells the story of Cathy, a cat widowed when a dog murders her husband, and who sets out to find the killer. When her car runs out of fuel in a forest and she finds that she has also run out of money, Cathy resorts to stealing from a gas station, leading her to assault the attendant leading to an explosion that destroys the gas station. Realising that she has become like the killer she is chasing, Cathy calls 911 and waits.

Video 5.1: Fuelled (2021) © KilledtheCat Productions. ☝️

We can download this film from YouTube and place it in our Data folder. Using av::av_media_info() we can get a summary of the video file, including the duration of the film in seconds and information about width, height, number of frames, and the frame rate of the video and sampling and bit rates and the number of audio samples of the audio, as well as the video and audio codecs.

# Load the here and av packages
pacman::p_load(av, here)

# Get a summary of the video file fuelled.mp4
av_media_info(here("Data", "fuelled.mp4"))

## $duration
## [1] 542.6735
## 
## $video
##   width height codec frames framerate  format
## 1  1280    536  h264  13024        24 yuv420p
## 
## $audio
##   channels sample_rate codec frames bitrate layout
## 1        2       44100   aac  23371  128004 stereo

5.2.3.1 Sample and reduce

Each frame of Fuelled is made up of 1280 by 536 pixels, with each pixel comprised of the three channels of the RGB colour space. Therefore, to represent the complete colour information of a single frame we will need $3 \times 1280 \times 536 = 2,058,240$ numbers. With a frame rate of 24 frames-per-second and a running time of 542.6735 seconds, there are a total of 13024 frames. To completely represent the colour information in Fuelled requires $2,058,240 \times 13,024 = 26,806,517,760$ data points. A data set comprising almost 27 billion data points is too large for analysis – and this represents a very small dataset (how many data points would be needed for a for a two-hour long 4K Ultra HD film at 60fps?). Furthermore, much of this data will be redundant as frames in the same scene will be very similar to one another.

All analyses of film colour are therefore based on the same two-stage process of in order to reduce the amount of data to a manageable level and to eliminate redundant data. First, frames are sampled from the film, selecting either every n-th frame or n frames per second. The second stage is to reduce the colour data in each sampled frame to a small set of values or a single value that represents the frame, such as the average colour of a frame, its colourfulness, or any of its colour attributes (such as lightness, chroma, saturation, etc.).

For example, movie barcodes are a popular way of representing colour information in a film and are constructed by selecting a set of frames from a film before either reducing that frame to a single pixels width or smoothing the data by averaging the colours within each of the selected frames. We will represent colour information in Fuelled as a barcode.

Other methods employed in computational analyses of film colour, including cinemetrics, z-projections, palettes, treemaps, Color_dT plots (see Flueckiger & Halter (2020) for an overview), all apply some version of the sample-and-reduce approach.

From the Fuelled .mp4 file we want to extract a large enough number of frames that our sample will be representative of the film without oversampling redundant information that will only serve to slow down our processing of the data.

The function av::av_video_images() splits a video file into frames, with the number of frames sampled controlled by the fps argument. A sampling rate of $\frac{1}{1}=1$ will return one frame-per-second. Increasing the numerator will increase the sample rate – a sampling rate $\frac{2}{1}=2$ will sample two frames every second; whereas a increasing the denominator will reduce the sampling rate – a sampling rate of $\frac{1}{2}=0.5$ will sample one frame very two seconds. Sampling Fuelled at a rate of two frames-per-second will give us a data set of 1085 frames to analyse.

av_video_images(here("Data", "fuelled.mp4"), destdir = here("Data", "Frames"), 
                format = "jpg", fps = 2)

We can create a folder to store the frames within our Data folder directly within the call to av_video_images(). Sampled frames can be either .jpg or .png files, with the latter returning lossless images with larger file sizes.

👈 Click here to find out how to collect colour data using the chromaR package

Collecting data using chromaR

As part of the chromaR toolkit for analysing colour in motion pictures, Tommaso Buonocore has provided a MATLAB script that will create a .csv file containing the average colour in the sRGB colour space of every frame of video file. You can access the script (videoProcessing.m) on the GitHub repository for chromaR.

In order to run this script you will need to have access to MATLAB, which is much quicker than performing the same process of sampling frames and calculating the average colour in R. However, MATLAB is paid-for software and is not cheap.

For those who do not have access to MATLAB or cannot afford to purchase it, I have produced an alternative version of Buonocore’s MATLAB script that will run in Octave, a freely-available open-source alternative to MATLAB. You can find this script and the instructions for using it on my GitHub repository: VideoProcessingOctave.

The .csv file containing the RGB values of each frame created from this script can be used for colour analysis in chromaR or by the methods illustrated here instead of creating a data frame by sampling and processing frames from a film.

5.3 Analysing Fuelled

5.3.1 The average colour of a frame

One of the most common ways of reducing the colour data of an frame is to calculate the average colour, which is defined as the tuple of the average values of the individual attributes of a colour space. That is, to find the average colour of a frame in the CIELAB colour space we need to calculate the average of the L*, a*, and b* attributes separately and form the tuple to from those averages. Typically, the average used is the mean, but the median and mode can also be used to determine the average colour of a frame (Plutino et al., 2021).

First, we will get a list of all the images in the frames folder using the list.files() function. We want to include all the .jpg files so we set pattern = "*.jpg$, where * is a wildcard that will ignore the part of the filename before the extension .jpg and by adding $ after the extension we tell R to find the pattern jpg at the end of the string that is the filename of the image.

# get list of jpg files in the Frames directory
images <- list.files(here("Data", "Frames"), pattern = "*.jpg$")

head(images)

## [1] "image_000001.jpg" "image_000002.jpg" "image_000003.jpg" "image_000004.jpg"
## [5] "image_000005.jpg" "image_000006.jpg"

To do this we will loop over all the frames sampled from Fuelled and extract the red, green, and blue colour channels of each image. The RGB values are multiplied by 255 for scaling. We will then convert these colour values using farver::convert_colour() to the CIELAB colour space and get the mean value of each parameter ($\widehat{L^{*}}$, $\widehat{a^{*}}$, $\widehat{b^{*}}$). Next, we convert the tuple of the mean colour in CIELAB to the LCH(ab) colour space and to RGB for plotting. It is important to remember that names in R are case-sensitive, and so we will name colour attributes in the CIELAB and LCH(ab) colour spaces using UPPERCASE letters and we will name the RGB attributes using lowercase letters. This will help us to avoid confusing the b* attribute in L*a*b* colour space with the blue channel of the RGB colour space, which will be named B and b respectively. farver::convert_colour() gives all attributes in all colour spaces lowercase names by default, so that b* and blue are both represented by b, and without taking this into account we would overwrite the b* attribute with the blue channel when we do the final conversion to RGB and lose some key data.

In this loop we will also do a couple of other tasks. We will calculate the time of each frame so that we can analyse how colour evolves over time. We will also calculate the saturation of a frame from its chroma (C) and lightness (L), taking care to ensure that the saturation is set to 0 for achromatic colours (i.e., when C = 0%) and avoiding NA values in the data frame. This is easily done using dplyr::if_else, which will check to see if C = 0 (NB: note that this requires double equal signs == to represent the logical version of is equal to) and return 0 when this condition is TRUE and calculate the saturation from C and L when it is FALSE. We will also tidy up as we go, using the function rm() to remove objects from the workspace once they are no longer required so that we don’t have large objects hanging around when they are not needed.

# Load the tidyverse and farver packages
pacman::p_load(tidyverse, farver)

# Create an empty data frame to store the result of each frame
df_average_colours <- data.frame()

# Set frame to zero; frame is used as an index and to calculate the time of a frame
frame <- 0

for (i in images){
  
  # Increment frame each time the loop starts
  frame <- frame + 1
  
  # Grab the name of each image to use as an identifier and drop the file extension
  frame_id <- gsub(pattern = ".jpg$", "", basename(as.character(i)))
  
  # Calculate the time of the frame
  time <- frame * 0.5
  
  # Load the i-th frame using imager::load.image()
  im <- load.image(here("Data", "Frames", i))
  
  # Separate R, G, B channels and gather as a data frame
  rgb <- cbind(R(im), G(im), B(im)) %>% as.data.frame %>% 
    rename(r = 1, g = 2, b = 3) %>%
    mutate(r = r * 255, g = g * 255, b = b * 255) # multiply by 255 to scale correctly
  
  # Remove the image from the workspace once we've collected the colour data
  rm(im)
  
  # Convert to RGB to CIELAB colour space using farver::convert_colour()
  lab <- convert_colour(rgb, from = "rgb", to = "lab")
  # Remove the rgb data frame as we no longer need it
  rm(rgb)
  
  # Calculate the mean values of each attribute in the L*a*b* colour space
  pix_lab <- data.frame(L = mean(lab$l), 
                        A = mean(lab$a), 
                        B = mean(lab$b))
  # Remove the lab data frame as we no longer need it
  rm(lab)
  
  # Convert to LCH(ab) colour space using farver::convert_colour()
  pix_lch <- convert_colour(pix_lab, from = "lab", to = "lch") %>%
    rename(L = l, C = c, H = h) # using uppercase letters for consistency
  
  # Calculate the saturation from the chroma and lightness:
  # if a colour is achromatic (i.e. C = 0) then the saturation is also 0 - 
  # this avoids NA values in the data frame
  pix_lch <- pix_lch %>% mutate(S = if_else(C == 0, 0, 100*(C/sqrt(C^2 + L^2))))
  
  # Drop the L parameter from pxi_lch because it is the same as the L parameter in pix_lab
  pix_lch <- pix_lch %>% select(C, H, S)
  
  # Convert the mean CIELAB values to RGB using farver::convert_colour() for plotting -
  # the RGB values will be identified by lowercase letters
  pix_rgb <- convert_colour(pix_lab, from = "lab", to = "rgb")
  
  # Collect all the results for a frame
  pix <- cbind.data.frame(frame, frame_id, time, pix_lab, pix_lch, pix_rgb)
  
  # Add the results to the data frame for export
  df_average_colours <- rbind.data.frame(df_average_colours, pix)

}

Table 5.2 displays the average colour for outputted by the for loop selected frames.

👈 Click here to learn about vectorisation in R

Vectorisation

R functions are vectorised and will operate on all elements in a vector without requiring an explicit for loop to apply a function to elements in a vector one at a time.

For example, when we convert the colour values of RGB colours from the $[0, 1]$ scale to $[0, 255]$ we multiply the red colour channel RED by 255 and every individual value in that vector is multiplied by 255.