RedCV: Motion detection with differential images

The Red code is based on the following paper: Collins, R., Lipton, A., Kanade, T., Fijiyoshi, H., Duggins, D., Tsin, Y., Tolliver, D., Enomoto, N., Hasegawa, O., Burt, P., Wixson, L.: A system for video surveillance and monitoring. Tech. rep., Carnegie Mellon University, Pittsburg, PA (2000).

The idea is to use the webcam and analyse motion using differential images.

Differential Images

Differential Images are the result of the subtraction of two images:  Idif (x, y) = I1(x, y) − I2(x, y).

A differential image shows the pixels differences between two images. With those images you can make movement visible.

Based on the quoted paper,  we use a differential image calculated from three consecutive images It−1 (previous image),  It (current image) and It+1 (next image). The advantage of this technic is that the motionless  background is removed from the result, and only motion pixels are visible.

RedCV offers the possibility to subtract two images from each other using rcvAbsDiff. Since logical operations on images are also implemented, we use rcvAnd to achieve the final differential image.

Code

Implemented code is simple to understand and just requires a time event to activate the camera. When the time event occurs, differential image is calculated and updated in the base object. To-image Red function allows to copy the webcam image to a Red image. Thanks Red!

Red [
    Title:   "Test image operators and camera Red VID "
    Author:  "Francois Jouen"
    File:    %motion.red
    Needs:   'View
]

; all we need for computer vision with red
#include %../../libs/redcv.red ; for red functions

iSize: 320x240
prevImg: rcvCreateImage iSize
currImg: rcvCreateImage iSize
nextImg: rcvCreateImage iSize
d1: rcvCreateImage iSize
d2: rcvCreateImage iSize
r1: rcvCreateImage iSize
r2: rcvCreateImage iSize

margins: 10x10
threshold: 32

to-text: function [val][form to integer! 0.5 + 128 * any [val 0]]

view win: layout [
        title "Motion Detection"
        origin margins space margins
        text "Motion " 50 
        motion: field 50 rate 0:0:1 on-time [face/text: to-text rcvCountNonZero r2]
        btnQuit: button "Quit" 60x24 on-click [
            rcvReleaseImage prevImg
            rcvReleaseImage currImg
            rcvReleaseImage nextImg
            rcvReleaseImage d1
            rcvReleaseImage d2
            rcvReleaseImage r1
            rcvReleaseImage r2
            quit]
        return
        cam: camera iSize
        canvas: base 320x240 r2 rate 0:0:1 on-time [
            rcv2gray/average nextImg currImg    ; transforms to grayscale since, we don't need color
            rcvAbsdiff  prevImg currImg d1      ; difference between previous and current image
            rcvAbsdiff  currImg nextImg d2      ; difference between current and next image
            rcvAnd d1 d2 r1                     ; AND differences
            rcv2BWFilter r1 r2 threshold        ; Applies B&W Filter to base/image
            prevImg: currImg                    ; previous image contains now the current image
            currImg: nextImg                    ; current image contains the next image             
            nextImg: to-image cam               ; updates next image
        ]
        return
        text 40 "Select" 
        cam-list: drop-list 180x32 on-create [
                face/data: cam/data
            ]
        onoff: button "Start/Stop" 80x24 on-click [
                either cam/selected [
                    cam/selected: none
                    canvas/rate: none
                    motion/rate: none
                    canvas/image: black
                ][
                    cam/selected: cam-list/selected
                    rcvZeroImage prevImg
                    rcvZeroImage currImg
                    rcvZeroImage nextImg
                    canvas/rate: 0:0:0.04;  max 1/25 fps in ms
                    motion/rate: 0:0:0.04
                    ]
            ]
        text "Filter" 40
        sl1: slider 180 [filter/text: to-text sl1/data threshold: to integer! filter/data ]
        filter: field 40 "32" 
        do [cam-list/selected: 1 motion/rate: canvas/rate: none sl1/data: 0.32 ]
]

Result

RedCV: how to blend images

Sometimes it's interesting to superpose and mix two images to create a new image. Once again,  the variation of intensity of image can help us to solve this problem. The idea is to apply a factor (between 0. 0 and 1.0) on the image to modify intensity.  This is done for the first image and the second image. The complement of the factor (1.0 - factor) is applied to the second image.

rcvSetIntensity function

RedCV includes  a  function to do the job.

rcvSetIntensity: function [src [image!] dst [image!] alpha [float!]]

The function applies the float factor to the src image and update the result in the destination image.

Then it's very simple to combine a first call of the function on the first image with a factor and a second call on the second image with the complement of the factor:

rcvSetIntensity src1 dest1 alpha

rcvSetIntensity src2 dest2 1.0 - alpha

Then you just need to add both images to create the blended image with the rcvAdd function:

rcvAdd dest1 dest2 dst

Code sample

Red [
    Title:   "Blend Operator "
    Author:  "Francois Jouen"
    File:    %blend2.red
    Needs:   'View
]

;this version uses rcvSetIntensity and not rcvBlend

#include %../../libs/redcv.red ; for redCV functions
margins: 10x10
img1: rcvLoadImage %../../images/lena.jpg
img2: rcvLoadImage %../../images/test.jpg
dst: rcvCreateImage img1/size
tmp1: rcvCreateImage img1/size
tmp2: rcvCreateImage img2/size
alpha: 0.5

blending: function [] [
    rcvSetIntensity img1 tmp1 alpha 
    rcvSetIntensity img2 tmp2  1.0 - alpha 
    rcvAdd tmp1 tmp2 dst
]

; ***************** Test Program ****************************
view win: layout [
        title "Blend Operator Test"
        text 60 "Image 1" 
        f1: field 50 "0.5"
        sl: slider 170 [alpha: face/data * 1.0
                    f1/text: form alpha
                    f2/text: form (1 - alpha)
                    blending
                    ]
        text 60 "Image 2" 
        f2: field 50  "0.5"
        button 60 "Quit" [  rcvReleaseImage img1 
                            rcvReleaseImage img2
                            rcvReleaseImage tmp1
                            rcvReleaseImage tmp2 
                            rcvReleaseImage dst 
                            Quit]
        return
        canvas: base 512x512 dst
        do [sl/data: alpha blending]
]

And the result for a factor = 0.5

image 1

image 2

mixed image

RedCV: Basic Thresholding Operations

Note: The explanation below can be found in the book Learning OpenCV 
by Bradski and Kaehler.

Thresholding is a basic technique for separating out regions of an image corresponding to objects which we want to analyze. This separation is based on the variation of intensity between the object pixels and the background pixels. To differentiate the pixels we are interested, we perform a comparison of each pixel intensity value with respect to a threshold. Once we have separated the important pixels, we can set them with a determined value to identify them.

Types of thresholding

RedCV offers the function rcvThreshold to perform thresholding operations. This function allows 5 refinements that effectuate different types of thresholding operations. This function requires grayscale image and thus applies an automatic conversion to colored image.

rcvThreshold/binary: if the intensity of the pixel is higher than thresh, then the new pixel intensity is set to maxValue (i.e., 255)

rcvThreshold/binaryInv: If the intensity of the pixel is higher than thresh, then the new pixel intensity is set to 0. Otherwise, it is set to maxValue.

rcvThreshold/trunc: The maximum intensity value for the pixels is thresh. if pixel is greater, then its value is truncated.

rcvThreshold/toZero: If the pixel is lower than thresh, the new pixel value will be set to 0.

rcvThreshold/toZeroInv: If the pixel is greater than thresh, the new pixel value will be set to 0.

Code sample

As can be seen, the code is easy to understand and illustrates the elegance of Red language. Thresholding operations are defined into the ops block as an ordered list. Selected operation op is called by using Do instruction. Since maxVal is only required by /binary and /binaryInv refinements, the slider controlling maxValue is hidden in other cases.

    Title:   "Thresholding Operations"
    Author:  "Francois Jouen"
    File:    %threshold2.red
    Needs:   'View
]

#include %../../libs/redcv.red ; for redCV functions
margins: 10x10
defSize: 512x512
img1: rcvCreateImage defSize
dst:  rcvCreateImage defSize
isFile: false
thresh: 128
maxValue: 255

ops: [
    [rcvThreshold/binary img1 dst thresh maxValue]
    [rcvThreshold/binaryInv img1 dst thresh maxValue]
    [rcvThreshold/trunc img1 dst thresh maxValue]
    [rcvThreshold/toZero img1 dst thresh maxValue]
    [rcvThreshold/toZeroInv img1 dst thresh maxValue]
]


loadImage: does [
    isFile: false
    canvas/image/rgb: black
    canvas/size: 0x0
    tmp: request-file
    if not none? tmp [
        fileName: to string! to-local-file tmp
        win/text: fileName
        img1: rcvLoadImage tmp
        dst:  rcvCloneImage img1
        ; update faces
        if img1/size/x >= defSize/x [
            win/size/x: img1/size/x + 20
            win/size/y: img1/size/y + 260
        ] 
        either (img1/size/x = img1/size/y) [bb/size: 60x60] [bb/size: 80x60]
        canvas/size: img1/size
        canvas/image/size: canvas/size  
        canvas/offset/x: (win/size/x - img1/size/x) / 2
        bb/image: img1
        canvas/image: dst
        isFile: true
        do op: ops/1
        r1/data: true
        r2/data: false
        r3/data: false
        r4/data: false
        r5/data: false
    ]
]

; ***************** Test Program ****************************
view win: layout [
        title "Basic Thresholding Operations"
        origin margins space margins
        button 60 "Load"        [loadImage] 
                    
        button 60 "Quit"        [rcvReleaseImage img1 
                                rcvReleaseImage dst Quit]
        return
        bb: base 80x60 img1
        return
        r1: radio "Binary"          [sl2/visible?: true  do op: ops/1]
        r2: radio "Binary Inverted" [sl2/visible?: true  do op: ops/2]    
        r3: radio "Truncate"        [sl2/visible?: false do op: ops/3]   
        r4: radio "To 0"            [sl2/visible?: false do op: ops/4]
        r5: radio "To 0 Inverted"   [sl2/visible?: false do op: ops/5]       
        return
        text 100 "Threshold value"  
        sl1: slider 340 [thresh: to integer! face/data * 255 f1/data: form thresh do op]
        f1: field 50 "128" 
        return
        text 100 "Maximal value" 
        sl2: slider 340 [maxValue: to integer! face/data * 255 f2/data: form maxValue do op]
        f2: field 50 "255"
        return
        canvas: base 512x512 dst    
        do [r1/data: true sl1/data: 0.5 sl2/data: 1.0]
]

And the result

RedCV: How to fade an image

Image processing software such as GIMP includes various techniques to fade an image. Here, the idea is to use pixel intensity of the image to fade in or out the image. Red images are 8-bit. So, each ARGB channel can have a value from 0 to 255, and these values represent the intensity of the image. 

Power function can help us for a faster image rendering. In mathematics, a power function is a function of the form f(x)=x^p, where p is constant and x is a variable. When p= 0, f(x) is the constant function f(x) = 1, and when p = 1, f(x) is the identity function f(x) = x. This mean that if we use a p value between 0 and 1,we can modify each RGB value to control image intensity and fading effects. This can be done since rcvPow function does accept float or integer values for exponent.

Code sample

    Title:   "Simple Fading Sample"
    Author:  "Francois Jouen"
    File:    %fading.red
    Needs:   'View
]
#include %../../libs/redcv.red ; for redCV functions
margins: 10x10
defSize: 512x512
img1: rcvCreateImage defSize
dst:  rcvCreateImage defSize
isFile: false
delta: 0.0

loadImage: does [
    isFile: false
    sl/data: 0%
    delta: 0.0
    tmp: request-file
    if not none? tmp [
        img1: rcvLoadImage tmp
        dst:  rcvCloneImage img1
        canvas/image: dst
        isFile: true
    ]
]

; ***************** Test Program ****************************
view win: layout [
        title "Simple Contrast Test"
        origin margins space margins
        button 60 "Load"        [loadImage]                     
        sl: slider 240 [if isFile [
                            delta: 1.0 - to float! sl/data * 1
                            vf/data: form delta 
                            rcvPow img1 dst delta   
                        ]                
        ]   
        vf: field 50 "1.0"                              
        button 50 "Quit"        [rcvReleaseImage img1 rcvReleaseImage dst Quit]
        return 
        canvas: base 512x512 dst
        do [sl/data: 0%]    
]

And the images

RedCV: How Red stores pixel values in memory

Each ARGB pixel component is represented by a byte, i.e. a 8-bit value. The byte! datatype's purpose is to represent unsigned integers in the 0-255 range.

Many image processing libraries use a byte pointer to access ARGB components of a pixel, but Red proposes an optimized approach which uses an integer to store ARGB values in a single value.

Since the memory size of an integer is 32 bits, is really easy to store 4 bytes (8-bit) value within an integer. Consequently a int-ptr! will be used to access pixel value.

Now to access to ARGB values stored in the integer, Red applies right shift operator, both unsigned right shift (>>>) and signed right shift (>>).  

a: pix1/value >>> 24                ; byte 1 [0-255]
r: pix1/value and 00FF0000h >> 16   ; byte 2 [0-255]
g: pix1/value and FF00h >> 8        ; byte 3 [0-255]
b: pix1/value and FFh               ; byte 4 [0-255]




Then to write back  a modified pixel, Red uses signed left shift operator (<<) to make a signed integer with 4 bytes.

pixD/value: (a << 24) OR (r << 16) OR (g << 8) OR b

This approach using an integer to store pixels is very smart and very efficient. The access to any pixel is really easy with a simple int-ptr! You just need to know to base address of the image in memory and use pointer arithmetic to access the pixel you want.  Of course, to be efficient this implementation requires Red/System DSL which is faster than Red in image processing. Fortunately, Red/System objects give us access to useful macros and methods. With routine! it is simple to call Red/System code inside a pure Red/Code.

Reading image

Let's start by an example to illustrate how we can copy an image with pointers.

_rcvCopy: routine [
    src1 [image!]
    dst  [image!]
    /local
        pix1 [int-ptr!]
        pixD [int-ptr!]
        handle1 handleD h w x y
][
    handle1: 0
    handleD: 0
    pix1: image/acquire-buffer src1 :handle1
    pixD: image/acquire-buffer dst :handleD
    w: IMAGE_WIDTH(src1/size)
    h: IMAGE_HEIGHT(src1/size)
    x: 0
    y: 0
    while [y < h] [
       while [x < w][
            pixD/value: pix1/value
            x: x + 1
            pix1: pix1 + 1
            pixD: pixD + 1
       ]
       x: 0
       y: y + 1
    ]
    image/release-buffer src1 handle1 no
    image/release-buffer dst handleD yes
]

This RedCV routine has 2 parameters: the source and the destination images and uses 2 int-ptr to access to the source and destination memory address. 

There is no control on images and you need to ensure that both images have the same size.

The image/acquire-buffer method gives access to the memory base address for each image and returns a pointer. Since Red uses a classical row-major order for accessing pixel, it's really easy to read the whole image. First of all, get the image size with IMAGE_WIDTH and IMAGE_HEIGHT macros and just call 2 imbricated while instructions for reading image line by line and column by column.

pixD/value: pix1/value affectation simply copies source image value to destination image and then you just need to increase both source and destination images pointers for accessing the next pixel.

Modifying image

Of course you can modified pixel values as you want. The next (incomplete) example illustrates how create a grayscale image.

while [y < h] [
       while [x < w][
            a: pix1/value >>> 24
            r: pix1/value and 00FF0000h >> 16 
            g: pix1/value and FF00h >> 8 
            b: pix1/value and FFh 
            s: (r + g + b) / 3 
            pixD/value: (a << 24) OR (s << 16 ) OR (s << 8) OR s]
            x: x + 1
            pix1: pix1 + 1
            pixD: pixD + 1
        ]
    x: 0
    y: y + 1
]

Here, we read argb values and then calculate the mean value for rbg values with  s: (r + g + b) / 3. Then we udpate destination image pixel value:

pixD/value: (a << 24) OR (s << 16 ) OR (s << 8) OR s] ;RGB2Gray average

Red is fantastic!