How to dynamically delete the background of an image with OpenCV plus Unity

Thing you want to do

I wanted to make an app like Rakugaki AR, so this time I used OpenCV to delete the background of the image imported into Unity (transparency). )To do.
The source code is uploaded below.
https://github.com/AzetaTakuya/MakeImageBackgroundTransparentUsingOpenCV

result

Like this

From the left, the background of the image could be made transparent from the process of [Original image-> Grayscale-> Binarization-> Masking-> Delete background].

Implementation

environment

• Windows10 Home
• Unity 2019.4.8f1
• OpenCV plus Unity 1.7.1

Try OpenCV plus Unity

First, create a Unity project and import OpenCV plus Unity from the AssetStore.

When the import is complete, an error [ error CS0227: Unsafe code may only appear if compiling with / unsafe. Enable "Allow'unsafe' code" in Player Settings to fix this error. </Font>] I think that will come out.  Since the unsafe code is used, check [File-> BuildSettings-> PlayerSettings-> Player-> OtherSettings-> Allow'unsafe' Code]. Now that the error is gone, let's take a look at the OpenCVplus Unity demo. This time, I want to extract the outline of the image, so open the demo [Assets-> OpenCV + Unity-> Demo-> Identifiy_Contours_by_Shape-> ContoursByShapeScene.scene]. The execution result looks like this.  Black and white images with color and text.  Make the background of the image transparent based on this demo.

Remove image background

Import image

Import the image file into Unity. Since it cannot be read as it is, check Read / Write Enabled.

code

From the original image (m_texture)
[Original image-> Grayscale image-> Contour extraction image-> Masked image-> Background transparent image] is displayed.

namespace OpenCvSharp.Demo
{
    using UnityEngine;
    using System.Collections;
    using OpenCvSharp;
    using UnityEngine.UI;
    using System.Threading.Tasks;
    using System.Collections.Generic;

    public class MakeImageBackgroundTransparentUsingOpenCV : MonoBehaviour
    {
        #region public members
        public Texture2D m_texture;

        public RawImage m_image_origin;
        public RawImage m_image_gray;
        public RawImage m_Image_binarization;
        public RawImage m_image_mask;
        public RawImage m_image_backgroundTransparent;

        public double v_thresh = 180;
        public double v_maxval = 255;
        #endregion

        private void Start()
        {
            #region load texture
            Mat origin = Unity.TextureToMat(this.m_texture);
            m_image_origin.texture = Unity.MatToTexture(origin);
            #endregion

            #region  Gray scale image
            Mat grayMat = new Mat();
            Cv2.CvtColor(origin, grayMat, ColorConversionCodes.BGR2GRAY);
            m_image_gray.texture = Unity.MatToTexture(grayMat);
            #endregion

            #region Find Edge
            Mat thresh = new Mat();
            Cv2.Threshold(grayMat, thresh, v_thresh, v_maxval, ThresholdTypes.BinaryInv);
            m_Image_binarization.texture = Unity.MatToTexture(thresh);
            #endregion

            #region Create Mask
            Mat Mask = Unity.TextureToMat(Unity.MatToTexture(grayMat));
            Point[][] contours; HierarchyIndex[] hierarchy;
            Cv2.FindContours(thresh, out contours, out hierarchy, RetrievalModes.Tree, ContourApproximationModes.ApproxNone, null);
            for(int i = 0; i < contours.Length; i++)
            {
                Cv2.DrawContours(Mask, new Point[][] { contours[i] }, 0, new Scalar(0, 0, 0), -1);
            }
            Mask = Mask.CvtColor(ColorConversionCodes.BGR2GRAY);
            Cv2.Threshold(Mask, Mask, v_thresh, v_maxval, ThresholdTypes.Binary);
            m_image_mask.texture = Unity.MatToTexture(Mask);
            #endregion

            #region TransparentBackground
            Mat transparent = origin.CvtColor(ColorConversionCodes.BGR2BGRA);
            unsafe
            {
                byte* b_transparent = transparent.DataPointer;
                byte* b_mask = Mask.DataPointer;
                float pixelCount = transparent.Height * transparent.Width;

                for (int i = 0; i < pixelCount; i++)
                {
                    if (b_mask[0] == 255)
                    {
                        b_transparent[0] = 0;
                        b_transparent[1] = 0;
                        b_transparent[2] = 0;
                        b_transparent[3] = 0;
                    }
                    b_transparent = b_transparent + 4;
                    b_mask = b_mask + 1;
                }
            }
            m_image_backgroundTransparent.texture = Unity.MatToTexture(transparent);
            #endregion
        }

    }
}

scene

Prepare a GameObject and 5 RawImages to attach the above script.
MakeImageBackgroundTransparentUsingOpen Attach the imported image to CV and RawImage.

Run

When executed, [Original image-> Grayscale image-> Outline extraction image-> Masked image-> Background transparent image] is displayed.

• If it doesn’t work, try playing with the value of v_thresh.

Summary

――It seems that you can use it for AR apps.
–It is necessary to be able to handle images other than those with a white background.

reference

-Coloring book AR made with OpenCV / ARCore / Unity:
Takashi Yoshinaga