FTC CV Tutorial

Introduction

Hello, I am Zhenkai from Team Wolf Corp (#12525). In FTC autonomous period, computer vision allows your robot to identify and distinguish scoring elements and make the optimal move. In this tutorial, I will cover the basics of EasyOpenCV and how to identify SkyStones from the 2019-2020 season. This tutorial assumes basic knowledge of FTC programming using Java.

Set up the Environment

OpenCV is a robust Computer Vision library that might be a little difficult to setup for FTC. We are going to use OpenFTC/EasyOpenCV here which allows us to create OpenCV Opmodes with ease. You can install it by following the official guide on GitHub or follow the guide here (same thing)

1. Open your project in Android Studio
2. Add the following within the repository block in build.common.gradle file

   repositories {
       // ...
       jcenter()
   }
       

3. Append the following to the build.gradle file in the TeamCode module

   dependencies {
       implementation 'org.openftc:easyopencv:1.3.2'
   }
       

4. Do a Gradle sync
5. Enable MTP mode on your Robot Controller phone and copy libOpenCvNative.so to the FIRST folder on the USB storage of the phone so that you can run OpenCV on the phone.
6. If you are interested, you can check out other official OpMode examples that make use of EasyOpenCV

Identifying Skystones

Let’s say the controller on the robot can capture at most two skystones at a time (as is the case for our team), we can determine if the blocks are regular stones or Skystones if we

• extract all yellow regions
• check if left/right side contains a significant amount of yellow; if yes, then there’s a regular stone, if not, then it’s a skystone.

HSV

Instead of RGB, we use HSV to represent colors. HSV stands for Hue (what kind of color), Saturation (intensity of the color), Value (brightness). It is commonly used because it’s easier to specify ranges and deal with different lighting conditions.

Find Skystone Location using an OpenCV Pipeline

The SkystoneDetector class is a OpenCvPipeline that will process the video stream from your phone canera. The processFrame() function does all the hardwork. Read the comments to find out what it does.

package org.wolfcorp.skystone;

import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.MatOfPoint;
import org.opencv.core.MatOfPoint2f;
import org.opencv.core.Rect;
import org.opencv.core.Scalar;
import org.opencv.imgproc.Imgproc;
import org.openftc.easyopencv.OpenCvPipeline;

import java.util.ArrayList;
import java.util.List;

public class SkystoneDetector extends OpenCvPipeline {
    enum SkystoneLocation {
        LEFT,
        RIGHT,
        NONE
    }

    private int width; // width of the image
    SkystoneLocation location;

    /**
     *
     * @param width The width of the image (check your camera)
     */
    public SkystoneDetector(int width) {
        this.width = width;
    }

    @Override
    public Mat processFrame(Mat input) {
        // "Mat" stands for matrix, which is basically the image that the detector will process
        // the input matrix is the image coming from the camera
        // the function will return a matrix to be drawn on your phone's screen

        // The detector detects regular stones. The camera fits two stones.
        // If it finds one regular stone then the other must be the skystone.
        // If both are regular stones, it returns NONE to tell the robot to keep looking

        // Make a working copy of the input matrix in HSV
        Mat mat = new Mat();
        Imgproc.cvtColor(input, mat, Imgproc.COLOR_RGB2HSV);

        // if something is wrong, we assume there's no skystone
        if (mat.empty()) {
            location = SkystoneLocation.NONE;
            return input;
        }

        // We create a HSV range for yellow to detect regular stones
        // NOTE: In OpenCV's implementation,
        // Hue values are half the real value
        Scalar lowHSV = new Scalar(20, 100, 100); // lower bound HSV for yellow
        Scalar highHSV = new Scalar(30, 255, 255); // higher bound HSV for yellow
        Mat thresh = new Mat();

        // We'll get a black and white image. The white regions represent the regular stones.
        // inRange(): thresh[i][j] = {255,255,255} if mat[i][i] is within the range
        Core.inRange(mat, lowHSV, highHSV, thresh);

        // Use Canny Edge Detection to find edges
        // you might have to tune the thresholds for hysteresis
        Mat edges = new Mat();
        Imgproc.Canny(thresh, edges, 100, 300);

        // https://docs.opencv.org/3.4/da/d0c/tutorial_bounding_rects_circles.html
        // Oftentimes the edges are disconnected. findContours connects these edges.
        // We then find the bounding rectangles of those contours
        List<MatOfPoint> contours = new ArrayList<>();
        Mat hierarchy = new Mat();
        Imgproc.findContours(edges, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);

        MatOfPoint2f[] contoursPoly  = new MatOfPoint2f[contours.size()];
        Rect[] boundRect = new Rect[contours.size()];
        for (int i = 0; i < contours.size(); i++) {
            contoursPoly[i] = new MatOfPoint2f();
            Imgproc.approxPolyDP(new MatOfPoint2f(contours.get(i).toArray()), contoursPoly[i], 3, true);
            boundRect[i] = Imgproc.boundingRect(new MatOfPoint(contoursPoly[i].toArray()));
        }

        // Iterate and check whether the bounding boxes
        // cover left and/or right side of the image
        double left_x = 0.25 * width;
        double right_x = 0.75 * width;
        boolean left = false; // true if regular stone found on the left side
        boolean right = false; // "" "" on the right side
        for (int i = 0; i != boundRect.length; i++) {
            if (boundRect[i].x < left_x)
                left = true;
            if (boundRect[i].x + boundRect[i].width > right_x)
                right = true;

            // draw red bounding rectangles on mat
            // the mat has been converted to HSV so we need to use HSV as well
            Imgproc.rectangle(mat, boundRect[i], new Scalar(0.5, 76.9, 89.8));
        }

        // if there is no yellow regions on a side
        // that side should be a Skystone
        if (!left) location = SkystoneLocation.LEFT;
        else if (!right) location = SkystoneLocation.RIGHT;
        // if both are true, then there's no Skystone in front.
        // since our team's camera can only detect two at a time
        // we will need to scan the next 2 stones
        else location = SkystoneLocation.NONE;

        return mat; // return the mat with rectangles drawn
    }

    public SkystoneLocation getLocation() {
        return this.location;
    }
}

Create an OpMode

Now that we have the detector, we can create an OpMode to use the camera for autonomous.

package org.wolfcorp.skystone;

import com.qualcomm.robotcore.eventloop.opmode.Autonomous;
import com.qualcomm.robotcore.eventloop.opmode.LinearOpMode;

import org.openftc.easyopencv.OpenCvCamera;
import org.openftc.easyopencv.OpenCvCameraFactory;
import org.openftc.easyopencv.OpenCvCameraRotation;
import org.openftc.easyopencv.OpenCvInternalCamera;

@Autonomous(name="Auto: SkyStone Detector")
public class AutoMode extends LinearOpMode {
    // Handle hardware stuff...

    int width = 320;
    int height = 240;
    // store as variable here so we can access the location
    SkystoneDetector detector = new SkystoneDetector(width);
    OpenCvCamera phoneCam;

    @Override
    public void runOpMode() {
        // robot logic...

        // https://github.com/OpenFTC/EasyOpenCV/blob/master/examples/src/main/java/org/openftc/easyopencv/examples/InternalCameraExample.java
        // Initialize the back-facing camera
        int cameraMonitorViewId = hardwareMap.appContext.getResources().getIdentifier("cameraMonitorViewId", "id", hardwareMap.appContext.getPackageName());
        phoneCam = OpenCvCameraFactory.getInstance().createInternalCamera(OpenCvInternalCamera.CameraDirection.BACK, cameraMonitorViewId);
        // Connect to the camera
        phoneCam.openCameraDevice();
        // Use the SkystoneDetector pipeline
        // processFrame() will be called to process the frame
        phoneCam.setPipeline(detector);
        // Remember to change the camera rotation
        phoneCam.startStreaming(width, height, OpenCvCameraRotation.SIDEWAYS_LEFT);

        //...

        SkystoneDetector.SkystoneLocation location = detector.getLocation();
        if (location != SkystoneDetector.SkystoneLocation.NONE) {
            // Move to the left / right
        } else {
            // Grab the skystone
        }

        // more robot logic...
    }

}