Using Python OpenCV, How would you extract an image area inside a particular color bounding box?

Given a photo that I have manually drawn a colored bounding box,
I want to copy / crop the image content, to keep the content inside the bounding box.

The goal is to detect that color bounding box, and then use that to tell the script where to copy/crop.

I have experimented with contours, but it appears I need extra steps.

Perhaps a way to:

• detect the bounded area


• find the smallest area (the box lines could be of variable thickness, so I need the inside bounding area - the bounding eventually will be a colored poster board cutout box in the physical world)


• the script creates a mask for that area


• grab the image

There could be a better way;
What is the best way to go about this?

And what Python OpenCV methods would I use?

Based on my current experimental code (I was exploring getting the area by contour size, but I think I need better contour code):

import numpy as np
import cv2

image_dir = "/Users/admin/Documents/dir/dir2/"

im = cv2.imread(image_dir+'test_image_bounded.png')

imgray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)
ret,thresh = cv2.threshold(imgray,176,190,43)
#ret,thresh = cv2.threshold(imgray,127,255,0)
contours, hierarchy =         cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)

areaArray = []
count = 1

for i, c in enumerate(contours):
    area = cv2.contourArea(c)

    areaArray.append(area)

#first sort the array by area
sorteddata = sorted(zip(areaArray, contours), key=lambda x: x[0], reverse=True)

#find the nth largest contour [n-1][1], in this case 2
largestcontour = sorteddata[0][2]

#draw it
x, y, w, h = cv2.boundingRect(largestcontour)

cv2.drawContours(im, largestcontour, -1, (255, 0, 0), 2)
cv2.rectangle(im, (x, y), (x+w, y+h), (0,255,0), 2)
cv2.imwrite(image_dir+'output.jpg', im)

edit --------------------------------

I've managed to get some pretty good results by color detection, morphology, and grabbing the second largest threshold

Here is some relevant code:

green_MIN = np.array([45, 25, 25],np.uint8)
green_MAX = np.array([55, 255, 255],np.uint8)

hsv_img = cv2.cvtColor(img,cv2.COLOR_BGR2HSV)

frame_threshed = cv2.inRange(hsv_img, green_MIN, green_MAX)


#image = cv2.imread('...') # Load your image in here
# Your code to threshold
#image = cv2.adaptiveThreshold(image, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 45, 0)    

# Perform morphology
se = np.ones((20,20), dtype='uint8')
image_close = cv2.morphologyEx(frame_threshed, cv2.MORPH_CLOSE, se)

The HSV values are a pain; I'd like to automate that part.

This was helpful to get value:
https://achuwilson.wordpress.com/2012/02/14/hsv-pixel-values-in-opencv/

Answer

My quick suggestion would be to:

1) Filter by color since the rectangle is green. Green might also exist in the image itself, but this will reduce the false positives.

2) Detect lines that form the rectangle.

Now this might be done in various ways. A more general way to do it would be to use the Hough transform. I'm not aware of an implementation that would directly search for rectangles, although you could implement it as well. The HoughLinesP function will find lines and you can choose those that form a rectangle.

However, in your application, you might have quite strict assumptions that will make this problem much easier. If the bounding box is never rotated, you can simply iterate over the rows and columns to find those that have the largest amount of pixels of the color you are looking for. That can be extended to look for contiguous pixels to find line segments, but might not even be necessary.