fischertechnik community forum

Hi,

Before I make any attempt to do so: would it be possible to install OpenCV on the controller to use with Python?

Thanks

hvn

Hi hvn,

hvn hat geschrieben: Before I make any attempt to do so: would it be possible to install OpenCV on the controller to use with Python?

Yes, that is possible. The easiest way is to use the community firmware - OpenCV (version 2.4.13 at the moment) is already included there

For the original firmware, you'll probably have to cross-compile and install it yourself (along with python and all dependencies), and you might run into space issues on the internal flash. If you want to go that way, Torsten Stuehn (the author of ftrobopy) is probably the best person to ask since he has done quite a bit with python on the original firmware.

Best regards,

Richard

PS: I've just had a look at opencv.org, and OpenCV 3.3 (released in August) has some interesting new features - especially the ability to run (pre-trained) deep neural networks for image recognition. Guess it's time for an upgrade to OpenCV 3.3 for the CFW

Hi Richard,

That great news. This means I can already make a start, and on the original firmware would be a very nice addition as well, so I'll ask Torsten. When do you think OpenCV 3.3 will be included?

Regards,

hvn

hvn hat geschrieben: When do you think OpenCV 3.3 will be included?

Depends a bit on when I find the time, and on how many extra dependencies I have to integrate/upgrade in Buildroot for OpenCV 3.3 - maybe in November or December.

Of course, if someone else wants to beat me to it: Pull requests are always welcome

Of course, if someone else wants to beat me to it: Pull requests are always welcome

Since OpenCV is present, I'm fine waiting for it

Hi Richard,

Is there an example on how to use opencv with cfw? The usual way using

import cv2
import numpy as np
image = cv2.imread(filename)

gives errors on cv2, numpy and imread: ImportError on both cv2 and numpy and NameError: global name 'imread' is not defined (after removing cv2). So can you tell me where I go wrong?

Thanks

hvn

Hi hvn,

hvn hat geschrieben: Is there an example on how to use opencv with cfw? The usual way using

import cv2
import numpy as np
image = cv2.imread(filename)

gives errors on cv2, numpy and imread: ImportError on both cv2 and numpy and NameError: global name 'imread' is not defined (after removing cv2).

Strange. The exact same commands work just fine on my TXT with the current CFW in an interactive python shell:
How did you execute your code when you had the error (interactive or as part of a script, which user, any special settings for PYTHONPATH, ...)?

I'm using OpenCV from my code, and I just find that the errors occur using Python 2.7 and not Python 3. And I'm still having the habit to use my Python 2.7 script so I guess I better remove that. But I've used OpenCV with Python 2.7 as well in the past. So is this for the TXT indeed only to be used with Python 3 ?

hvn hat geschrieben:I'm using OpenCV from my code, and I just find that the errors occur using Python 2.7 and not Python 3.

OK, that explains it - CFW has Python 3 preinstalled, and of course all extra packages are for the preinstalled Python 3 as well. And a (presumably self-compiled) Python 2.7 will not find the Python OpenCV and numpy packages (and I'm not sure if these will work, either, even if you adjust the PYTHONPATH accordingly).

hvn hat geschrieben:So is this for the TXT indeed only to be used with Python 3 ?

Yes, CFW uses Python 3 by design.

And given the fact that Python 3 came out nearly 10 years ago, Python 2.7 is clearly labeled as "legacy", and supporting two Python versions in CFW is a lot of work for relatively little gain, I'm not much inclined to support Python 2.7 in the CFW...

Now I really get puzzled. In my code (a part which has nothing to with OpenCV) I use:

while pic == None:
pic = txt.getCameraFrame()

Where it used to work, apart from getting quite a few 0 byte images, now the code hangs on this line. The final error I get after ^C is "struct.error: ubyte format requires 0 <= number <= 255" with a traceback to ../python3.5/threading.py line 914 in _bootstrap_inner and ../python3.5/site-packages/ftrobopy.py line 1616.

@Richard,

Can you tell me how much of OpenCV is actually implemented in the cfw? I tried to create a gray image using a standard OpenCV instruction which was unknown during execution. Also, with all errors it shows a path inside /home/Richard/..... missing something.

hvn

hvn hat geschrieben: Can you tell me how much of OpenCV is actually implemented in the cfw?

Almost everything that is included in Buildroot. The only packages that are not included are some video i/o packages that do not make sense for CFW (e.g. video i/o via gstreamer).

And btw. we're actually using version 3.2.0 at the moment, not 2.4.13 as I said before (Buildroot provides both opencv2 and opencv3, and I looked at the wrong package )

hvn hat geschrieben: I tried to create a gray image using a standard OpenCV instruction which was unknown during execution.

Can you paste your code, the commands used to run your code, and the error message?

And which version of the CFW are you running? You can see the version in the "About" app by pressing the menu icon in the upper left corner of the window and choosing the "Versions" menu entry - the interesting value in this list is the "Firmware" entry.

All errors I could get:

1: using txt.getCameraFrame()

Exception in thread Thread-1:
Traceback (most recent call last):
File "/wheel/SSDSpace/devel/ftcommunity-TXT-0.9.3/output/target/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
File "/wheel/SSDSpace/devel/ftcommunity-TXT-0.9.3/output/target/usr/lib/python3.5/site-packages/ftrobopy.py", line 1616, in run
struct.error: ubyte format requires 0 <= number <= 255

^CTraceback (most recent call last):
File "Mirft2.py", line 106, in <module>
camera_action()
File "Mirft2.py", line 30, in camera_action
pic = txt.getCameraFrame()
KeyboardInterrupt

2:

Using OpenCV:

#opencv
image = cv2.imread(filename)
red = image[0, 0, 100]
print ("red = ", red)

error:

File "Mirft2.py", line 40, in camera_action
red = image[0, 0, 100]
TypeError: 'NoneType' object is not subscriptable

3: Using OpenCV;

gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.imwrite('opencv-image.png',gray_image)

error:

OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /wheel/SSDSpace/devel/ftcommunity-TXT-0.9.3/output/build/opencv3-3.2.0/modules/imgproc/src/color.cpp, line 9748
Traceback (most recent call last):
File "Mirft2.py", line 106, in <module>
camera_action()
File "Mirft2.py", line 42, in camera_action
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.error: /wheel/SSDSpace/devel/ftcommunity-TXT-0.9.3/output/build/opencv3-3.2.0/modules/imgproc/src/color.cpp:9748: error: (-215) scn == 3 || scn == 4 in function cvtColor

Firmware version: 0.9.3

I hope this helps in debugging or telling me what I do wrong.

Hi hvn,

Using OpenCV:

#opencv
image = cv2.imread(filename)
red = image[0, 0, 100]
print ("red = ", red)

error:

File "Mirft2.py", line 40, in camera_action
red = image[0, 0, 100]
TypeError: 'NoneType' object is not subscriptable

Looks like cv2.imread() failed (maybe the image path is wrong?). In that case, it returns "None", which will cause the error you see when you subsequently try to access a component of nothing

See e.g. https://docs.opencv.org/3.0-beta/doc/py ... splay.html for help on cv2.imread().

Ok, actually the base of the problem is getCameraFrame(). The path is correct, I just made a slight mistake commenting out the creation of filename. So because getCameraFrame doesn't work anymore, this code:

pic = None
while pic == None:
pic = txt.getCameraFrame()
print("pic")

gives this output:

<snip of long output of...>
pic
pic
pic
pic
pic
pic
pic
pic^CTraceback (most recent call last):
File "Mirft2.py", line 106, in <module>
camera_action()
File "Mirft2.py", line 31, in camera_action
print("pic")
KeyboardInterrupt

Since getCameraFrame() doesn't work, the image remains empty hence the file remains empty. Because of that, this is the new output:

filename = txtimage-2017-11-01-15-42-5.jpg
Traceback (most recent call last):
File "Mirft2.py", line 106, in <module>
camera_action()
Exception in thread Thread-1:
Traceback (most recent call last):
File "/wheel/SSDSpace/devel/ftcommunity-TXT-0.9.3/output/target/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
File "/wheel/SSDSpace/devel/ftcommunity-TXT-0.9.3/output/target/usr/lib/python3.5/site-packages/ftrobopy.py", line 1616, in run
struct.error: ubyte format requires 0 <= number <= 255

File "Mirft2.py", line 40, in camera_action
red = image[0, 0, 100]
TypeError: 'NoneType' object is not subscriptable

Does the camera work? Yes, because when I use the code (not OpenCV, just getCameraFrame) on the default firmware, very nice images are taken. Which brings another issue forward: since installing the new cfw on this TXT, the motors don't do anything anymore. Which brings another fact to light: just checking the firmware versions of this TXT and the other (driving robot): 0.9.3 vs. 0.9.2-3. But that's for likely another thread....

Hi hvn,

I played around with ROS, opencv and cameras on my TXT running CFW.

What worked quite nice is this (beside the fact that txt.cameraOnline() throws an error and needs to be commented out):
txt=ftrobopy.ftrobopy('auto')

def camera_thread():
print("camera_thread")
image_pub = rospy.Publisher("/camera/image_raw/compressed", CompressedImage, queue_size=2)
txt.startCameraOnline()
time.sleep(2.5)
# if (not txt.cameraOnline()):
# return
rate = rospy.Rate(15)
while not rospy.is_shutdown():
pic = txt.getCameraFrame()
if (pic):
print("camera_thread got CameraFrame")
np_arr = np.frombuffer(bytearray(pic), dtype='uint8')
image_np = cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
cv2.imshow('img', image_np)
cv2.waitKey(2)
msg = CompressedImage()
msg.header.stamp = rospy.Time.now()
msg.format = "jpeg"
msg.data = np_arr.tostring()
image_pub.publish(msg)
rate.sleep()

I am not sure if it is good style (I am a Python newbie) but it works, you can even see the camera capture on the display.
Please note the camera thread refused to start every other time with a networking error (even if I run it locally):
Network error [Errno 104] Connection reset by peer

So you have to start it twice.
And you have to start "FT-GUI" to make the camera work.

-Thomas

Hi thomass,

Nice to see your efforts on CFW and ROS. I'm using Python and ROS with a different experiment involving RaspberryPi and may start doing it here as well.
On the error of the camera thread: yes, that's my experience as well. Either connection to the TXT is aborted, the camera thread is cut or it just hangs and I have to kill the entire process.

hvn

Hi hvn,

Currently I am switching back to C, or C++, respectively.
Using v4l2 works quite well, grabbing images and publishing them compressed over WiFi gives over 15fps, without decoding on TXT though.
With cv::imdecode and cv::imshow is is in the range of 7fps, I think. And umcompressed transmission < 5fps.

Currently I use my own code as ROS-node but ROS usb_cam package should work as well, I will try later today.

-Thomas