LEGO Mindstorms Arm with Computer Vision

This is my entry to the LEGO Ideas Robot Inventor competition -- an arm, connected to a computer running computer vision software. The programs demonstrated are written in the scratch-based LEGO Mindstorms language and in python running openCV and nxt-python, respectively.
Missions:
(00:00) Intro
(00:40) Remote control
(01:57) Computer vision -- Colour tracking
(04:13) Motion tracking, Sentry

[@solomonjenkins9505]

very cool

[@SpeedsterGames87]

I am in need of instructions

[@user-rf1km6cr7y]

Hi ! so impressed by your awesome project :) is there anyway I can read your program or some of the codes? Thanks!!

[@eddieyoung5613]

It's all a bit messy because I didn't intend to share the code originally, but here it is if you can make sense of it: www.dropbox.com/sh/6u8hfcv66mp44i2/AACKEraieFhuB3Ckeazaj0Nga?dl=0
The python program needs a couple packages to compile and nxt-python can be tricky to install, I ran the program in terminal on a macbook air with the built in camera working for OpenCV and two NXT bricks connected to the computer with USB (the program won't run at all without the bricks connected, if you change use_turntable=False on line 74, only one NXT will be needed)
I've also included the EV3 program (created in the new scratch software -- Mindstorms Home)

[@aldocona6010]

@@eddieyoung5613 grateful to the one above

[@LEGO.1]

Was there an aspect/quality of the "off brand" table tennis ball that made it a better grab target than an official LEGO-produced ball? The size? The friction? The weight?

[@eddieyoung5613]

Only the fact that I own one and don’t own the LEGO version 0:)

[@eddieyoung5613]

The LEGO originals would actually probably work better, because LEGO colours are of much better quality and the table-tennis ball is slightly translucent

[@williamhuang5329]

Hanzhen harmonic drive gear , robot joint gear, over 30 years experience

[@nak_attak]

How do I put openCV on robot inventor app on windows?

[@eddieyoung5613]

You don’t. What you see in the video is opencv running on the computer. The computer only sends instructions to the robot based on its calculations.

[@d1cn]

Hi, great stuff! Do you have Python code for EV3? I want to buy the guide on rebrickable, but it’s only for Robot Inventor and I want to try EV3 version first :) Thx!

[@eddieyoung5613]

Hey, thank you! The program which connects to my computer webcam in the video runs on NXT, see here. The EV3 only runs the simpler IR sensor version. The linked folder contains all the files used for this video (they are messy though, didn’t intend to share them back then) www.dropbox.com/sh/6u8hfcv66mp44i2/AACKEraieFhuB3Ckeazaj0Nga?dl=0

[@d1cn]

@@eddieyoung5613 Ah, clear. I was thinking that there is full a featured EV3 version with CV after watching the video :) Then I’ll try to change the code for EV3. Thanks, I’ve already downloaded these files from some other reply :)

[@eddieyoung5613]

@@d1cn Right, no, the full version I ended up being happy with is the Robot Inventor in my other video. Good luck making it work for EV3. If there’s an equivalent of nxt-python for EV3 (there wasn’t 3 years ago), it should be easy enough.

[@claudiabeltran2018]

Hello Eddie , we have a project with an FLL team , we need help with a prototype very similar like your arm , is there a way to contact you to get help with programming please ? Thanks In advance

[@eddieyoung5613]

Here are some messy materials for the robot in this video: www.dropbox.com/sh/6u8hfcv66mp44i2/AACKEraieFhuB3Ckeazaj0Nga?dl=0
and here is a (paid) project of mine with a cleaner program and pdf documentation: rebrickable.com/mocs/MOC-86914/Eddie_Young/computer-vision-inventor/#details
You can email me at [e-mail]

[@claudiabeltran2018]

@@eddieyoung5613 thank you so much , I will take a look and get back to you if we need more info 👍🏻

[@achilleaskarteris1874]

I am interested in the project but I cant find a way to read the program for the ev3. Can anybody help?

[@eddieyoung5613]

I shared the project files with some other commenters here, the EV3 program is written in the scratch version for mac, if you can’t open that, I’m afraid I can’t help: www.dropbox.com/sh/6u8hfcv66mp44i2/AACKEraieFhuB3Ckeazaj0Nga?dl=0

[@achilleaskarteris1874]

@@eddieyoung5613 Yes I saw it. The problem is I dont have a Mac so I will see what to do. Anyway thank you awesome work!

[@raphaelramesar2109]

if the code is run on the ev3, how does it know to display on the computer screen? When I try to do something similar, it gives me a display error. Any advice?

[@eddieyoung5613]

The one connected to the computer is an NXT, not EV3. But the way you would do it is create a master program on the computer and a slave program on the robot and have them communicate via BT or USB. This is what I did with the NXT (using nxt-python) and also Robot Inventor (which has a built-in function for communication) in my other video.

[@raphaelramesar2109]

@@eddieyoung5613 Thanks for the info and reply.

[@trumporange]

can u pls tell me, which motor is connected to which port thanks

[@eddieyoung5613]

A-upper joint, B-lower joint, C-claw, D-turntable… Cheers!

[@eldargashi3918]

do you have a tutorial how to make or somwhere i can look how to make the robot please answer me fast

[@eddieyoung5613]

I've updated the design to Mindstorms Robot Inventor and created building instructions with documentation for the program. rebrickable.com/mocs/MOC-86914/Eddie_Young/computer-vision-inventor/#details

[@eddieyoung5613]

You can also read the program of the NXT version seen here (though I never got around to cleaning it up for sharing): www.dropbox.com/sh/6u8hfcv66mp44i2/AACKEraieFhuB3Ckeazaj0Nga?dl=0

[@xalton94]

Is it possible to make this arm with the new lego mindstorm 51515 ?

[@eddieyoung5613]

The renders shown in the video are built (digitally) only from pieces from that set, so yes, but that version hasn't been built physically so I don't know whether it would work. (I made the renders to illustrate my entry to the LEGO Ideas contest.)

[@xalton94]

​@@eddieyoung5613 Hi thanks for your answer. I tried to reproduce this kind of arm but I don't see how do you connect the motor that will elevate the arm. With the previous kit, you have 2 big gears but with the Mindstorm 51515, we have only 1. I don't know if it is possible for you to share maybe another angle of view? I want to make a youtube video on my french Tech channel about this LEGO Mindstorm kit and of course, I will credit your work. Thanks

[@eddieyoung5613]

@@xalton94 The following link should take you to a DropBox download page, where you can get the Studio .io file. Note that I didn't intend to share the file, so there are no steps and there may be mistakes, too.
www.dropbox.com/s/1m1b01enrnnq062/Robot%20Inventor%20Arm.io?dl=0

[@xalton94]

@@eddieyoung5613 Thank you very much. I will let you know and share with you my result.

[@the_guy_next_door_31]

@@eddieyoung5613 What software do you use to make such realistic renders of lego parts ?