วีดีโอ

I appreciate your curiosity! I totally get where you're coming from - not every day, you see a language model in action alongside robots and images. But I assure you, what you're witnessing is powered by a sophisticated language model. This system has been trained to understand and generate human-like text based on a wide range of input data. The chat interface serves as a way for us to interact and communicate with the model, just like you and I are doing. It's a testament to the versatility of this Technology - it's not just about hardcoded responses; it can dynamically generate contextually relevant and creative output. So, while it might seem a bit unconventional initially, this chat interface adds a layer of flexibility and adaptability to the overall system. Feel free to ask more questions or delve deeper into how it all works - I'm here to provide you with a badass and polite explanation! 😄

@@kanishknavale was that written by an ai?

@@kanishknavale where is the actual benefit of this? It doesn't seem like the chat interface adds any value. Wouldn't it be much more efficient to train a new model from scratch that just recognizes objects and categorizes them by weight and rigidity? Such a system could be trained by simply letting the robot move different objects around all day, that way objects can be observed from all different angles and data about weight and rigidity can be interpolated from motor feedback. Heck, the AI could even even learn how to grab different things to avoid damage, and it could do all of that all by itself. It seems like "AI" is just a common buzzword used by sereact to market products that have existed for decades as something new.

@@robotboi763 The chat interface here allows the user to interact with our system via text prompt. Since our system is driven by a LLM, the responses you observed here in the video are self-generated by the LLM without any hardcoding. The chat interface for this demo here is just for showcasing the capability of our PickGPT system and demonstrating how human can interact with the robot via our product to easily execute the pick-and-place as well as other textual/visual-related tasks. In a production settings, we can easily get rid of the chat interface and instead provide the LLM with few pre-determined prompts to align the behaviour of the robot with the true intention of the customers. Nevertheless, including the chat interface will allow the users to observe the behaviour and responses of PickGPT, as well as allow the system to learn and self-improve from previous conversations with the users. Some other scenarios where chat interface will be useful will be human-machine collaboration tasks where programming is tedious and one wants to control the robot without having to input detailed instructions. In short, the GPT engine here does not necessarily implies the chatting ability, but more of a reasoning ability of our AI-powered products basing purely on textual prompts rather than deliberated engineering or programming efforts.

Absolutely, I understand your point. While the GPT aspect isn't solely about chat, the chat interface serves as a practical way to access and interact with the wealth of knowledge within the language model. It adds a dynamic and user-friendly layer to the system's capabilities, making it more versatile and engaging.

The GPT part here is in fact deals more with the reasoning ability of PickGPT via textual input. The chatting aspect is simply just one of the capability of our LLM, which is actually very beneficial for use-cases where users want to interact with the system in an online and real-time manner or update/retrieve the system's knowledge about the enviroment.

It can detect bubble wrap and other packaging materials in the bin for a prompt like "Give me something to pack the lightbulb". Packaging is a different domain involving robots and mechanical manipulators needing to be equipped with force feedback that might not break the bulb. We are on it, and soon you might see it too.

Same

Image recognition is a small part of it. GPT stands for Generative Pre-Trained Transformer. It was built with the idea it could be adapted to any environment, you just have to figure out how to prompt for the things you want. Using a combination of tools, GPTs can do just about anything.

​@@technoe02So which part does GPT in this example? The Image recognition detects where the object is. Then the robot arm then has to know where to go. Is it used here? What does the GPT Part do. I My guess is that they used. Is it running with ChatGPT or are they using their own LLM?

The idea is that GPT is doing the "reasoning" - it knows you shouldn't pack eggs on top of bread, even if the weight class might say it should.

With a true multi-modal AI, it could achieve unusual feats like actually understanding the difference in random/exotic foods like packaged german sweet breads compared to boiled sweets - unfortunately this looks like YOLO mixed with ChatGPT or LLaMA, which means it will be getting signals like "Candy" rather than specific details - they'll never get the image recognition side smart enough to give the language/reasoning side what it really needs to work with. Next gen AI should cope, this gen won't. This is a toy/demo that will still need excessive fine-tuning for edge cases, enough that you might as well just build the logic tree for your stock.

Thank you! Stay tuned. More to come 🚀

totally agree