Google Releases AI AGENT BUILDER! 🤖 Worth The Wait?
ฝัง
- เผยแพร่เมื่อ 18 พ.ค. 2024
- Google announced Google Agent Builder through its Vertex AI product. Let's examine it and other announcements from Google Cloud Next.
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewberman.com
Need AI Consulting? ✅
forwardfuture.ai/
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
Full Video - • Google Cloud Next '24 ... - วิทยาศาสตร์และเทคโนโลยี
@19:25 you set your Tool Type to OpenAPI, and the endpoints in the instructions will work. I imagine it's 1 function per tool for modularity. It's more or less exactly the same setup as the Custom GPTs configuration.
Hey Matt.
When you select "function", you're responsible for writing the code and driving the conversation via API calls. The flow goes a little like this. You submit a message and you check the status and it tells you you need to run the function with these inputs. YOU then submit the output back to vertex via the API, where it uses the function output to generate the next message. You can implement the function wherever you're DRIVING the convo. Makes sense?
This is a very similar flow to the assistants API which I have a video about on my channel if you're interested
You’re so right Matt on that agents in the Mercedes. There’s this weird movement of bells and whistles, and what we can do rather than true functionality that is efficient. Especially when it’s not level five self driving, we still have to drive and less distraction equals more attention to the road and thus is safer driver.
@26:10 they're using the equivalent of Agent (preview) app in the Agent Builder service which as they word it "can answer questions from data, connect with business systems through tools"
thanks for the review . I was short on time today , perfect timing
Matt on it! Thanks for going over this. Nobody else is really going into it like you.!👍👏
I've been eagerly awaiting a local AI solution to sort through our family's extensive collection of decades-old photos. This AI should be capable of identifying and eliminating duplicate or blurry images, learning to recognize faces, and categorizing photos by individual. Moreover, it should have the ability to estimate the age range of each person depicted. Ideally, this solution would allow for easy exportation of organized data so that Stable Diffusion models could learn from it. Additionally, integrating facial recognition capabilities into home security cameras would streamline access control, eliminating the need for physical keys.
There's a program called Excire
This is something a program would take care of, not an AI. If you need a program with these specifics, you can write one with the help of Claude3 in 6-12h given availability of one or two components.
Digikam already does MOST of this. And all of this would be programmatic, not requiring an AI to run. Digikam can individualize people. All that's left over is a model that recognizes age, which Im sure exists, but you may want to write a program that iterates through all those images and adds subject age to the EXIF data.
Whats the main objective if i may ask
@@fintech1378Going to take a wild guess and say to have an organized, searchable collection of family photos which allows for anyone in the family to quickly find any photo.
and a search bar, so you can search whatever from your photos
Hey Matt, with all of these models and apps along with their updates, it would be nice to see a real time list of your favorites. For example you said Gemma wasn't to impressive but it may be now with its new update. Maybe like a place where you can rank the best models or frameworks and update the source as things change. For example crew Ai, autogen, Gemma,, Devin, Devin alternatives, and all the different llms. It be nice to see an up to date list of your rankings in each category. Or does something like this already exist?
Considering Google ought to be a solid year or so behind OpenAI, despite their best efforts, what kind of monster is OpenAI planning to release that beats all this?
You're assuming this isn't all vaporware. It's Google. Virtually every 'tech demo' they do is a lie.
😂😂, does OpenAI have any such infrastructure?
If Microsoft is the one who will launch. 😂😂
Altman says they are making these things smarter so they get better at everything. Then there's also their humanoid robot. Oh and they just released vision for chatgpt and newer turbo.
Google invented a lot of what made OpenAI famous. Idk where you get that notion from Ive been expecting Google to wake back up on their ML work and lead us to the next breakthrough.
I think they’re mostly just doing different things. Google is catering to enterprise, commerce, and search capabilities. Sora will be more for Hollywood, artists, and content creators. OpenAI is working on a robot, a supercomputer, a voice generator, and an upgrade on their consumer chatbot with the ultimate goal of achieving AGI. I don’t see how those things really compete with each other, besides what Microsoft is trying to do. But obviously that’s a different company. I think Microsoft could lose to Google on that front and it wouldn’t affect what OpenAI is doing.
#Using ai to select ai, #Using ai to debug code and create insertion points (~cross cutting)....it could then analyze and respond by altering the code to fit conditions (~FPGLA).
I'm working on knitting Neo4J and ai-models for an interactive teal time graph dashwindow to use user priveldged based access.
It seems that ai could refine alternative models based upon a user's project and use rag to learn preferences...and present them ranked with pros and cons of each.
It's like Autogen with frosting on top. I can dig it saving me a lot of design setup time for agent clouds.
i heard the keynote before, and I really appreciate your insights and commentary to make it more "real." There is often a strong corporate biase when we hear these kinds of releases. Great video very informative. Love your stuff
The tool definition is to avoid running arbitrary code on their server. If every tool *must* be implemented as an HTTP API then they can handle them all in exactly the same way and handle things like timeouts and errors in a standard manner.
Nice video Matt, your huge efforts are highly appreciated. Thanks for sharing!
Still watching but as always love the content and the way you deliver it, so glad I stuck around as you are my go to to keep up with whats going on
3:00 that’s great , I hope the model works well though
Great announcement, thanks for sharing!
I've been working with several of these offerings over the past few months, developing some great use cases. I was thinking of perhaps showcasing some of them on my channel soon.
Agent > Example > create example
The example is a dummy dialogue.
You need to actually mockup a weather convo. Each prompt/return is an ‘action’ in the example convo. There are other actions, like tool. Tool is where you put the code
I'll be ready for Google to put that kind of tech to use on youtube. No more trap videos where the title, description, and thumbnail don't match the content. TH-cam could auto-down-vote videos that don't match their title, description, and thumbnail. I can imagine that Google will be the king of multi-modality due to TH-cam.
I'm very much not ready for them to scan videos for anti-capitalist sentiments and shadow ban them, perpetuating the head first dive into the end of the human race this train of capitalism already has us on.
I bet they start automatically editing yt videos to make them better.
They can edit it based on what your searching for.
Or edit it based off your watching style that it learns over time.
LOL they cant even keep feature parity. open maps and tell it to play music through blackpplayer ex LOL it wont do it anymore
@@Samuel-er6tdHAHAHAHAH I never want to watch Indian videos... But guess what?! . So Google's AI can't do its main job of giving you the right content when you search using their search page (which is the foundation of Google, their vision). What's left, getting into a car powered from a Cloud service of a company (that same "G") that has an extremely worrisome reputation? Won't you finally start seeing beyond the sci-fi movie of little "ads", little "AI", little "lgbt" and many "G"?
@@Samuel-er6td How about a rating they can apply based on the content of the video matching that of the description and the thumbnail? If it goes below a certain percentage it gets held until the user adjusts the promo content.
Some of Matt’s videos were a bit lightweight in the past, but he really nailed this one. 👍 good analysis, brought in external knowledge and not just AI fanboy content. Nice job. Will be subscribing to this channel now!
Thanks :)
Thanks for sticking through it - I dropped out on the live original presentation when I saw all the customer agent stuff vs the more interesting
However, you pulled out a few more things that I missed
Seems fine, but I'm not switching from OAI just yet. I've got too much built with them to walk away.
it's the new ecosystem lockin
the Mercedes missed opportunity is probably an insurance issue. They'd need to long term test of it's safe because they don't want to get sued if someone expects the ai to read their mind and then goes road rage and blames it on them
10:05 GM is already there. Every modern GM vehicle starting from the Cadillac Lyric, 2024 Acadia & Traverse, 2025 Enclave, all have the Google agent built in. As the agent improves, the software will be updated over the air. GM is even phasing out Apple CarPlay and Android Auto, because the Google voice agent will fulfill all the requirements, including infotainment and HVAC control, vehicle information query, mapping, POI search and navigation, traffic updates, etc. Eventually all GM models will get the Google interface.
Ok, so I'm not a computer programmer, but I do have something of an interest in A.I for many reasons.
One of them being that as an artist, it interests me in what they can do.
The other day, I was watching a guy creating agents and without any programming needed.
He simply typed what the task of the agent was, and it did the rest. He even connected 4 of them together to different tasks but then combining the outcome to achieve his goal.
The way I understand them to work, although I could be wrong, is that you give each a task to perform, and that is what it does.
Then, we can share and combine information between them if needed and also its possible to have another one keeping an eye so to speak on each agent and test them should you wish.
If I'm correct in my understanding then it's clear by the Google video that they are using them in such a limited way because I can see so many interesting and new uses for them, even with things that have never been done before.
Of course, I'm sure eventually you'll be able to do all this just by speaking to it rather than typing. Typing is so outdated.
An interesting video, and I look forward to seeing what gets created using agents in the future.
Could you link the video?
@@EyezaGotSKILLZ The one with the guy using agents and using language to tell each what their tasks were you mean?
@@EyezaGotSKILLZ I believe it was this one or at least something close to.
th-cam.com/video/e24BlWvSLNM/w-d-xo.htmlsi=rq-EAbFVCdqvROEn
@@08wolfeyes yes
Thank you. Your commentary adds real value to this video. I'll definitely watch more.
Nice video! Will you cover Mixtral 8*22b soon? That would be great to see. The 8*7b was already great.
Would love for you to do an in depth video in Gemini 1.5 pro 🙏
Don't bother yourself. It is useless
@@flv-hd7nnit is usefull indeed I have tried it a lot of times now seems google is getting serious about their quality of AI
I tested the Gemini 1.5 Pro, it is simply much worse on text than Gemini 1.0 Pro. It is indeed useless so far.
Love the analysis and call-outs.
The only thing I took from this is that they have optimized their chat sales bots to allow even small businesses to train them without tons of data etc. pretty small innovation, but they need this information pushed out there hard to allow companies to know the service exists as easy as it is.
DF and DFCX are not previous, they are still current and are now more mature. Vertex agent builder is different. When you need a mix of determinism (ala decision tree), entity extraction and fallback LLm, dialog flow it is. Plus it supports call campaigns, testing, agent assist and all of those things as one click features.
Thanks Matt for the review, as always you have perfect timing and bring a common sense perspective.
The thing that seems to confuse people is actually a very good engineering principle for decoupling systems so that they are not tightly coupled with each other.
The language models are only responsible for constructing the input parameters for YOUR code, you take those input parameters and pass them into your code and submit the result from YOUR code (running anywhere) to the vertex AI.
You are not bound to upload your code to the vertex AI, it can be running on you local computer or anywhere.
Gold video ❤ Thank you so much 🎉🎉
Thank you Matthew for all of your amazing videos
I would think the context window is super cool for training robots, cause you could have a video feed going to the llm, or even just training it on tasks, like show it a video of what you want being done and the llm can translate that into language.
It seems like the Google's assistant functions may be similar to OpenAI's assistant functions. Basically it allows you to observe the response you get from the assistant and call a function in your own code base. No code in the assistant itself for the functions. I haven't used them, but I'd love to see a solid use case for implementing them.
10:24 Note on Mercedes: Every new car has „Hey Mercedes“ - an assistant helping you out basically with everything. And it works for me.
Great video. I appreciate what you do.
Tried to setup crewai today and was not able to handle my use case of sending image to gpt vision and get output from it. Any suggestions?
If agent can isolate background noise, and enhance voice quality in interns of tone and sound within a norm that would be wonderful, because lots of support staffs are working from home and diverse background.
Regarding your questions- On the tools and API - it works like GPT plugins - you provide your backend api spec using API, and the agent will trigger that endpoint if it wants to. It’s on you to implement the backend logic, and the Ai will just call it.
Matt, you mentioned better use case for the agent approach. It would have been awesome if you would have comment after the keynote on each like you did on Mercedes-Benz.
Hey Matt,
What's the ticker thing on your bookshelf?
I believe Mercedes didn't mention the most desired use case because the in-car assistant would need to function offline. Google's Agent technology has not addressed this scenario, even though their open LLM gemma 2B has the capability to operate on a car's infotainment system.
Thanks to our sponsor - every news release
Love the analysis
#Google/Agent builder
It seems very sales oriented....I believe we all need personal agents to cope with other agents and the information, and data flows.
I want something that doesn't involve having to use a custom IDE or decrypt a starship console. Is this better than crewAI?
Agents that can do knowledge work is the real juicy technology.
Agree on voice for car
Hey Matt. Have you seen platforms like Kisai that do a lot of the heavy lifting with getting from design to production code? Lots of Gen work in there that makes this seem a bit rudimentary.
Probably offers a little improvement in the performance but I do not see how it is any different from passing an intial prompt and then starting the session
I think you're spot on about Google "playing it safe." They've got this super-global brand; as soon as they make an agent that says or does something questionable, it's huge news. Microsoft figured out how to solve this problem decades ago: let small companies do innovative things, then buy the successful ones.
They just don't got that yet. In this case, you are way ahead of them. Keep on 😊👍
When it comes to a personal AI agent, I would NEVER allow any government or corporation have tabs on it for what should be obvious reason
It's enterprise ready if not futuristic. The compliance controls, frozen model in tenant, seems huge when I am freelancing for big corps.
I thought about what you said, that Google is playing it safe, but I think what it wants is to bring AI to everyone, and that is not easy at the infrastructure level, I think we have to do that step by step, so as not to falling into the attempt, and thinking like a common user, an AI assistant for me is the best option, which is what we already do with chatgpt, I use it for everything, even cooking, but I pay for it directly
Advanced enough to be omnipresent in the marketplace but not too advanced to make your company obsolete. This is priority number one for companies
@11:30 - I believe this is intelligent of Google to enable others to first launch their own AI Agents. I'm sure they have many of their own ideas for future capabilities in the making. Incredible tech. Thanks for the share.
30:00 pretty sure they can already create entire codebases based on a figma design. (Even create figma designs based on a design system -- e.g. Material UI)
Right now they're telling us it's "helping us" but it won't be long before the entire process is fully automated.
Brb, learning how to farm.
How could one use it to generate figma designs based on UI components?
Nice and honest review
You mentioned Dialog Flow, is it worth taking the time to learn CCAI CX or will AI Agent builders and Vertex supercede take the place? I want to do conversational AI and meet my customers where they're at. Dialog Flow looked a good option, is that still true today?
Would love to know answer to this!
I am going through the same problem
Totally agree
Hey Mathew, did you make any video on a web app that prompts gemini vision pro with image and some fixed text prompts and get the output displayed on the UI. if you did , please share the link and if not please make one.
You are awesome, sir!
@Matthew; 33:40, you sure as hell will when your shit don't run after the change(s)!
cheers brother!
Now, i have everything, can someone just give me a good startup idea as well? 😂😂
CEO replacement
@@WhyteHorse2023 Google has already done that I think. 😅
Google is building back their "moat". Very interesting to see the giants battle it out with AI. Hoping to catch some of their scraps!
I like the Mercedes @digital experience” when it really is data gathering, and digital surveillance😂
Functionality of a device always has to take precedent. A car is . Point A to point B
Very interesting analysis
I feel like the problem here is trying to find the right hardware and OS configuration to properly take advantage of the AI model.
And let's not forget about the fact that this needs to be a good value proposition to be marketable to a large enough audience.
It seems they are going for businesses but can't make the transition to the home market.
Seems very reminiscent of IBM.
You were too impatient with the red-top lady use case 😀. It seemed nice
Great content and thanks for the hard work. This may be out of your knowledge but can you please do a video on why these companies are offering so many great features for little or nothing. We have seen that there is always a quotation marks "you are the product" type of language interceptions of these features for example in the Mercedes Benz hey Google collaborations. Many have complained that the OnStar incredible features turn out to be that they were selling data to insurance companies leading to cancellation are much higher insurance rates in the future, as they were basically spying on the drivers of their cars. The EU has incredible protection policies that really need to be implemented into all AI products
Doesn't Zapier do the email automation though ?
Verbiage and jargon.
I thought that the various AI products would separate the task and send it off to specialist agents to work on,
An example. Task: "Create a business plan for me to take possession of my very own 200km long O'Neil cylinder at L4."
Or Task: "Destroy Einstein's Bent Nothingness conjecture an use the newly discovered physics to design and build an anti gravity craft,"
I hope it won't get discontinued. Like most of the google products
"Looks like VS Code": Could it be Google's IDX? I think that's build on the same (open source?) library as VS Code. (I don't have access to Google's IDX yet)
What is the significance of 1 million tokens versus 10 million tokens?
Are tokens related to embeddings?
Thanks!
Can you compare gemini code assistant vs copilot? Some deep testing because I didn't find it that good.
I hope it is ok for me to post a question. Do I need to pay in order to use Google Assistant to review my email like the benefits example in the video? Do I need Google Workspace to do that? I think that is a paid version. What about Google vertex? I hope I am spelling it correctly.
You have “Hey, Mercedes” - it needs a lot of improvement…but they are working on it!!
Please do a video that shows how to access those features within Google. It looks to me like vapourware when I have a look at what we have available.
When I hear agents and AI together, I can't help but think of agents from The Matrix, lol. Thanks for the great video and summary to clear that up mate.
How do I access this? I've been searching everywhere but I literally cannot find even 1 of these features. Is it not available in Europe?
Thanks!
Thanks so much!
LMAO the new Mercedes looking like a sentra
31:00 this is their new IDE in the browser.
Kinematics for sports agents?
So true Mathew. Refactoring the message around shopping assistants as new A.I. is a sad day for the org that saved search from a flood of spam. How the mighty can fall.
Im thinking about teaching an a.i. master class with no coding required,and we will all explore together and ask questions and answer them
with all the AI talent at their command, why should I keep reducing the audio volume in youtube when the ads pop up, it's insanely high vis-a-vis the content audio. Would love to know the easiest way to solve this with AI. The same can be applied for music playlists on my phone.
#Agents re-incarnate
I think it was Bell that came up with the name for their hypercard system...then Apple made their program....in between there was object oriented chip set (432) that Intel, Siemens and the DOD created....built for transparent distributed multi-proccessor workloads it ran Smalltalk and ADA it was bullet proof. ..all resources were regarded as agents with tasks assigned available resources and error checking and correction.
its just a modilar code stack to link together mac had this decades ago
Matt I completely agree, this whole showing was very unimpressive
I think one of the reasons we don't see strong agent use is due to limitations from inherent statelessness of inference. Current ai uses the chat history to 'seem' continuous, but do not retain unsaid thoughts/intent/goals. This causes issues in Long Term Planning, having to rely on low information density hacks like reason aloud your actions, goals, actions; but if any text is missed in the Chat History then the next inference may not be able to infer that missing context. Those internal goals, intent, reasons, thoughts do not survive inference to inform the next inference if not explicitly said aloud and current tech has no way of getting that dense representations across inference -- once the inference ends it's practically a new ai given the chat history up to that point (no previous inference's 'thoughts'); hence the discontinuity causes issues.
possible solutions:
~memory-informed inferences (as a vehicle to get dense context across to next inference)
~state based system like a giant Mamba/Jamba perhaps
Great input, thanks
I love your comments!🤪
Approximately one week ago my Google homes are unable to play my TH-cam playlist or turn off or on smart plugs or bulbs. I'm wondering if it has anything to do with Gemini. I have gone through all the suggested fixes but nothing is solving the problem. I am thinking the assistant was updated or maybe Gemini has something to do with it not being able to recognize simple requests.
Let me know if Im not understanding this...
1M tokens are a posibility but costs 7$?
Even if you can fit your entire code base into the context window, you would have to pay for all the input accounts for every prompt. It doesn't seem practical.