i think people are underestimating the potential here for agents building - it is now a lot easier for GPT4 to call other models, or itself. while i was taking notes for our emergency pod yesterday (https://www.latent.space/p/function-agents) we had this interesting debate with Simon Willison on just how many functions will be supplied to this API. Simon thinks it will be "deep" rather than "wide" - eg a few functions that do many things, rather than many functions that do few things. I think i agree.
you can now trivially make GPT4 decide whether to call itself again, or to proceed to the next stage. it feels like the first XOR circuit from which we can compose a "transistor", from which we can compose a new kind of CPU.
It was already quite easy to get GPT-4 to output json. You just append ‘reply in json with this format’ and it does a really good job.
GPT-3.5 was very haphazard though and needs extensive babysitting and reminding, so if this makes gpt3 better then it’s useful - it does have an annoying disclaimer though that ‘it may not reply with valid json’ so we’ll still have to do some sense checks into he output.
I have been using this to make a few ‘choose your own adventure’ type games and I can see there’s a TONNE of potential useful things.
It literally does it everytime perfectly. I remember I put together an entire system that would validate the JSON against a zod schema and use reflection to fix it and it literally never gets triggered because GPT3.5-turbo always does it right the first time.
> It literally does it everytime perfectly. I remember I put together an entire system that would validate the JSON against a zod schema and use reflection to fix it and it literally never gets triggered because GPT3.5-turbo always does it right the first time.
Danger! There be assumptions!!
gpt-? is a moving target and in rapid development. What it does Tuesday, which it did not do on Monday, it may well not do on Wednesday
If there is a documented method to guarantee it, it will work that way (modulo OpenAI bugs - and now Microsoft is involved....)
What we had before, what you are talking of, was observed behaviour. An assumption that what we observed in the past will continue in the future is not something to build a business on
Are you saying that it return only JSON before? I'm with the other commenters it was wildly variable and always at least said "Here is your response" which doesn't parse well.
If you want a parsable response, have it wrap that with ```. Include an example request/response in your history. Treat any message you can’t parse as an error message.
This works well because it has a place to put any “keep in mind” noise. You can actually include that in your example.
Coincidentally, I just published this JS library[1] over the weekend that helps prompt LLMs to return typed JSON data and validates it for you. Would love feedback on it if this is something people here are interested in. Haven’t played around with the new API yet but I think this is super exciting stuff!
Looks promising! Do you do retries when returned json is invalid? Personally, I used io-ts for parsing, and GPT seems to be able to correct itself easily when confronted with a well-formed error message.
Great idea, I was going to add basic retries but didn’t think to include the error.
Any other features you’d expect in a prompt builder like this? I’m tempted to add lots of other utility methods like classify(), summarize(), language(), etc
It's harder to form a tree with key value. I also tried the relational route. But it would always messup the cardinality (one person should have 0 or n friends, but a person has a single birth date).
even with gpt 4, it hallucinates enough that it’s not reliable, forgetting to open/close brackets and quotes. This sounds like it’d be a big improvement.
99% of the time is still super frustrating when it fails, if you're using it in a consumer facing app. You have to clean up the output to avoid getting an error. If it goes from 99% to 100% JSON that is a big deal for me, much simpler.
If you're building an app based on LLMs that expects higher than 99% correctness from it, you are bound to fail. Negative scenarios workarounds and retries are mandatory.
Honestly, I suspect asking GPT-4 to fix your JSON (in a new chat) is a good drunken JSON parser. We are only scraping the surface of what's possible with LLMs. If Token generation was free and instant we could come up with a giant schema of interacting model calls that generates 10 suggestions, iterates over them, ranks them and picks the best one, as silly as it sounds.
It shouldn't be surprising though. If a human makes an error parsing JSON, what do you do? You make them look over it again. Unless their intelligence is the bottleneck they might just be able to fix it.
I already do this today to create domain-specific knowledge focused prompts and then have them iterate back and forth and a ‘moderator’ that chooses what goes in and what doesn’t.
In my experience, telling it "no thats wrong, try again" just gets it to be wrong in a new different way, or restate the same wrong answer slightly differently. I've had to explicitly guide it to correct answers or formats at times.
It's fine, but the article makes some good points why - less cognitive load for GPT and less tokens. I think the transistor to logic gate analogy makes sense. You can build the thing perfectly with transistors, but just use the logic gate lol.
Is there any publicly available resource replicate your work? I would love to just find the right kind of "incantation" for the gpt-3.5-t or gpt-4 to output a meaningful story arc etc.
Any examples of your work would be greatly helpful as well!
I'm not the person you're asking, but I built a site that allows you to generate fiction if you have an OpenAI API key. You can see the prompts sent in console, and it's all open source:
Pass in an agent message with "Sure here is the answer in json format:" after the user message. Gpt will think it has already done the preamble and the rest of the message will start right with the json.
GPT-4 was already a massive improvement on 3.5 in terms of replying consistently in a certain JSON structure - I often don't even need to give examples, just a sentence describing the format.
It's great to see they're making it even better, but where I'm currently hitting the limit still in GPT-4 for "shelling out" is about it being truly "creative" or "introspective" about "do I need to ask for clarifications" or "can I find a truly novel away around this task" type of things vs "here's a possible but half-baked sequence I'm going to follow".
It is “good enough”. Where I struggle is maintaining its memory through a longer request where multiple iterations fail or succeed and then all of a sudden its memory is exceeded and starts fresh. I wish I could store “learnings” that it could revisit.
Blah Blah "...is NOT the correct implementation to replicate paper results. In fact, people have reported that his code cannot properly run, and is probably automatically generated by ChatGPT, and kyegomez has done so for other popular ML methods, while intentionally refusing to link to official implementations for his own interests"
Love a good GitHub Identity Theft Star farming ML story
But this method could have potential for a chain of function
It's interesting to think about this form of computation (LLM + function call) in terms of circuitry. It is still unclear to me however, if the sequential form of reasoning imposed by a sequence of chat messages is the right model here. LLM decoding and also more high-level "reasoning algorithms" like tree of thought are not that linear.
Ever since we started working on LMQL, the overarching vision all along was to get to a form of language model programming, where LLM calls are just the smallest primitive of the "text computer" you are running on. It will be interesting to see what kind of patterns emerge, now that the smallest primitive becomes more robust and reliable, at least in terms of the interface.
Exactly, we humans can use specialized models and traditional tool APIs and models and orchestrate the use of all these without understanding how these things work in detail.
To do accounting, GPT 4 (or future models) doesn't have to know how to calculate. All it needs to know how to interface with tools like calculators, spreadsheets, etc. and parse their outputs. Every script, program, etc. becomes a thing that has such an API. A lot what we humans do to solve problems is breaking down big problems into problems where we know the solution already.
Real life tool interfaces are messy and optimized for humans with their limited language and cognitive skills. Ironically, that means they are relatively easy to figure out for AI language models. Relative to human language the grammar of these tool "languages" is more regular and the syntax less ambiguous and complicated. Which is why gpt 3 and 4 are reasonably proficient with even some more obscure programming languages and in the use of various frameworks; including some very obscure ones.
Given a lot of these tools with machine accessible APIs with some sort of description or documentation, figuring out how to call these things is relatively straightforward for a language model. The rest is just coming up with a high level plan and then executing it. Which amounts to generating some sort of script that does this. As soon as you have that, that in itself becomes a tool that may be used later. So, it can get better over time. Especially once it starts incorporating feedback about the quality of its results. It would be able to run mini experiments and run its own QA on its own output as well.
"Trivial" is misleading. From OpenAI's docs and demos, the full ReAct workflow is an order of magnitude more difficult than typical ChatGPT API usage with a new set of constaints (e.g. schema definitions)
Even OpenAI's notebook demo has error handling workflows which was actually necessary since ChatGPT returned incorrect formatted output.
As long as each action has proper validation and returns meaningful error messages, you don't need to even change the control flow. The agent will typically understand what went wrong, and attempt to correct it in the next "ReAction".
I've been refactoring some agents to use "functions" and so far it seems to be a HUGE improvement in reliability vs the "Return JSON matching this format" approach. Most impactful is that fact that "3.5-turbo" will now reliability return JSON (before you'd be forced to use GPT-4 for an ReAct style agent of modest complexity).
My agents also seem to be better at following other instructions now that the noise of the response format is gone (of course it's still there, but in a way it has been specifically trained on). This could also just be a result of the improvements to the system prompt though.
The first transistors were slow, and it seems this "GPT3/4 calling itself" stuff is quite slow. GPT3/4 as a direct chat is about as slow as I can take. Once this gets sped up.
I am sure it will, as you can scale out, scale up and build more efficient code and build more efficient architectures and "tool for the job" different parts of the process.
The problem now (using auto gpt, for example) is accuracy is bad, so you need human feedback and intervention AND it is slow. Take away the slow, or the needing human intervention and this can be very powerful.
I dream of the breakthrough "shitty old laptop is all you need" paper where they figure out how to do amazing stuff with a 1Gb of space on a spinny disk and 1Gb RAM and a CPU.
I agree with this. We’ve already gotten pretty good at json coercion, but this seems like it goes one step further by bundling decision making in to the model instead of junking up your prompt or requiring some kind of eval on a single json response.
It should also be much easier to cache these functions. If you send the same set of functions on every API hit, OpenAI should be able to cache that more intelligently than if everything was one big text prompt.
They're the only one commenter on HN I noticed keeps writing "smol" instead of "small", and is associated with projects with "smol" in their name. Surely I'm not the only one who missed it being a meme around 2015 or sth., and finds this word/use jarring - and therefore very attention-grabbing? Wonder how much that helps with marketing.
This is meant with no negative intentions. It's just that 'swyx was, in my mind, "that HN-er that does AI and keeps saying 'smol'" for far longer than I was aware of latent.space articles/podcasts.
i mean hopefully its relevant content to the discussion, i hope enough pple know me here by now that i fully participate in The Discourse rather than just being here to cynically plug my stuff. i had a 1.5 hr convo with simon willison and other well known AI tinkerers on this exact thing, and so I shared it, making the most out of their time that they chose to share with me.
100%, if the API itself can choose to call a function or an LLM, then it's way easier to build any agent loop without extensive prompt engineering + worrying about errors.
You still have to worry about errors. You will probably have to add an error handler function that it can call out to. Otherwise the LLM will hallucinate a valid output regardless of the input. You want it to be able to throw an error and say I could produce the output given this format.
> "you can now trivially make GPT4 decide whether to call itself again, or to proceed to the next stage."
Does this mean the GPT-4 API is now publicly available, or is there still a waitlist? If there's a waitlist and you literally are not allowed to use it no matter how much you are willing to pay then it seems like it's hard to call that trivial.
Not GP, but it's still the latter...i've been (im)patiently waiting.
From their blog post the other day:
With these updates, we’ll be inviting many more people from the waitlist to try GPT-4 over the coming weeks, with the intent to remove the waitlist entirely with this model. Thank you to everyone who has been patiently waiting, we are excited to see what you build with GPT-4!
If you put contact info in your HN profile - especially an email address that matches one you use to login to openai, someone will probably give you access...
Anyone with access can share it with any other user via the 'invite to organisation' feature. Obviously that allows the invited person do requests billed to the inviter, but since most experiments are only a few cents that doesn't really matter much in practice.
Good to know, but I've racked up a decent bill for just my GPT 3.5 use. I can get by with experiments using my ChatGPT Plus subscription, but I really need my own API access to start using it for anything serious.
"With these updates, we’ll be inviting many more people from the waitlist to try GPT-4 over the coming weeks, with the intent to remove the waitlist entirely with this model. Thank you to everyone who has been patiently waiting, we are excited to see what you build with GPT-4!"
Interesting observation, @swyx. There seems to be a connection to transitive closure in SQL queries, where the output of the query is fed as the input to the query in the next iteration [1]. We are thinking about how to best support such recursive functions in EvaDB [2].
The thing is the relevant context often depends on what it's trying to do. You can give it a lot of context in 16k but if there are too many different types of things then I think it will be confused or at least have less capacity for the actual selected task.
So what I am thinking is that some functions might just be like gateways into a second menu level. So instead of just edit_file with the filename and new source, maybe only select_files_for_edit is available at the top level. In that case I can ensure it doesn't try to overwrite an existing file without important stuff that was already in there, by providing the requested files existing contents along with the function allowing the file edit.
Not sure that’s true. I haven’t completely filled the context with examples but I do provide 8 or so exchanges between user and assistant along with a menu of available commands and it seems to be able to generalize from that very well. No hallucinations either. Good idea about sub menus though, I’ll have to use that.
I think big context only makes sense for document analysis.
For programming you want to keep it slim. Just like you should keep your controllers and classes slim.
Also people with 32k access report very very long response times of up to multiple minutes which is not feasible if you only want a smaller change or analysis.
writing code of higher complexity (we know from CICERO that longer time spent on inference is worth orders of magnitude more than the equivalent in training when it comes to improving end performance), or doing real world tasks with unknown fractal depth (aka yak shave)
Hey, I understand the frustration (both the frustration of endless links on an over-hyped topic, and the frustration of getting scolded by another user when expressing yourself) - but it really would be good if you'd post more in the intended spirit of this site (https://news.ycombinator.com/newsguidelines.html).
People sometimes misunderstand this, so I'd like to explain a bit. (It probably won't help, but it might, and I don't like flagging or banning accounts without trying to persuade people first if possible.)
We don't ask people to be kind, post thoughtfully, not call names, not flame, etc., out of nannyism or some moral thing we're trying to impose. That wouldn't feel right and I wouldn't want to be under Mary Poppins's umbrella either.
The reason is more like an engineering problem: we're trying to optimize for one specific thing (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...) and we can't do that if people don't respect certain constraints. The constraints are to prevent the forum from burning itself to a crisp, which is where the arrow of internet entropy will take us if we don't expend energy to stave it off.
It probably doesn't feel like you're doing anything particularly wrong, but there's a cognitive bias where everyone underestimates the damage they're causing (by say 10x) and overestimates the damage others are causing (by say 10x) and that compounds into a (by now 100x) bias where everyone feels like everyone else is the problem. We need a way out of that dynamic if we're to have any hope of keeping this place interesting. As you probably realize, HN is forever on the brink of caving into a pit. We need you to help nudge it back from that, not push it over.
Of course you're free to say "what do I care if HN burns itself to a crisp, fuck you all" but I'd argue you shouldn't take that nihilistic position because it isn't in your own interests. HN may be annoying at times, but it's interesting enough for you to spend time here—otherwise you wouldn't be reading the site and posting to it. Why not contribute to making it more interesting rather than destroying it for yourself and everyone else? (I don't mean that you're intentionally destroying it—but the way you've been posting is unintentionally contributing to that outcome.)
I'm sure you wouldn't drop lit matches in a dry forest, or dump motor oil in a mountain lake, trample flower gardens, or litter in a city park, for much the same reason. It's in your own interest to practice the same care for the commons here. Thanks for listening.
If someone were egregiously out of line, typically, I feel community sentiment reflects this.
Personally, I feel your assessment of cognitive bias at play is way off base. I don't think it's a valid comparison to claim that someone is causing "damage" by merely expressing distaste. That's a common tool that humans use for social feedback. Is cutting off the ability for genuine social feedback or adjustment and forcing people to be saccharine out of fear of reprisal from the top really an optimal solution to an engineering problem? It seems more like a simulacrum of an HR department where the guillotine is more real: your job and life rather than merely your ability to share your thoughts on a corner of the Internet.
Think about the engineering problem you find yourself in with this state of affairs: something very similar to the kind of content you might find on LinkedIn, a sort of circular back-patting engine devoid of real challenge and grit because of the aforementioned guillotine under which all participants hang.
And, quite frankly, you do see the effects of this in precisely the post in this initial exchange: hyperbole and lack of deep critical assessment are artificially inflated. This isn't a coincidence: this has been cultured very specifically by the available growing conditions and the starter used -- saccharine hall monitors that fold like cheap suits (e.g. very poorly, lots of creases) when the lowest level of social challenge is raised fo their ideas.
You know what it really feels like? A Silicon Valley reconstruction of all the bad things about a workplace, not a genuine forum for debate and intellectual exploration. If you want to find a place to model such behavior, the Greeks already have you figured out - how do you think Diogenes would feel about human resources?
That being said, I appreciate the empathy.
Obviously, I feel a bit like a guy Tony Soprano beat up and being forced to apologize afterwards to him for bruising his knuckles.
Not to belabor the point but from my perspective you've illustrated the point about cognitive bias: it always feels like the other person started it and did worse ("I feel a bit like a guy Tony Soprano beat up and being forced to apologize afterwards to him for bruising his knuckles") and it always feels like one was merely defending oneself reasonably ("merely expressing distaste"). This is the asymmetry I'm talking about.
As you can imagine, mods get this kind of feedback all the time from all angles. The basic learning it adds up to is that everybody always feels this way. Therefore those feelings are not a reliable compass to navigate by.
This is not a criticism—I appreciate your reply!
Edit:
> forcing people to be saccharine [...] like a simulacrum of an HR department
We definitely don't want that and the site guidelines don't call for that. There is tons of room to make your substantive points thoughtfully without being saccharine. It can take a little bit of reflective work to find that room, though, just because we (humans in general) tend to get locked into binary oppositions.
The best principle to go by is just to ask yourself: is what I'm posting part of a curious conversation? That's the intended spirit of the site. It's possible to tell if you (I don't mean you personally, I mean all of us) are functioning in the range of curiosity and to refrain from posting if you aren't.
It is true that the HN guidelines bring a bit of blandness to discourse because they eliminate the rough-and-tumble debate that can work well in much smaller groups of close peers. But that's because that kind of debate is impossible in a large public forum like HN—it just degenerates immediately into dumb brawls. I've written about this quite a bit if you or anyone wants to read about that:
I think your argument is reasonable from a logical perspective, and I would generally make a similar argument as I would find the template quite persuasive.
However, I, again, feel you're improperly pushing shapes into the shape-board again. Of course, understanding cognitive bias is a fantastic tool to improve human behavior from an engineering perspective, and your argumentum ad numerum is sound.
That being said, you're focusing too much on what my emotional motivation might be rather than looking at the system - do you really think there isn't an element of that dynamic I outlined in an interaction like this? Of course there is.
Anyhow, you know, I don't have the terminology in my back-pocket, but there's definitely a large blind-spot when someone is ignoring the spirit of intellectual curiosity in a positive light rather than a negative one.
In this case, don't you think a tool like mild negative social feedback might be a useful mechanism? Of course, there's a limit, and if such a person were incapable of further insight, they'd probably not be very useful conversants. That's obviously not happening here.
One final thing is relevant here - you just hit on a pretty important point. There is a grit to a certain type of discourse that is actually superior to this discourse, I'd happily accept that point. Why not just transfer the burden of moderation to that point, rather than what you perceive to be the outset? Surely, you'll greatly reduce your number of false positives.
I provide negative social feedback sometimes because I feel it's appropriate. In the future, I probably won't. That being said, it's obvious that I've never sparked a thoughtless brawl, so the tolerance is at least inappropriately adjusted sufficiently to that extent.
What’s your problem? There’s nothing overhyped about that comment. People, including me, are building complex agents that can execute multi stage prompts and perform complex tasks. Comparing these first models to a basic unit of logic is more than fair given how much more capable they are. Do you just have an axe to grind?
you can now trivially make GPT4 decide whether to call itself again, or to proceed to the next stage. it feels like the first XOR circuit from which we can compose a "transistor", from which we can compose a new kind of CPU.