Build something. Play with it. Use it everyday and watch the models evolve.
Not everything can be (easily) measured, especially when it comes to creative output. Heck even things like "which UI API is better to use" is incredibly subjective, people working at a high level building CRUD apps will look at VB6 and think it is amazing, while others want low level graphics primitives.
Programmatically using an AI is almost like having a partner you have to negotiate with. Miscommunications are common, the AI occasionally decides to change its answer format, or just improvise brand new things. For the https://www.generativestorytelling.ai project I had to come up with a a syntax and parser that is adaptable to changes, while staying under a token limit. The token limit is one reason I didn't use JSON, but the other reason is that I've had friends tell me that GPT will sometimes throw brand new syntax in with its answers, so I wanted a response format that was loose enough to allow the AI to be creative without breaking my parser!
It is an incredibly different experience than any other system I've ever worked with.
Not everything can be (easily) measured, especially when it comes to creative output. Heck even things like "which UI API is better to use" is incredibly subjective, people working at a high level building CRUD apps will look at VB6 and think it is amazing, while others want low level graphics primitives.
Programmatically using an AI is almost like having a partner you have to negotiate with. Miscommunications are common, the AI occasionally decides to change its answer format, or just improvise brand new things. For the https://www.generativestorytelling.ai project I had to come up with a a syntax and parser that is adaptable to changes, while staying under a token limit. The token limit is one reason I didn't use JSON, but the other reason is that I've had friends tell me that GPT will sometimes throw brand new syntax in with its answers, so I wanted a response format that was loose enough to allow the AI to be creative without breaking my parser!
It is an incredibly different experience than any other system I've ever worked with.