Happy Friday everyone,

Updates

This last week was filled with interesting concepts, from the update on Neuralink to discussions of the metaverse. Honest AI appeared in a call-in with Betaworks last Friday, joining panelists discussing how NLP is currently the hottest trend for startups, even if you exclude GPT-3. More improvements to the website have also been shipped, including a new article design. (love it? hate it?)

Meanwhile, the community has launched! While it is still in it's earliest form, the first brave users signed up last week. I will continue to add topics, links, and answer questions there so please join in and invite some friends or colleagues.

I encourage you to Sign up to the community for free!

Should you build with GPT-3?

In the first honest article, I break down the pros and cons of using the OpenAI API as the core part of a business. As with any new piece of technology it is important to understand the risks ahead of time, without getting too caught up in the hype.

Read the article here

Pricing

OpenAI debuted their pricing model for GPT-3 this week, which resulted in some pained grimaces. With pricing based per-token, it looks like many fixed-price apps will be rendered DOA. This limits the fun and creative cases along with the daily-helper apps. I am not saying they are wrong in pricing it high; If you have the best, you can charge the most. Based on my personal usage I would already be in the $400/month category. Notably this is just for the beta period, and the largest engine, davinci. We will have to wait and see for more details later. Pricing is enacted starting 10/1.

Deployment

New guidelines were also released as to which applications can actually use GPT-3. Tiered into 3 categories based in risk, they are taking an active stance to limit bad press. Notably they are banning their API's use for generating synthetic articles for SEO, pretty much anything to do with politics or medicine, and open-ended chatbots. Every usage will be hand-reviewed by the OpenAI team, which should help to ease fears that the internet will be soon overrun by intelligent bots.

It is safe to assume that with these updates the crazy hype seen over the last 3 months will start to cool off.

Interesting links this week –

• aitextgen is a smaller, simpler GPT-2 model you can train for free

If the pricing for GPT-3 has you a little disappointed (🙋‍) then maybe it is time to take a look at a more scaleable option. I came across Max Woolf's project aitextgen on Twitter and it looks quite interesting for specific use cases. Featuring a teeny-tiny model (1,400 times smaller than the flagship GPT-3), it should be very fast when trained for specific text generation. Just don't expect the same level of fake smarts shown by others.

• Neuralink's summer 2020 update.  Brains + AI = ✨(Youtube)

Neuralink remains one of the most interesting companies in the world. While the aspirational, world-changing goals remain years or decades away, it is hard to not get caught up in the what-ifs.  Machine learning is a core component of making sense of the complex waveforms emitted by the brain. If you haven't seen the video demo yet, check it out.

• Who is behind OpenAI? (Youtube)

Do you want to understand a bit more about who designs these complex AI systems? While a few months old, this is a great conversation between Lex Fridman and Ilya Sutskever (co-founder of OpenAI) to see logical thinking in action.

Coming up –

• The first feature essay! A lengthy piece on AI, APIs, and the Metaverse. This is one you won't want to miss.
• A dive into why synthetic personas and other virtual media are going to be huge in the near future.
• More info on a sister project to Honest AI, a site that makes it very easy to use transformers (GPT2/3)!

Have a good weekend,

-Tyler

The hype around GPT-3 right now is frothy. It feels like a 'game-changer' to many and the potential it unlocks is stated at every turn. But should you really build (or invest in) a company based on GPT-3? Note: these are solely my opinions. I have used GPT-3 quite extensively now, for a number of different tasks and to build demos.

My take: Not yet.   (well, maybe)

Let's back up for a second.  Here is what we know about GPT-3.

• It is a huge language model (350gb, 175B parameters)
• It has a 2048 token (~word) attention span, with a maximum output of 512 tokens
• It is in limited, invite-only beta right now. (the slack group is ~1k total members)
• It is run in the Azure cloud and you can only access it via their API.
• When doing complex tasks it is relatively slow to use.
• It is relatively expensive (~2-16¢ per API call) *Edit: see chart below
• In order to use it in production you need to go through a review process and actively limit toxic outputs.

That all sounds a little scary, right? It should  –these are non-trivial platform risks that would likely be critical paths for a GPT-3 based business. Are there other competitors to it?  Sure.  But none match the ease of use, and general quality of output that the huge 'davinci' engine returns.

Risks

You are at their mercy for pricing

Even if the initial pricing is cheap to start with, without immediate competition they have serious pricing-power.  You will be a price taker.  Good luck with your chat bot though (sorry).

**Edit 9/1/20: We now have an idea of how expensive API calls will be. In short it is fairly expensive for an API, costing as much as $0.16 per call.  Practically speaking, this means that most applications using GPT-3 will need to bill users based on usage rather than a monthly subscription price. I don't believe this price point is unfair, but does limit the practical usage of the API. In the personas demo I made, each page load requires 5 api calls (averaging ~1200 tokens each), then another n calls for chat messages.  So the rough math looks like 6,000 + 500(n) tokens per generation. I can safely say that just in playing with GPT-3 I have used millions of tokens. Ouch.

You are at their mercy for latency

GPT-3 is not a tiny little model.  It will take some time for results to be returned. With the caveat of it is just in beta testing the model does already show slowdowns under high traffic times (as mentioned in the slack group). This means that the speed might be different in the middle of the day vs at night, or that traffic spikes could be impactful. While smaller prompts for chat have worked quickly in my testing, complicated prompts that are generating written prose take much longer. This lead to problems because....

The models attention span is short.

You need to be able to perform all of your work in roughly 2000 words. If  you cannot do that you will need to add more complex execution methods such as context-stuffing to help GPT-3 to retain some understanding of what you are trying to accomplish across multiple API calls. This means that you will be making calls in serial to the API, considerably slowing your app down. In a demo I built, I make 5 serial calls to GPT-3 and it takes ~50-80 seconds for the main process to complete. Ouch. I might be able to shave a little time off of this, but not much.  That is a long time to wait for a 3rd party API to return. (I remain hopeful the execution speed will get much faster by the end of the beta period!)

What does this all mean? It means you have to understand what you can reasonably create based on how long operations take.

Consider an app that is trying to summarize a long input of text (book, article) using basic context-stuffing. You can get more complicated with this, but bear with for a moment.

You start with 2048 total tokens.  From that you need to remove:

• At least one sentence to set initial context (assume 48 tokens for simplicity)
• 0-512 tokens to recap previously summarized context
• 0-512 tokens for output

That leaves ~1000-1800 new tokens that can be summarized per call. This equates to roughly 2 - 4 written pages or  00:07:30 - 00:13:50 of transcribed audio.

Meaning, you can summarize a book with roughly 4:1 - 10:1 compression in one pass in (pages/4)*5s.  I am being generous with the completion time, but a 100 page book would take 00:02:05, and a 500-pager would be over 10 minutes. That firmly puts long-form summarization in a 'we will email you when your task is done' category.

**Edit 9/1/20: In the recently published pricing, both the input prompt text and output text count against the charged token count. This makes the use-case of summarization as described above exceedingly expensive.  By some napkin maths this means you are looking at a few dollars to summarize a podcast or short book. ~$100 to summarize a Harry Potter book.

Toxic outputs

I don't want to dwell too much on this as I know that OpenAI is actively working on making this a great user experience. That said, right now the early content filter they have in place just isn't very good. 'Toxic' output is just incredibly hard to characterize as context and nuance matters. I was writing movie scenes and it flagged nearly every output as toxic. I am very hopeful they improve on this quickly, but for now I would recommend every company looking to use GPT-3 in user-facing apps needs to implement their own filters. Which is a downer as it significantly reduces the it just works nature of the OpenAI API.

Everyone has the same engines

As it stands right now, everyone has access to the same four pre-trained engines: ada, babbage, curie, and davinci (ranging from small to large). This means that both you and your competitor will have the ability to get the same information back from the model. All you have the power to modify is the input prompt, and what you do with the outputs. Simply put, the value of your business will not be built using a commodity (the API), it will be built on all the other surrounding processes. For example, there have been many designer or no-code plugins demoed. I promise there is a lot of work happening in the background to show you the end results that have nothing to do with GPT-3. These processes are actually what will make your company or product special, rather than the fact it is 'powered by OpenAI'.  

Benefits

It is seriously awesome

I can't stress this enough. The OpenAI API is just a great piece of engineering all around and I am very grateful that so many worked hard to produce it 👏. It has sparked the interest of so many people to care about AI and in a way is the reason this project (Honest AI) was started now.

It is incredibly easy to use.

Using GPT-3 for simple tasks is about as easy as writing your first 'hello world'. This is part of the reason that it feels like magic and has allowed so many people to build weekend demos. As it is a transformer, the input you feed it determines the output. This means that instead of engineering code, you have to creatively come up with input prompts. It is really amazing how small differences in prompt setup have material impact on how well the API accomplishes your desired task. This means that creativity is highly valuable when working with GPT-3.

The results are consistently great

The largest engine (davinci) is the first language model that I have ever been really blown away by.  The results it gives are truly amazing. It is creative in ways that most people aren't. It forms relationships between objects, people, themes, and narratives that are enjoyable to work with.  It also works very well for straight forward NLP tasks, such as picking out characteristics from an input sentence.

They allow for fine-tuning

OpenAI has already started giving people access to fine-tune specific engines for more accurate results. This has the potential to drastically improve response quality and latency. It is worth noting that they do not allow for fine-tuning of the largest 'davinci' model, so it will be important to have sufficient input data to allow for the smaller models to make accurate predictions. There is an additional waitlist for this, even for the limited number of people that already have beta access. It will be many months before this will be commonly available (via slack).

This is the future ('directionally correct')

Whether you are a founder, engineer, or investor it is clear that this type of abstract problem solving will only become more prevalent as machine learning advances. Regardless of the risks, history has shown that people front-running major trends** have been overall successful, if only in understanding the risks more clearly than others for future ventures.

**This is not to say that companies started now will succeed. It could very well be too early yet.

It works perfectly as a human helper

While anecdotal, I have started leaving a tab with the GPT-3 playground open, just to feed it things I come across during the day. It will easily summarize emails and small documents, answer questions, or just provide some entertaining ideas. This will be a serious use case for OpenAI to pursue.

It has the hype

Hype is a real benefit. Everyone is talking about GPT-3, in a way that I have never seen in AI before. For press coverage or pitching technical users,  is always easier to go with the flow than against it. Just be sure the product really uses GPT-3 rather than only slapping the label on.

So, should you build or invest?

It is simply too early to bank an entire startup or product line on the usage of GPT-3. In order to be comfortable with adding it as a critical path you should think through:

• How much it will cost. What are your unit costs? Unless each API call returns $1 I would wait until they release the pricing details in early September to get serious.
• Could end-users be exposed to toxic outputs? If the answer is yes, your startup will be a little harder. If nothing else, you will face increased scrutiny during the production review.

Exceptions:

• You are using it to build end-products, where one generation will lead to many sales. ex. Writing a book or movie script. Summarizing public documents. Creative works.
• You are building infrastructure products to support future use-cases. There is plenty of green space for picks and shovels businesses here.

My 2¢

Investors -
You should be taking all the NLP/GPT-3 meetings they can.  Learn about this stuff early rather than later.

Founders -
You should think about how to build a business that is enhanced by GPT-3, but not dependent upon it.

I am very bullish on the API overall. I think the OpenAI team is working quickly, very responsive to feedback, and has users best intentions at heart. For their very first product, this has been quite the launch! Even with the risks present, I am currently making a few different apps myself (both fall into the exceptions above).  
Building something or investing in GPT-3? Feel free to contact me at tyler@honest-ai.com.

First off, thank you for being one of the very first to believe in Honest AI!

It has been a fast and furious few weeks here, but I have been happily surprised by the reception this project has received at every turn! I pushed four significant updates to the website code this week and have few more ready to go –this truly is just the beginning.

The community is almost ready!

The centerpiece of Honest will be a community where technical and non-technical people work together to create positive applications with AI. There gap between what people read in the media and what really needs to be understood is getting wider, let's work to fix that!

This week involved creating the structure for the community using the brand new Circle platform.  After a great deal of research I found this is clean piece of software to be the clear leader – I hope you will really enjoy it.

This will be a paid community to keep the quality high, but as early believer you and anyone you refer in the near future gets free access for life!  I will be sending out invites this week, so be on the lookout 👀.

What is in the community?

• Show Us Something - Product Hunt for AI and demos. Promote your project! Posts are public and indexed by Google. 💥
• Introductions - Say hello and build connections in the AI space
• Ideas & Honest Feedback - Have an idea? Post it here for clear, no-punches-held feedback.
• Jobs - A free to post place to find jobs in AI and ML.
• Creators Chat - Building something? Publish a worklog and get help when you need it.

Demo Day

Last Wednesday, Honest made its first public appearance when I was selected to present at the Pioneer GPT-3 Demo Day. Despite some jokes about creating a bot army, I am happy to say that my demo on synthetic personas was voted to second place by the audience! You can see a short example of the demo in action below.

GPT-3

Are you tired of GPT-3 yet? I sure hope not! The last few months have be abuzz with talk of the language model from OpenAI. I expect GPT-3 to continue to dominate over every other AI conversation for a foreseeable future. With that in mind, I have a number of different demo apps in-progress to continue to push its limits and show off the real-world utility it can offer. I will be discussing these in-depth in the community, so if you are interested,  join in. :)

OpenAI has been continually changing details behind the scenes, so I have chosen to make honest-ai.com/gpt-3 a living document that will be updated regularly. Check back if you are ever wondering about specifics.

Last week OpenAI decided to extend the free beta period for the API until the first week of September and put a pause on taking new production applications until 8/31. They also are slowly opening up the capability to fine-tune your own model, with the wait time for access stated as up to 'a couple of months'.  The model has performed slower than usual as they continue to allow new users to access to the API. By the numbers, the OpenAI beta Slack group now counts just over a thousand members, 2x more than last month.

You can help!

A community is only as strong as its members. If you like the concept of Honest AI, I would truly appreciate it if you shared it with someone.

Follow Honest's new profiles around the web:

Twitter
Youtube
LinkedIn
Instagram

Thank you,
-Tyler

PS: If you ever have any feedback just hit reply, I will respond to all comments.

Can we really build virtual people from scratch?

Synthetic media is simply described as media that is not real. This can include CGI (computer generated imagery) characters we see in movies, virtual assistants we talk to, or content generated entirely by computers.

While this is not a new concept, the technology to produce it is rapidly advancing. From generating images using GANs (generative adversarial networks) to writing backstories and personality with GPT-3, the virtual identity is closing in on reality.

GPT-3 is a machine learning language model created by OpenAI, a leader in artificial intelligence.  In short, it is a system that has consumed enough text (nearly a trillion words) that it is able to make sense of text, and output text in a way that appears human-like. I use 'text' here specifically, as GPT-3 itself has no intelligence –it simply knows how to predict the next word (called a token)  in a sentence, paragraph, or text block. It does this exceedingly well.

As an analogy you can think of GPT-3 like a freshly hired intern, who is well read, opinionated, and has a poor short-term memory. It is clever and offers fresh perspectives on how to solve problems, yet you don't really trust it to run your company or talk directly to customers.

GPT-3 works by taking a section of input text, and predicting the next section of text that should follow directly after. When hearing this people often compare it to autocorrect. The biggest difference being creativity.  When you use GPT-3, you supply your input, and a few options.  The most important of which is called temperature, the measure of how creative the outputs will be. Computers are not designed to be creative, so effectively this is option gives GPT-3 the freedom to questionable choices.

If you read everything there is to read, and stored how likely words are to appear together, in context, then you should be able to 'guess' how a sentence or story will sound. This is hard to conceptualize because we as humans don't process information like this.

GPT-3 is like a freshly-hired intern, who is well read, opinionated, and has a poor short-term memory

‌
‌More specifically GPT-3 stands for 'Generative Pretrained Transformer 3', with transformer representing a type of machine learning model that deals with sequential data.

Why is fancy autocorrect interesting?

Well, it turns out that given enough input data, an AI like GPT-3 is able to repeatably perform non-trivial tasks. If you supply it well-structured input text you can get GPT-3 to respond very naturally, often appearing as if a person was generating the answers.  This makes GPT-3 well suited for tasks such as creative writing, summarization, classification, and transactional messaging.

How can I use it for my business?

Access

Right now GPT-3 is only available through OpenAI's API product offering. They will likely never release the full model to the public as they did with previous versions (GPT-2). The OpenAI API is the companies first public product and is still relatively early in its development cycle. As of June 2020, OpenAI has opened up a waitlist to be invited to join the beta. They claims 10s of thousands of people have asked to be invited, so if you have a special use-case you will likely have to email a an OpenAI employee with a specific request. They have been slowly on-boarding people over the course of the last month.

Join the waitlist

Product offerings

There are two types of API endpoints that companies can access, namely completions and search.

‌
‌Completions: The headline product, a completion lets GPT-3 take in an input prompt (which we will cover in detail below), and complete that to return a result. ‌
‌‌
‌Search: GPT-3 is very competent at parsing natural language inputs, making it ideal for building search tools. ‌
‌‌
‌Models: OpenAI currently offers 4 models (ada, babbage, curie, davinci), each of a different size. The flagship model is 'davinci', representing the 175 billion parameters touted in the media. Model size directly correlates to how fast the API calls are handled, so for time-sensitive requests davinci will likely be too slow. See 'Speed' below for more.

Fine-tuning: OpenAI will offer to train a model specifically for you, based on a dataset that you supply.  Fine-tuning will be ideal for companies to produce extremely accurate outputs for tasks with more than a few dozen input prompts, or that require a very specific output structure. Read more in Fine-tuning below.

Playground

Beyond just making calls using code to the API, OpenAI also offers a user interface called the playground.  This lets developers quickly test ideas and refine input prompts before committing to create an application. It is easy to foresee that some variant of this playground could be made in a public-facing manner for anyone to get help with smaller problems.

Pricing

This API is currently in beta testing stage and was free to use until October. Starting 10/1 the following prices are in place:

While there is a free 'Explore' tier offered, it exists more as an interest builder than a useful allotment of tokens. It is quite easy to use up 100k tokens in an afternoon. Each engine varies greatly in size, and this variance is reflected in the 'token per credit system'. The smaller models are significantly more economical to run, though they will lack the serendipity and creativity of the largest Davinci engine.

Assuming 1,000 tokens used per request (prompt+output) a subscriber on the create tier would be allowed to use:

~$0.05/call for davinci (2,000)
~$0.005/call for curie (20,000)
~$0.001/call for babbage (100,000)
~$0.0007/call for ada (150,000)

‌
It is important to understand that GPT-3 has no direct competition for creative output, meaning they can easily price the API as they wish going forward.

Limitations and Warnings

Stability

The OpenAI API is very much still a beta product, and has already had a few periods of unplanned downtime since I have used it. It is not yet suitable for critical business use cases. The OpenAI team has been communicative and very fast to respond to these outages, so I have no doubt they will be ironed out before a general release happens.

Speed

In my usage the latency of the API has been poor to OK. It is certainly one of the slowest APIs I have recently used. This is likely one of the main areas OpenAI will need to improve on before going live to a large audience. Serving a model the size of davinci (~350gb) efficiently is a non-trivial problem!

Content Moderation

As it stands API calls are made to the 'raw model'. This means that the outputs are unfiltered and may contain explicit, hateful, or racist content. 'davinci' was trained using Common Crawl, a service that records the content of web pages -including reddit, forums, and other

OpenAI is upfront with this and clearly states that applications should not let the user interact directly with the raw model. They currently require applications to go through an approval process before going into production.

Limited Inputs & Outputs

Currently the OpenAI API only allows for an input prompt to be 1000 tokens long (or around 500 words).  This means that you must fit all of the context you are supplying in that prompt, quite a challenge.

Should I build a GPT-3-based company?

tl;dr – Maybe. There are some specific concerns to be aware of though.

Almost everyone I have talked to who plays with GPT-3 has that ah-ha! moment where they think of a great project to use it on. The play‌

You are taking a significant platform risk.

• OpenAI owns the API and they are free to do whatever they want with it, regardless of how that impacts your business.
• GPT-3 has no direct competitors (yet). This gives OpenAI incredible pricing power, access control, and the ability to add/enforce policies.
• Your service is tied to GPT-3 working correctly. GPT-3 is a very large ML model, that requires tangible compute resources to operate, which means it has the potential to run into scaling issues. (as alerady seen during the beta period)
• The speed of your application will be directly dependent on the OpenAI API.

Differentiation is hard

• GPT-3 is a few-shot model, therefore it is relatively trivial to reverse-engineer an input prompt. (you can read more in the scientific paper)
• Be careful that your app itself provides value, and doesn't simply let the OpenAI API do all the 'real work'.

Garbage in >> Garbage out

In working with GPT-3 one thing becomes very clear, the inputs you feed it matter –a lot! As a few-shot model, it is extrapolating what it thinks the right response is exclusively from the up to 1000 tokens of input you provide.

Where do language models go from here?

This OpenAI is still an early implementation of AI.  As compute power and algorithm optimization continue to increase we will be able to effectively train much larger models. Based on estimates, GPT-3 cost roughly $4.6M worth of compute time to train.‌
‌‌
‌GPT-2: 1.5 billion parameters‌
‌GPT-3: 175 billion parameters‌
‌GPT-4: ???‌
‌Human brain: 100 trillion synapses (analygous to parameter)‌