Conversation

Notices

1. Bernie (codewiz@mstdn.io)'s status on Sunday, 19-Mar-2023 23:58:53 UTC Bernie

   I've been playing a bit with #OpenAssistant, an open-source #AI chatbot similar in scope to #ChatGPT:
   https://open-assistant.io/

   While it's in early stages, it looks very promising. If you have some time, you can contribute by manually labeling / ranking prompts and responses.

   • Srevin Saju (srevinsaju@mstdn.io)'s status on Monday, 20-Mar-2023 11:54:32 UTC Srevin Saju

     in reply to

     @codewiz recently, I gave my shot with the Pygmalion AI 6B model, the results were not as bad as I thought it would be. I tried running it on Google Collab, and it worked pretty okay. Testing it locally is going to be a pain, since my laptop doesn’t have a GPU. The model was able to load and generate dialogues in 10 to 15 seconds, at a rate of 1 second per token, at 8 GB GPU RAM, 3 GB RAM and optimal CPU usage.

   • Bernie (codewiz@mstdn.io)'s status on Monday, 20-Mar-2023 18:47:11 UTC Bernie

     in reply to

     • Srevin Saju
     • smol, desu ne?

     @srevinsaju @smoldesu For a GPT, is the model size affect the number of neurons in each layer? Then, it would be somewhere between N^2 and N^3, without considering the effects of caches and the difference between layer type.

     I imagine changing the number of layers is more complicated, but for inference layers should have linear cost... right?

     My experience is limited to reading ML blogs and watching video lectures 😅

   • Srevin Saju (srevinsaju@mstdn.io)'s status on Monday, 20-Mar-2023 18:47:12 UTC Srevin Saju

     in reply to

     • smol, desu ne?

     @smoldesu @codewiz right! looks like the time taken grows exponentially with the size of the model.

   • smol, desu ne? (smoldesu@mastodon.social)'s status on Monday, 20-Mar-2023 18:47:14 UTC smol, desu ne?

     in reply to

     • Srevin Saju

     @srevinsaju @codewiz On my 4 core ARM rack I can get ~240 tokens in <10 seconds if I use gpt-j-125m