Yann LeCun is head of Meta AI.
an aside..
1/1
Dane Vahey of OpenAI says the cost per million tokens has fallen from $36 to $0.25 in the past 18 months, and as such AI is the greatest cost-depreciating technology ever invented
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
Dane Vahey of OpenAI says the cost per million tokens has fallen from $36 to $0.25 in the past 18 months, and as such AI is the greatest cost-depreciating technology ever invented
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
Microsoft 1-bit era paper (released in Feb) is really a masterpiece.
Requires almost no multiplication operations for matrix multiplication and can be highly optimized.
Exciting news! We’ve finally cracked the code for BitNet @huggingface ! no pre-training needed! With just fine-tuning a Llama 3 8B, we've achieved great results, reaching a performance close to Llama 1 & 2 7B models on key downstream tasks!
Q-Sparse can achieve results comparable to those of baseline LLMs while being much more efficient at inference time;
We present an inference-optimal scaling law for sparsely-activated LLMs; As the total model size grows, the gap between sparsely-activated and dense model continuously narrows.
Q-Sparse is effective in different settings, including training-from-scratch, continue-training of off-the-shelf LLMs, and finetuning;
Q-Sparse works for both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the cornerstone and a clear path to revolutionize the efficiency of future LLMs.
