RSS GitHub
The Ledger A sourced historical record of AI

Meta Releases LLaMA to Researchers, and Its Weights Leak Within a Week

A ledger entry in the open source archive, dated 2023-02-24.

Major
References

Summary

On February 24, 2023, Meta announced LLaMA, a family of four large language models of 7 billion to 65 billion parameters, and offered the weights to researchers under a noncommercial license granted case by case. Meta reported that the 13-billion-parameter model outperformed OpenAI's much larger GPT-3 on most benchmarks. On March 3 the weights were posted as a torrent on 4chan and spread across the internet, which put a model close to the state of the art into anyone's hands and made LLaMA an early test case in the public argument over whether powerful model weights should be released.

What Happened

Meta's AI research group announced LLaMA (Large Language Model Meta AI) in a blog post and a research paper. The models came in 7B, 13B, 33B and 65B parameter sizes; the two largest were trained on 1.4 trillion tokens and the smallest on one trillion, using public data in the 20 languages with the most speakers. Meta argued that smaller models trained on more data were easier for researchers without large computing budgets to retrain, study and fine-tune. To "maintain integrity and prevent misuse," Meta released the models under a noncommercial license focused on research, with access granted case by case to academic researchers, government, civil society and industry research labs. Mark Zuckerberg announced the release on Facebook, saying Meta was "committed to this open model of research."

The research paper claimed that LLaMA-13B outperformed the 175-billion-parameter GPT-3 on most benchmarks, and that LLaMA-65B was competitive with DeepMind's Chinchilla-70B and Google's PaLM-540B. LLaMA was a base model without the conversational fine-tuning of ChatGPT, so it was a research tool rather than a product.

About a week after Meta began approving access requests, a downloadable torrent of the weights was posted on 4chan on March 3, 2023, and spread through AI communities. The Verge reported that researchers who compared the leaked files with the official release confirmed they matched. Meta's Joelle Pineau said in a statement that "some have tried to circumvent the approval process." On June 6, 2023, Senators Richard Blumenthal and Josh Hawley wrote to Mark Zuckerberg questioning whether Meta had assessed the risks of releasing LLaMA and asking how many people had received access before the leak.

Why It Matters

Meta offered outside researchers weights that it said matched or beat much larger closed models, and the leak made that access effectively universal. Because LLaMA-13B could run on a single data-center GPU, people well outside large labs could now run, study and fine-tune a model close to the state of the art, which Shawn Presser, one of the researchers who helped spread the leaked files, told The Verge was "a huge leap." Meta's next release, Llama 2 in July 2023, was licensed for commercial use, so the research-only gate of February lasted five months.

The leak became an early reference point in the open-versus-closed debate. Critics quoted by The Verge called open release "a terrible idea" that would bring personalized spam and phishing; supporters argued that wide access was needed to find and fix the problems of language models. The senators' letter brought the question to Congress, and it raised a point that later policy debates about open-weight models kept returning to: once weights are out, no access policy can call them back.

§ How to read the metadata
Landmark
Fundamentally alters the trajectory; 2–5 per year.
Major
Meaningfully shifts the landscape; 2–4 per month.
Notable
Worth documenting; significance can be upgraded later.
Confidence
High = primary sources corroborate. Medium = credible secondary only. Low = provisional. Disputed = credible sources disagree.
Contestation
Uncontested = no formal challenge. Contested = at least one challenge filed and not resolved by a correction. Corrected = a challenge was upheld and the entry corrected; see its revision history. Superseded = replaced by a later entry. Unresolved = dispute still open.

References

  1. Introducing LLaMA: A foundational, 65-billion-parameter large language model , Meta AI () primary document archived copy
  2. LLaMA: Open and Efficient Foundation Language Models , arXiv () primary document archived copy
  3. Meta has a new machine learning language model to remind you it does AI too , The Verge () secondary reporting archived copy
  4. Meta's powerful AI language model has leaked online - what happens now? , The Verge () secondary reporting archived copy
  5. Blumenthal & Hawley Demand Answers & Warn of Misuse After "Leak" of Meta's AI Model , Office of Senator Richard Blumenthal () primary document archived copy

See also