Llama 3.1 and Open Weights: Run It Yourself or Call an API?

A couple of weeks ago Meta released Llama 3.1, and it raises an obvious question for any small team. If the weights are free to download, why pay per token for somebody’s API? It’s a fair question, and the honest answer is that “free to download” covers less ground than it sounds like. The license has real terms, the bigger models need serious hardware, and running a model is a service you now own.

Here’s how I’d think it through, sticking to what Meta and the big clouds have published.

What changed between April and July

Meta’s Llama 3 announcement on April 18, 2024 shipped two sizes, 8B and 70B, trained on sequences of 8,192 tokens. The Llama 3.1 announcement on July 23 upgraded both of those and added a 405B model. All three now have a 128K context window, and the model card lists eight supported languages, roughly 15 trillion tokens of pretraining data, and a December 2023 data cutoff.

The quieter change is in the license, and for a lot of teams it matters more than the benchmarks. The Llama 3 license said you could not use the model or its outputs to improve any other large language model, except Llama 3 and its derivatives. The 3.1 announcement says Meta changed that so developers can use outputs from Llama models, including the 405B, to improve other models, and the model card names synthetic data generation and distillation as uses the license allows. There’s a string attached. If you use Llama 3.1 or its outputs to create, train or fine tune a model that you distribute or make available, the model’s name has to start with “Llama”.

Open weights is not the same as open source

What you get is the trained weights plus the code to run and fine tune them. Meta’s pitch is that you can run them on prem, in the cloud, or on a laptop without sharing data with Meta, and that part is true. What you don’t get is the training data. The model card describes the pretraining set as a new mix of publicly available online data, plus over 25 million synthetically generated fine tuning examples, and that’s the extent of it. You can adapt the model, but you can’t rebuild it.

You also don’t get an open source license in the sense most developers mean. The model card itself calls the Llama 3.1 Community License a custom commercial license. Back in July 2023 the Open Source Initiative published a post saying Meta’s Llama 2 license is not open source, pointing to points 5 and 6 of the Open Source Definition, which forbid discrimination against persons, groups or fields of endeavor. The two features the OSI objected to, a commercial restriction for some users and an acceptable use policy, are both still in the 3.1 license.

The commercial restriction is the famous one. Section 2 of the Llama 3.1 Community License says that if your products or services had more than 700 million monthly active users in the calendar month before the 3.1 release date, you need a separate license from Meta, which Meta can grant or refuse. As I read the wording, it’s a snapshot tied to the release date, so it’s aimed at a handful of giant companies rather than at your startup’s growth curve. I’m not a lawyer, and if your business depends on that reading, have one read it too.

The terms a small team will actually bump into are more mundane. If you distribute the Llama materials or a derivative, or make available a product or service that contains them, you have to include a copy of the agreement and prominently display “Built with Llama” somewhere reasonable, such as your website, UI, docs or about page. Copies you redistribute need a Notice file with Meta’s attribution line. Your license terminates if you sue Meta claiming Llama or its outputs infringe your intellectual property, Meta can terminate the agreement if you breach it, and once it’s terminated you have to delete the materials.

Then there’s the Acceptable Use Policy, which the license pulls in by reference. Much of it is what you’d expect, but a few items are worth checking against your product. It prohibits using the model for the unlicensed practice of professions like law, medicine and finance, for operating critical infrastructure or heavy machinery, and for representing Llama outputs as human generated. It also says you must disclose known dangers of your AI system to end users. If you’re building something that sounds like a doctor, read it twice.

One thing people miss: calling Llama through a cloud doesn’t make the license go away. AWS’s own post on deploying Llama 3.1 in SageMaker JumpStart notes that you accept the EULA and acceptable use policy before the model deploys.

What the hardware actually looks like

This is where the 405B headline meets reality. Meta’s announcement is upfront that a model at that scale requires significant compute resources and expertise, and says Meta quantized it from 16-bit (BF16) to 8-bit (FP8) numerics so it could run within a single server node.

The most concrete public numbers I’ve found are in AWS’s Llama 3.1 in SageMaker JumpStart post. The default instance for the 8B models is ml.g5.4xlarge, which supports a 27,000 token context there, and AWS’s context table shows the full 128K only on its larger ml.p4d.24xlarge and ml.p5.48xlarge instances. The 70B models default to ml.p4d.24xlarge, listed as 128,000 tokens of context on 8 A100s. The FP8 build of the 405B goes on ml.p5.48xlarge, listed as 72,000 tokens of context on 8 H100s.

Two things jump out. First, AWS’s default for both the 70B and the 405B is an eight GPU machine, which is a very different budget conversation from the 8B’s default instance. I’m not going to quote prices, since they vary by region and commitment, but pull up your provider’s pricing page for those instance types before anyone says “free”. Second, the context window you can actually use depends on the hardware you pay for. A 128K model on an instance that holds 27K is a 27K model for your purposes.

Data control is the strongest argument, but not an automatic win

The best reason to self host is that prompts and completions never leave infrastructure you control. For regulated data, customer contracts with strict data handling clauses, or anything air gapped, that can settle the question on its own. JumpStart is a middle ground here: AWS describes the models running on dedicated instances isolated inside your own VPC.

But “we don’t want the model vendor seeing our data” doesn’t automatically mean you have to self host. AWS announced all three Llama 3.1 sizes for Amazon Bedrock on release day, and Bedrock’s data protection documentation says Meta and other model providers don’t have access to Bedrock logs or to customer prompts and completions. Google Cloud launched the 405B on Vertex AI as a managed API in preview, said the 8B and 70B would follow as managed APIs in the coming weeks, and made all three available to deploy yourself in Model Garden from day one. If your data already lives in one of those clouds, the trust question may already be answered.

Self hosting does give you one kind of control that’s hard to get any other way: you decide when the model changes. The model card describes Llama 3.1 as a static model trained on an offline dataset, and the file on your disk stays that file until you replace it.

The ops bill nobody puts on the slide

Running your own model means owning capacity planning, GPU availability, autoscaling or paying for idle hardware, monitoring, and the inference server itself. Meta says it worked with vLLM, TensorRT and PyTorch on day one support, and those projects are now in your upgrade path too.

Safety becomes your job as well. Meta shipped Llama Guard 3 and Prompt Guard alongside 3.1, and back in April it recommended checking and filtering all inputs and outputs. Self hosted, those are extra models you deploy, scale and patch.

And the pace is brutal. Llama 3 to 3.1 took about three months and changed the context window, the language support and the license. Every one of those is a migration you plan, test and roll out yourself.

How I’d decide

For most small teams, I’d start by calling Llama 3.1 through a managed API. You learn what your real traffic and quality needs are, and you can move later because nothing ties you to a proprietary model.

I’d switch to self hosting when one of a few things is true: your data can’t go to any third party, cloud included; you need fine tuning the platforms don’t offer yet (Bedrock’s launch post listed it as coming soon); or your volume is steady enough to keep an eight GPU box busy. If you go that way, the 8B is a sane first project. The 405B is not a side quest for a team of four.

The new license opens up a combination I find more interesting than either pure option. Use the 405B through an API to generate training data, fine tune the 8B on it, and host that small model yourself. It’s allowed now, just remember that your model’s name has to start with “Llama” if you share it, and put “Built with Llama” where people can see it.