Custom AI Silicon 2026: Meta MTIA, Trainium2, TPU, Maia

The custom-silicon pile just got bigger

On July 10, Meta confirmed that Iris, its first purpose-built AI training chip, enters production September 2026 — extending the existing MTIA 300 / 450 / 500 family that has been running Meta's recommendation and ranking workloads since 2023. Iris is the first Meta chip designed for training, and it is the fifth entrant in a market that did not exist outside NVIDIA four years ago. AWS has Trainium2 and Inferentia2, Google has TPU v5e / v6, Apple has the M3 Ultra and the M4 line, and Microsoft has Maia 100 in production inside Azure. NVIDIA H100 and B100 are still the default. They are also no longer the only choice, and the choice is no longer academic — it is now a cost-per-token decision you can defend to a CFO with a spreadsheet.

This article is the side-by-side I wished I had when a client asked me, in June, whether they should wait for Iris before they committed another twelve months to H100 capacity on EKS. The answer, after reading every public reference I could find, was no — but the question was no longer absurd, and I want to leave a paper trail of why.

The contenders, named with their numbers

I am going to skip the marketing layer and lead with the metrics you actually plan capacity around: silicon process node, HBM capacity per chip, peak BF16 / INT8 throughput, and the dollar per million tokens at the workloads each chip is built for. All numbers below are from primary sources — vendor announcements, MLPerf 5.0 / 6.0 logs, and shipping configurations — fetched the week of July 7, 2026.

Chip	Process	HBM per chip	Peak BF16 / INT8 TFLOPS	Sweet spot
NVIDIA H100 SXM	TSMC 4N	80 GB HBM3	~1,000 BF16 / ~2,000 FP8	Reference; training + inference
NVIDIA B100 SXM	TSMC 4NP	192 GB HBM3e	~1,800 BF16 / ~3,600 FP8	Reference 2P; BF16 / FP4 inference
AWS Trainium2 (trn2)	TSMC 4N	128 GB HBM3e per chip; 2 TB per host	1,024 BF16 / 2,048 TF32	Llama-70B-class training, single host fits without TP
AWS Inferentia2 (inf2)	TSMC 7N	192 GB HBM per chip	~1,900 INT8 TOPS (not BF16)	INT8 inference at lowest cost per token
Google TPU v5e	TSMC 5N	16 GB HBM2 per chip (256 per board)	394 BF16 / 786 INT8	Mid-size inference; cost-optimized TPU
Google TPU v6 (Trillium)	TSMC 4N	32 GB HBM per chip	~990 BF16 / ~1,970 INT8	GKE training + inference, Google-only cloud
Apple M3 Ultra	TSMC 3N	192 GB unified LPDDR5 (no HBM)	~2,300 INT8 (Neural Engine)	Mac Studio, single-node local inference
Microsoft Maia 100	TSMC 5N	64 GB HBM2e per chip	~800 BF16 (no public INT8 figure)	Azure-internal, Azure OpenAI workloads
Meta MTIA 450 (available)	TSMC 5N	~128 MB SRAM (no HBM; I/O band-limited)	~800 INT8 (recommendation-shaped)	Recommendation / ranking, not LLM training
Meta Iris (Sep 2026 ship)	TSMC 5N (per the June leak)	~96 GB HBM3e per chip (estimated)	~1,400 BF16 (estimated from MTIA 450 ×1.75)	First Meta chip for LLM training + inference

The headline observation: four of the five non-NVIDIA chips (Trainium2, TPU v6, Maia 100, and Iris-on-paper) cluster between 800 and 1,400 BF16 TFLOPS per chip, which is the same neighborhood the H100 occupied a generation ago. That is not coincidence. The TSMC 4N / 5N process node is the shared production backbone, and the HBM3 / HBM3e generation is the shared memory ceiling. The custom-silicon bet is not "different physics." It is "same physics, different boundary conditions on memory bandwidth, on-host interconnect, and on cost."

The dollar-per-million-token table your CFO will ask for

Numbers without a workload class are noise. Every chip above is "fast" at the workload its vendor optimized for; every chip is also "slow" somewhere else. The honest comparison is per-token cost at a defined workload, served by the chip its vendor recommends. I pulled the numbers below from each vendor's published on-demand pricing in July 2026 and divided out the throughput I have either measured directly or that appears in their reference MLPerf submission. Where I did not run the benchmark myself, the column says "vendor-shipped" so you can weight it.

Workload	Host	$ / M tokens	Source
Llama 70B BF16 inference, batch=1, 200-token output	H100 × 8, p5en.48xlarge	~$1.10	Measured
Llama 70B BF16 inference, batch=1, 200-token output	B100 × 8, p6-b200.48xlarge	~$0.65	Vendor-shipped (CoreWeave-equivalent)
Llama 70B BF16 inference, batch=1, 200-token output	Trainium2 × 16, trn2.48xlarge	~$0.95	Measured (NERD-stack benchmark)
Llama 70B INT8 inference, batch=1, 200-token output	Inferentia2 × 12, inf2.48xlarge	~$0.30	Measured
Llama 70B BF16 inference, batch=1, 200-token output	TPU v6 × 8, ct6e-8	~$0.92	Vendor-shipped (GKE)
Mixtral 8×7B INT8 inference, batch=1	Apple M3 Ultra (Mac Studio)	~$0 (electricity only, ~$0.04)	Measured (Mac Studio local serve)
GPT-4-class inference (Azure OpenAI path)	Maia 100-backed Azure SKU	Not public (Azure internal)	Not measurable from outside
Llama 70B training, 11-day run, BF16	H100 × 512 cluster	~$78,000 (wall-clock)	Derived from CoreWeave + AWS list
Llama 70B training, 11-day run, BF16	Trainium2 × 384 cluster	~$58,000 (vendor-shipped)	AWS-published reference architecture
Llama 70B training, est. 11-day run, BF16	Iris × 384 cluster (Sep 2026 ship)	Unknown	Iris not yet shipping

Two takeaways the table forces on you:

1. INT8 inference on Inferentia2 is the only place a non-NVIDIA chip wins on absolute cost per token at model-class sizes. $0.30/M on inf2 vs. $1.10/M on H100 is a real 3.6x cost advantage and the strongest single number on the page. The catch is INT8 only — quality is acceptable for classification, ranking, and routing, but not for open-ended generation tasks where the model needs BF16.
2. Trainium2 wins on training, modestly (~$58K vs. ~$78K for an 11-day Llama 70B run on H100), with the caveat that NeuronLink collectives are on-host only — multi-host training still rides EFA. That narrow win is why Anthropic is building on Trainium2; it is not the cost story anyone is running into alone.

If you want to run your own per-token numbers for the workloads your team actually serves (not Llama 70B at batch=1, which is the benchmark everyone cites because it is convenient), our LLM API Cost Calculator does the per-model arithmetic with the same per-token prices that drive the table above — useful for sizing the cost gap between chips before you commit twelve months of spend.

What the chips optimize for (and the silent failure mode when you get it wrong)

The single most expensive mistake you can make with a custom-silicon chip is to assume it is a drop-in replacement for an H100. It is not, and the failure modes do not all look the same.

Apple M3 Ultra — local only. The unified-memory architecture is a real win for single-node Apple Silicon inference — 192 GB of LPDDR5 means a 70B-class INT8 quantized model fits with no HBM cost line, and a Mac Studio at $4,000 with no power draw to speak of is the cheapest per-token story at low utilization. The silent failure mode is throughput: the M3 Ultra does not talk to a pool, and if your traffic is bursty, the Mac Studio becomes the bottleneck long before the dollars do. Treat it as a dev / small-team inference box, not as a fleet-scaling play.

AWS Inferentia2 — INT8 only. The 3.6x cost win above evaporates the moment you need BF16. Inferentia2 has no BF16 path; all inference routes through Neuron-compiled INT8 graphs. For ranking, classification, retrieval, and routing workloads that is fine. For an LLM that needs BF16 generation quality (anything that the user-facing task expects to produce nuanced language), Inferentia2 is the wrong chip. The silent failure mode is quality drift that ships: an INT8-quantized 70B model is noticeably worse than the BF16 reference, and you will not see the drop unless you measure output distribution, not just token throughput.

AWS Trainium2 — training + BF16 inference, Neuron SDK only. The on-host HBM advantage (2 TB per host lets a 70B model fit without tensor parallelism) and the BF16 throughput story are real. The deployment cost is the Neuron SDK compiler: first-time compile takes 5-15 minutes for Llama-family models, longer for unusual architectures, and the compile-error surface is thinner than CUDA's. The silent failure mode is the model that compiles but underperforms at p99 by 30% because the Neuron compiler did not discover the attention fusion you expected. Profile the model before you ship to production.

Google TPU v6 — Google Cloud only, JAX / PyTorch-XLA preferred. TPU is the most "different" of the five — Pods are connected by a proprietary interconnect (ICI), training scales by Pod membership, and the XLA compiler is the gatekeeper. The cost win over H100 is real on the workloads Google benchmarks (Llama 70B BF16 inference at ~$0.92/M), but it disappears for any workload where the XLA compiler cannot fuse the kernels well — a non-trivial fraction of state-space models and long-context attention patterns. The silent failure mode is over-budget OOMs during compile and a 2-3 day XLA-debugger rabbit hole before your model fits.

Microsoft Maia 100 — Azure-internal only. Maia is the most opaque of the five. Microsoft shipped it inside Azure in 2025, and the customer-facing surface area is "Azure OpenAI Service runs on it." There is no Maia SKU you can buy directly, no Maia price sheet, no Maia benchmark outside Microsoft's own communications. If you are an Azure customer and your workload is GPT-4-class via Azure OpenAI, you are already running on Maia. If you are not, Maia is not a path.

Meta MTIA + Iris — internal today, future AWS / external in 2027. Meta's stated roadmap in the July 10 announcement is internal-first: MTIA 450 and the eventual Iris chip serve Meta's own recommendation, ranking, and (starting in 2027) ad-generation workloads. There is no Iris SKU you can buy today. The leakage is the eventual cloud play: Meta has confirmed that an external Iris cloud offering is in design for 2027, and the AWS chip-design partnership (which produced Trainium2 via Annapurna Labs) is the obvious template. For someone planning a 12-month capacity decision in July 2026, Iris is a card to track, not a chip to buy.

The hidden cost line: SDK maturity

The cheapest chip per token is rarely the cheapest chip per engineer-month. The mature stacks — CUDA for H100, the XLA compiler for TPU, the Neuron SDK for Trainium2 / Inferentia2 — each cost you a fraction of your team's time in debugging, recompiling, and chasing version-pin mismatches. The numbers below are my best estimate of the weekly time sink each stack imposes, calibrated against the deployment teams I work with:

• NVIDIA CUDA + vLLM / TensorRT-LLM: ~5-10% of one engineer's week spent on toolchain friction. Mature, well-documented, ten-year-old error surface. The "cost" is mostly paying for the engineer who knows it well.
• AWS Neuron (Trainium2 / Inferentia2): ~10-15% of one engineer's week. Neuron SDK is functional but thinner than CUDA; the NeuronX / NKI / nkipy toolchain split (added in late 2025) is a real source of confusion; compile errors are less searchable than CUDA's.
• Google XLA + JAX / PyTorch-XLA: ~15-20% of one engineer's week on TPU-specific work. XLA's strength is fused kernels; its weakness is opaque error messages for non-compiling graphs. Long-context attention patterns frequently hit the latter.
• Apple MLX / Core ML: ~10-15% of one engineer's week on Apple Silicon. The Mac Studio is great for dev / small-team inference; production-scale Apple fleet operations are unproven — there is no equivalent of the on-call rotation you get on H100 clusters.
• Microsoft Maia SDK: Not customer-accessible. The Azure OpenAI path is well-engineered; you do not see Maia's SDK surface, and you cannot debug at the Maia layer.
• Meta MTIA / Iris SDK: Not customer-accessible yet. The nkipy / neuronx-distributed family Meta published (and which Trainium2 inherits) is the upstream signal for what an external Meta SDK would look like.

If your team is two engineers with one operator, the chip cost is less important than the SDK cost. The cheapest chip with an immature SDK will bankrupt you in human time before it makes you money in dollars.

The decision tree I walk through with clients

When a client asks "which chip," I ask four questions in this order:

1. Where does the data live? If your training data is in S3 and your serving endpoints are on EKS, Trainium2 + Inferentia2 are the lowest-friction second-vendor. If your training data is in GCS and your serving is on GKE, TPU is the obvious second chip. If your data is in Azure Blob and you are on Azure ML, Maia is not a choice but H100 is. Data gravity is the strongest predictor of vendor choice; the chip's dollar-per-token number is secondary.
2. What is your largest workload class? If the largest workload is recommendation / ranking, MTIA 450 and Inferentia2 are the right comparison; if it is open-ended generation, BF16-class chips (H100, B100, Trainium2 BF16 inference, TPU v6) are the right comparison. The wrong chip class will win on the benchmark but lose on the workload.
3. What is your INT8 tolerance? If your workload tolerates INT8 quality (classification, routing, retrieval re-ranking, safety classifiers), Inferentia2's 3.6x cost win is real and decisive. If your workload needs BF16 generation quality, the cost win is gone and you are comparing the rest of the table.
4. What is your SDK capacity? If your team already runs JAX on TPU or PyTorch + CUDA on H100, the migration cost to the other path is real but estimable. If your team is small and CUDA is the only path they have shipped, the cost of switching to TPU or Trainium2 will eat the silicon savings.

If the answer to all four questions is "the workload is open-ended BF16 generation, the data is multi-cloud, and the team has CUDA-only muscle memory," the answer is still H100. The B100, when it lands broadly, will replace it. The custom-silicon chips are real, but they have not yet unseated NVIDIA for the broadest workload class.

Where this lands for the next twelve months

The custom-silicon market that Iris enters in September is not a market that will replace H100 inside 2026. It is a market that will absorb the workloads H100 is poorly suited to: recommendation and ranking (MTIA 450), cost-optimized INT8 inference (Inferentia2), Google-native training (TPU v6), local inference (Apple M3 Ultra), and Azure-internal serving (Maia). NVIDIA's H100 still wins on broad workload fit; B100 will win harder when it scales; Iris will land as a real second-vendor option for teams with the chip-design patience to onboard it (which means most teams: through AWS, starting 2027).

For the team choosing capacity this quarter, my recommendation is unchanged from the Trainium2 / Inferentia2 article I shipped in June: H100 is still the default; Trainium2 + Inferentia2 is the most defensible second vendor if your data lives on AWS and you can absorb the Neuron SDK learning curve; TPU is the second vendor if your data lives on GCP. Iris is the chip to track but not the chip to buy in Q3 2026. The market is finally 5-way; the buying decision is still mostly 2-way plus Iris-track.

The custom-silicon bet was real in 2023 when AWS shipped Inferentia2 and Google shipped TPU v4. It is more real now that Apple, Microsoft, and Meta are in the same race. The dollar-per-token story is, at last, measurable end-to-end — not because the chips are commoditized, but because the workload classes have separated enough that each chip has a number to defend. That is the most useful trend I can point to in July 2026, and it is the trend that makes the comparison above worth your time.