A family of open code models (1.3B-33B) trained from scratch on 2T tokens of project-level code, using a 16K-window fill-in-the-blank objective. Beats Codex and GPT-3.5 on code benchmarks and ships under a license permitting commercial use.
AI Train2024
Runs GPT-4o-class vision, speech, and full-duplex audio-video conversation on a 9B model small enough to deploy on phones and tablets. The 4.5 release scores 77.6 on OpenCompass and adds real-time bilingual voice with voice cloning.
Reaches 51.7% on the competition-level MATH benchmark with a 7B model and no tools or voting, rivaling Gemini-Ultra and GPT-4. Built on a 120B-token math corpus mined from Common Crawl, and introduces GRPO, a memory-efficient PPO variant for reasoning.
AI Dataset2024
Provides a multilingual, deduplicated corpus of public source code in Parquet for large-scale model training and evaluation. Includes license metadata, language splits, and streaming-friendly packaging for use with Hugging Face Datasets — suited to training code-focused foundation models but requires careful license/provenance review.
AI Dataset2024
Provides 300k annotated multilingual text examples for identifying and masking personally identifiable information (PII) across multiple domains and languages (EN, FR, DE, IT, ES, NL). Intended for training and evaluating token-level PII detection and masking models; includes a DOI for citation.
Hands-on coding tutorial series for large language models with slides and runnable notebooks covering fine-tuning, prompting, RLHF, safety, steganography, watermarking, multimodal models, GUI agents, and deployment. Community-maintained, free course materials for students and researchers.
AI Dataset2024
Provides a cleaned, deduplicated English web corpus optimized for LLM pretraining—over 15T tokens aggregated from CommonCrawl with per-dump snapshots and smaller sampled configs (10B/100B/350B). Includes the datatrove processing pipeline, MinHash deduplication, and an ODC-By v1.0 license; suited for large-scale model training and ablation studies but not specialized for code.
MLOps2024
Runs reproducible evaluations of large language models through a Python API with built-in solvers, scorers, and model-graded grading. Ships 200+ ready-to-run evals spanning capability and safety testing, and connects to most major model providers.
AI Dataset2024
Provides ~1.3 trillion tokens of web pages filtered for educational quality using an LLM-trained classifier; includes per-Crawl configs, smaller random samples (10B/100B/350B tokens), and the classifier code and model for reproducible filtering.
A 236B-parameter Mixture-of-Experts LLM that activates only 21B parameters per token, cutting training cost 42.5% versus a dense 67B model and shrinking the KV cache 93.3% via Multi-head Latent Attention, with 128K context.
Official code companion to the O'Reilly book by Jay Alammar and Maarten Grootendorst: 12 chapters of runnable notebooks on tokens, embeddings, Transformers, text classification, clustering, prompt engineering, semantic search, RAG, and fine-tuning.