AI Dataset2024
Provides a multilingual, deduplicated corpus of public source code in Parquet for large-scale model training and evaluation. Includes license metadata, language splits, and streaming-friendly packaging for use with Hugging Face Datasets — suited to training code-focused foundation models but requires careful license/provenance review.
AI Infra2024
Open-weights 314B-parameter Mixture-of-Experts language model (8 experts, 2 active per token, 8,192-token context) released under Apache 2.0. Ships a raw JAX checkpoint plus reference inference code; needs heavy multi-GPU memory to load.
AI Dataset2024
Provides 300k annotated multilingual text examples for identifying and masking personally identifiable information (PII) across multiple domains and languages (EN, FR, DE, IT, ES, NL). Intended for training and evaluating token-level PII detection and masking models; includes a DOI for citation.
AI API2024
Provides local inference, fine-tuning, and a server/CLI for vision–language and omni (image/audio/video) models via MLX. Supports multi-image chat, audio/video inputs, activation quantization (CUDA), TurboQuant KV cache, and LoRA/QLoRA fine-tuning for on-device workflows.
AI Dataset2024
Provides a cleaned, deduplicated English web corpus optimized for LLM pretraining—over 15T tokens aggregated from CommonCrawl with per-dump snapshots and smaller sampled configs (10B/100B/350B). Includes the datatrove processing pipeline, MinHash deduplication, and an ODC-By v1.0 license; suited for large-scale model training and ablation studies but not specialized for code.
AI Train2024
Accelerates video generation with a unified framework for inference, finetuning, LoRA, distillation, sparse attention, and distributed execution for research and demos.
AI Train2024
Pretrained foundation model for zero-shot point and quantile forecasting on unseen time series, no per-dataset training. Decoder-only, trained on 100B real-world time-points; v2.5 (200M params) handles up to 16k context and 1k-step horizons.
AI Client2024
Runs open LLMs entirely on your own machine — discover and download models from Hugging Face, chat in a desktop GUI, or expose an OpenAI-compatible local server. Native Apple MLX and llama.cpp backends; headless deploy via llmster.
AI Image2024
Segments each PDF page into 11 labeled regions — titles, tables, formulas, figures, footnotes and more — and recovers reading order. Offers two engines: an accurate VGT visual model (~0.96 F1) or a faster CPU-only LightGBM ensemble.
AI Train2024
Streamlines the full lifecycle of foundation models — data prep, fine-tuning (SFT/LoRA/QLoRA/GRPO), evaluation, and deployment — with ready-to-run recipes, multi-engine inference support, and cloud/CLI workflows for both laptop experiments and large-scale runs.
AI Dataset2024
Provides ~1.3 trillion tokens of web pages filtered for educational quality using an LLM-trained classifier; includes per-Crawl configs, smaller random samples (10B/100B/350B tokens), and the classifier code and model for reproducible filtering.
AI Train2024
A community speedrun to train a 124M GPT as fast as possible on 8 H100s, all chasing a fixed 3.28 FineWeb loss. Successive records cut the run from llm.c's 45 minutes to under 1.4, mostly via the new Muon optimizer rather than more hardware.