Large-scale LLM training requires curated, shardable corpora that can be streamed and reweighted for different recipes. TxT360-v2 supplies Parquet-backed subsets of web and QA-style text used in the K2 Horizon release, enabling straightforward integration into training pipelines and controlled sampling across sources. The release contains roughly 1.84 billion rows (~5.29 TB) across three named subsets.
What Sets It Apart
- Parquet shards and streaming-friendly layout: each subset is provided as Parquet shards (train split) so you can stream, shard, and pipeline without large full-file downloads — useful for distributed training and cloud workflows.
- Multiple focused subsets: includes web-high-nltk-qa, web-high-medium, and txt360-qa to let teams mix and weight web content and synthetic QA data independently rather than using a single monolithic corpus.
- Provenance and preprocessing transparency: records originate as JSON objects with metadata; subsets may include source-specific filtering, deduplication, quality scoring, or synthetic QA generation, which helps downstream filtering and decontamination.
- Open licensing: distributed under CC BY 4.0, allowing reuse with attribution.
Who It's For and Trade-offs
Great fit if you need modular, large-scale pretraining corpora that can be streamed and reweighted for ablation studies or training recipes; if you plan to combine web and synthetic QA data and want Parquet-backed shards for efficient I/O. Look elsewhere if you require a fully curated, human-reviewed benchmark-quality corpus out of the box — web-derived sources can contain noise, factual errors, and sensitive content and may need additional filtering or decontamination. Also note the size and scale necessitate substantial storage and compute to use effectively.
Where It Fits
TxT360-v2 is positioned as a pretraining source collection (part of the K2 Horizon series) rather than an evaluation benchmark or an SFT-ready instruction dataset; use it as raw or lightly-processed input to LM pretraining, or as a component mixed with other curated sources for large-scale training recipes.
