Jimi

Wow @tan_stack Start has server components now! And without « use client »/« use server » It happens within a serverFn so the code explicitly split the server only stuff, no directives needed! I much prefer this approach 🔥

I just released Kiru: a text chunking lib, 1000x faster than LangChain splitters. Built in Rust for Python. If you're building RAG systems, you know the basic drill: load documents, split them into chunks, feed them to your vector database. Simple enough, except most chunking libraries are painfully slow when you're processing millions of documents. kiru solves this bringing Rust native performance to Python via PyO3 bindings. The numbers: - 4000+ MB/s for byte chunking (Pure Rust) - 1,400+ MB/s for byte chunking (Python bindings) - 300+ MB/s for character-aware chunking - Constant memory usage, even on gigabyte files - Works with files, URLs, and glob patterns out of the box Why it's fast and memory efficient: Most chunkers load entire files into memory, then split them. Kiru streams data in blocks, chunks on-the-fly, and never breaks UTF-8 boundaries. The Rust implementation hits 4,000+ MB/s (for in memory string and byte chunking), and even the Python bindings crush slower pure Python implementation. Byte vs Character chunking: Byte chunking splits on byte boundaries (faster, perfect for token-limited models). So for a chunk of 500 bytes we just lookup the 500th byte in the sequence, if it's not a valid character boundary we take the closest one. Character chunking splits on exact characters count. It's slower because you need to compute all the character boundaries (characters can be multi-bytes). Both respect UTF-8 boundaries, so you never get corrupted text. Why it matters: When you're ingesting your entire knowledge base or processing documents in real-time, speed isn't just nice, it's the difference between a system that scales and one that doesn't. What's next: This is just the first release. Coming soon: 👉🏽 Delimiter-based splitting (newlines, sentences, paragraphs) 👉🏽 Recursive splitting with multiple delimiters 👉🏽 Markdown-native chunking (preserve structure) 👉🏽 Sitemap support (chunk entire websites from a single URL) 👉🏽 Code-aware splitting powered by tree-sitter Available now on PyPI and Rust Crates. Fully open source. Links below 👇🏽

😅

Putting it together. Each step is composable and testable.

Every step can fail: - File not found - Invalid YAML frontmatter - MDX compilation error - Schema validation error Effect's type system tracks all these.

Finally, validate against the user's schema. If validation fails, Effect gives us a detailed error with the exact field that's wrong. Type safety from file → database!

Merge frontmatter with compiled output. Now we have everything in one object!

If the user wants exports (like metadata, TOC) we extract them from the MDX content.

For each file, bundle the MDX with mdx-bundler. Effect.tryPromise converts Promise errors to typed errors.

First, stream all .mdx files from the folder. streamFiles is a helper that uses Effect's FileSystem service.

The loader returns an Effect Stream. Streams let us process files one-by-one without loading everything into memory.

Relations are first-class, and 3 types are supported: - single relations - array relations - map relations

Now whenever you need your data, jsut use the CMS service and: - getById -> Option<Collection<T>> - getAll -> Collection<T>[] - loadRelation -> map, array or single item

At build time run the build effect to build your CMS, it will: - process all the content - run your transformations - run your validations - check that all relations exist - output to the content store (sqlite if you use the SQLContentStore)

Create the minimum needed layers: - A Content Store (the lib comes with an SQL implementation that needs and SQLite client) - A Node or Bun context for filesystem access

Then define your CMS by providing all the collections.