codegraph 🇺🇸

If you're making software that is designed to parse complex, diverse data, especially when YOU don't control the data, and it is generated in varied ways by thousands of different organizations or users, then there's really no substitute for what's known as "fuzzing" tests. Fuzzing is a testing technique where you try to intelligently generate a huge number of varied inputs to your program, with the goal of finding parsing bugs that you'd be unlikely to come across if you're only testing with a fixed set of valid inputs. A naive approach to generating random inputs would be wasteful, since most inputs are trivially invalid and can be quickly rejected. What you really want are inputs that are pathological; ones that are nearly valid but have some bizarre issue that can confuse your parsing rules and trigger code pathways that you'd normally never traverse. Fuzzing used to be a somewhat esoteric approach used by security-focused engineers in high-stakes, sophisticated applications. But now anyone can use it. I have a skill specifically for fuzzing you can try here: jeffreys-skills.md/skills/te… It's part of a series of testing-related skills that also includes metamorphic testing, conformance harness testing, golden artifact generation, end-to-end integration testing, etc. The skill works across many popular languages and automatically installs the needed tooling for you, instruments your application for fuzzing, and then mounts extended fuzz testing campaigns. You can see an example of one in the screenshot, where it's been working for many hours, subjecting three different parsers in my hedge_fund_data_tool to torture tests. For instance, my elaborate parser for SEC insider trade forms (i.e., Forms 3, 4, and 5), which is thousands of lines of Rust, has been called over 780 million times, each invocation with a different, optimally selected input designed to best explore all the different possible code paths within the parser. There aren't that many forms of this kind in the history of the SEC's EDGAR database, across all companies. The only way to do something like that is with fuzzing. And now you can!

More and more, I’m realizing that the use of external library dependencies in Rust projects is an anti-pattern. Most libraries are one-size-fits-all, with scope creep and sprawling dependency chains that lead to you depending on hundreds of different projects and people, any one of which could suffer from a supply-chain attack. They also often rely on C or C++ libraries, which defeats the benefit of the strict memory safety that is the major selling point of Rust. This usually results in complex and annoying build processes that create headaches when you want everything to be cross-platform. And even when they don’t use C or C++ libraries, they almost always make liberal use of unsafe code. If you don’t make strict memory safety a basic requirement, it’s easy to slip into bad patterns out of convenience or a sense of misguided pragmatism. Perhaps worst of all, they are just not performance-optimized the way things can and should be today, with special tricks to optimize performance specifically on Apple silicon or on high-core-count AMD processors. Those two platforms have very different performance characteristics and need specialized designs to even approach the point of saturating the hardware. For my FrankenSearch project, which is the core of my cass tool and also used in various other projects of mine like agent mail and eidetic engine, I have been using the Tantivy library for fast lexical search. Tantivy is sort of a Rust descendant of Lucene, which is the core search system used in Elasticsearch. I realized recently that many of the things that annoy me about FrankenSearch come from Tantivy, and I hate the feeling that it’s a walled-off dependency that I don’t want to change for the obvious reasons. Since for my newest projects (e.g., FrankenOCR, FrankenMarkdown, FrankenSim, and FrankenGraphDB), I have a strict “no outside dependencies” rule other than my own projects like Asupersync and FrankenSQLite, I decided that I should start applying a similar standard in my older projects. That’s why I decided to create my own specialized lexical search system optimized specifically for FrankenSearch. I’ve made the plan for it and the implementation work is already far along. You can see the plan here: github.com/Dicklesworthstone… This will reduce the number of dependencies by 41 crates, increase safety and security, and dramatically increase performance. And because this module will be part of the core code of the system, it will benefit along with the rest of the projects from my endless continuous performance optimization campaigns, so it will keep getting faster and faster. As the plan explains, the biggest wins come from not paying the “generality tax.” By not trying to be all things to all people, we can do just what we need in the absolute optimal way possible.