Shawn Higdon (@Smiles_Higdon) header
Shawn Higdon (@Smiles_Higdon) profile picture
United States
Joined February 2016
Photo by Shawn Higdon (@Smiles_Higdon)Photo by Shawn Higdon (@Smiles_Higdon)Photo by Shawn Higdon (@Smiles_Higdon)
Shawn Higdon retweeted
AI Agents will be core infrastructure for genomic surveillance. We introduce BioSecBench-Surveillance, a verifiable benchmark for testing whether AI agents can make the analytical decisions required in these workflows. The benchmark contains 100 evaluations spanning seven task categories, six sample types, and both short- and long-read sequencing. Agents receive realistic sequencing data and sparse surveillance context, then must choose the right tools, references, thresholds, and analysis paths. As sequencing volumes increase, genomic surveillance is increasingly limited by analysis. Public health workflows depend on bespoke and sometimes tacit choices with sequence references, databases, filters, normalization, and thresholds. AI agents are promising because they can inspect files, run tools, and iterate through workflows autonomously. But surveillance is a challenging problem. Agents must chain complex scientific and analysis decisions correctly from messy biological context. We evaluated sixteen model-harness configurations across roughly 4,800 runs. Pass rates ranged from about 14% to 50%, with most frontier configurations clustered between 38% and 50%. Refusals varied sharply by harness and provider, from zero to nearly one-third of tasks. Performance varied more by task type and sequencing technology than by sample type or assay. Most task categories landed between 35% and 50%, but anomaly detection fell to 20%, with genetic-engineering characterization next at 35%. Long-read datasets were also harder, scoring 26% versus 41% for short-read datasets. Sample type, nucleic-acid target, and assay type moved performance much less: clinical and isolate samples were handled best, wastewater was somewhat worse, DNA and RNA differed only modestly, and shotgun and targeted assays were nearly identical. Agents usually found reasonable tools, but struggled with scientific judgment. The failures came from choices around how to invoke those tools in context, eg. selecting the wrong reference, threshold, normalization method, or final interpretation of biological signal. The hardest tasks were open-ended judgement calls where the agent had to decide what mattered without being told what target to look for. Anomaly detection requires deciding whether a weak signal was real or background. Genetic-engineering characterization required deciding whether a sequence pattern reflected deliberate construction rather than native or homologous biology. We are building toward a future where agents analyze surveillance data as it arrives, fast enough to shape an outbreak response while it still matters. Today’s agents might not be reliable enough to do so, but by measuring their capabilities, we get closer to this future.
3
12
3
38
7,081
Shawn Higdon retweeted
Can you read 900 words per minute? Try it.
4,674
28,432
9,002
205,237
31,926,877
Introducing 🎉sourmashconsumr 🎉, an R package that reads, parses, plots, and analyzes the outputs of the sourmash Python package. Pub: arcadia-research.pubpub.org/… GitHub repo: github.com/Arcadia-Science/s… Documentation site: arcadia-science.github.io/so… 🧵 [1/8]
2
27
3
104
17,320
sourmashconsumr does lots of things, but here's a sampling of the visualizations you can make. [2/8]
2
1
12
703
Can I get a few more followers to get to 2000? Thanks.
1
2
8
Flagellar motor of Salmonella bacteria, in complex with the the MotAB stators (Pink). MotA is a pentameric H+ channel that couples the electrochemical gradient into rotational energy. MotB dimers anchor the stator into the peptidoglycan layer. #Blender3d #b3d #sciart
29
419
92
1,856
A novel cis-element enabled bacterial uptake by plant cells biorxiv.org/cgi/content/shor… #biorxiv_plants
Shawn Higdon retweeted
Building a scientific movement in developing countries #iGEMBolivia decided to learn #synbio to solve its local problems by participating in #iGEM2021. Congratulations @igemBolivia!!🙌🤩💪🏼🧬🇧🇴
#iGEMBolivia: Un movimiento científico para el acceso a la bioinnovación blog.igem.org/blog/2022/3/16…
1
4
Inflation or recession. Pick your poison.
624
282
65
3,481
I made this figure out of my fascination with protein architectures. I call it "Molecular Machinery of Life." Thought to share it with you to enjoy your eyes. Can you detect, by heart, the Nobel Prize-Winning structures? @AcademicChatter #AcademicTwitter
35
274
33
1,488
This looks great: A GitHub resource that lists popular data cleaning, data viz and stats commands/procedures across different programming languages lost-stats.github.io/
6
243
11
1,140