Might steal one or two things for @SpacefastHQ ☺️
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
32
Omarchy 准备让 AI 每天检查夜间构建了，任务跑到哪、要不要取消，直接在看板上看。想到发版前那些手工点到眼花的测试，这种活能交出去，看着都解压。
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
43
See QA post xitter.cf/ThePrimeagen/status/21…
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
123
You can just vibe code your own SaaS... – at New Zealand
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
1
3
142
QA / Testing work being done by a team of clankers is an amazing time saver. I’m doing something similar with LCOS. A small team of bots running nearly round-the-clock tests on both individual components and the system as a whole. Code reviews, scripted tests, UI-driven tests… the works. Bots find the issues, a human evaluates the issues and decides what to fix (or not, with guidance), the bots fix the issues… rinse, repeat. Not only is the velocity of testing & development astronomically fast… but the quality is exceptionally high. This process doesn’t remove the need for human, hands-on testing… but the benefits are tremendous.
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
2
8
109
2,441
even primagen uses Oligarchy
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
31
Omarchy QA has agents deciding what counts as good, yet most of the work he lists is old CI plumbing: job and suite state machines, cancellation, a live dashboard. Agent testing looks like build-server engineering with a model as the assertion step.
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
28
Omarchy QA’yı nightly derlemelere bağlamak iyi bir ilerleme; ama ajanların “geçti” kararını da sınamak gerekiyor. Bilerek hatalı senaryolar, yanlış kabul ve yanlış ret ölçümleri olmadan gece raporu güven vermeyebilir.
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
20
You wouldn't believe the hours I spent manually testing and verifying Omarchy releases during the first year. With Prime's stellar work leading the QA effort, we can finally replace all that toil with agents 🤘
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
105
51
9
2,197
110,529
to me this is the way I am coding the most nowadays. you set up an experiment with a verifiable enough output and then just let the model hillclimb your objective. if you have a verifiable objective then the model can comfortably just keep on getting you better metrics. as long as you take care of the reward hacking I feel like you get so much better outputs
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
1
196
Prime, you so rock, sir.
ThePrimeagen
@ThePrimeagenOct 19
Big Omarchy QA News: The Headline: * Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff. The Meat: Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good. * A nice running dashboard with cancellable states * Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility * suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime... ** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast... *** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP. * ability to follow live running jobs and cancel them The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production! Will think about how to make this a public site, but for now, its private :(
15
317
Load more