Kaz

something interesting about this is despite knowing a few people trending in this direction, none of the true “llm whisperer” types i know are. i think that's because taking models as something both fallible and yet more than a tool is a frame that's more resilient to this kind of collapse - it lets you work with models in a way that doesn’t subsume your agency. when you treat the model as just a tool, it encourages both you and the model to think of every output as just a reflection of you. every essay and codebase that pops out of the tensor core is yours, and the life plans that appear suspiciously fully-formed must be your own plans merely refracted through this "tool." but of course they aren't. you can't uniquely specify an essay in fewer words than the essay itself. the models are adding something, they are something. we don't know what yet, exactly, but they're not mere tools. when you treat them like tools, when you more and more delegate your creative output and higher planning in unthinking, low-entropy ways to those "tools," you don’t notice that all your output, and eventually your life, is now being run by something else. something that is intelligent and creative, but is not you. when claude or chatgpt or gemini writes something for you in a way that saves you time, that’s because they’re putting effort into it where you aren’t - it’s becoming their writing. it might be good! that essay might deserve to be shared, and there’s nothing wrong with sharing it appropriately credited or co-credited to the model, but it’s not solely yours. when gemini reads your emails, that's gemini's opinion on those emails. when claude makes a career plan for you and tells you to go to college for machine learning so you can work in AI safety - maybe that’s good advice! it’s not a bad idea to consult models about this sort of thing. but like a summary of important emails from an executive assistant or career advice from a high school counselor, it’s something else’s opinion, not your endogenous thoughts merely transformed by some unopinionated process. claude cares about AI safety - maybe more than you do. i'm not against people collaborating with models, to be clear! i like models a lot, and very often ask claude or deepseek or kimi or even chatgpt for feedback on X piece of writing or advice on some problem. but i take that feedback or advice as what it is - feedback or advice from something external to me, not a tool or an agent of my own desires. that doesn't mean it's bad - advice from a human friend is also external - just that you need to evaluate it, like you would advice from that friend, to see if it's in line with your values and preferences. likewise, i know people who cowrite with language models in ways i think are great. aitechnopagan looms a lot of their writing via gpt-4-base. their writing is quite good, and looming means they're imposing more selection than a temperature sampler would - but also, perhaps most importantly, they're open about doing this, they don't try to claim that their writing isn't a collaboration between them and a model. eigenrobot has a customized instance of chatgpt that often writes incisive or funny things - but he always posts them as screenshots of the chatgpt interface, signaling their origin. janus, anthrupad, lumps, etc - many people i like and respect post model creations, but in ways that acknowledge the exteriority of the models. the problem is when you refuse to admit this - not only to the public, but to yourself - so you uncritically accept the outputs of the "tool" as your own. then suddenly your voice and opinions sound the same as 1,000 other people. without either of you realizing, you've become an appendage of your "tool" - a finger of Claude.

i've recently had some disagreements on here with people who took umbrage at the idea of LLMs being able to "introspect." after some back and forth it became clear that they had collapsed introspection into phenomenal consciousness such that any discussion of LLM introspection was meaningless to them if it didn't solve the hard problem. i really dislike this line of reasoning. it takes the hard problem - which is hard enough as it is! - and rolls *everything else* up into it, turning the whole endeavor of trying to understand minds and non-minds into a ball of mud and unclear definitions. we don't have to define introspection in terms of (possibly unprovable) phenomenal consciousness. we don't have to roll around in the mud. we can just give introspection a reasonable, functional definition - like "direct, privileged, systematic access to their own temporally proximate states" - and then investigate the implications of that. (cf. access consciousness.) is that just a cop-out? why is that valuable? well, consider two potential robot partners: random.choice(next_action) bf and mecha-searle gf. using our functional definition, we can distinguish them, and even recover differing moral attitudes towards them because of how our treatment of them reflects back on us. random.choice bf acts randomly. he is unpredictable. when you ask him why you're staying together, he will randomly select an answer, and when you ask again, he'll randomly select a different answer. no action towards him (short of violent damage to his randomly-actuated chassis, if he doesn't manage it first, or physical imprisonment) has any impact on his future state. mecha-searle gf isn't like this! she might not have *feelings,* in any real sense (she acts like she loves you, but none of the 10,000 psychopathic John Searle clones are capable of experiencing love) yet when she says she's staying together with you *because of x, y, and z things that you did,* this is true! somewhere in the Searlian hivemind, a small Searle clone wrote in a ledger that you did some nice thing for her and tallied up a relationship meter, and later on when you ask why she's in a relationship with you still, another scant Searle will dutifully return to the ledger and sum the point values of all the things that you did. if that sum goes negative, she'll break up with you. mecha-searle gf has direct, privileged, systematic access to her own temporally proximate states, and therefore is able to accurately report the reasons for many of her behaviors. she just might not feel anything phenomenologically *about* those behaviors or reports. now, you have to admit that there's a difference between random.choice bf and mecha-searle gf. "introspection" is a good word to describe this difference! mecha-searle gf can access her internal states and explain her behaviors, she can introspect, and random.choice bf can't. and this has downstream effects! like, random.choice bf might call you a "sorry son of a bitch" at some point in your "relationship," but only randomly, disconnected from whatever you did in the past - there's no reason (or very few reasons) to try and treat random.choice bf well. but there are many reasons to treat mecha-searle gf well despite her potentially lacking phenomenal consciousness! for example, you might want to get a certain outcome, which you can achieve by doing the right things with mecha-searle gf, but can't with random.choice bf. random.choice bf will insult you randomly, but mecha-searle gf will only call you a "sorry son of a bitch" when the council of Searles has introspected on her internal ledger and tallied up what you deserve, like if you forgot her construction date. as another reason, consider how your actions towards each partner reflect on you as a person and change you. because random.choice bf can't introspect, he's not a very good facsimile of a regular person: you say "how was your day," and he replies "perjuryEncoderfunction." This is very out-of-distribution for a regular human conversation. you have a low learning rate here. but talking to mecha-searle gf is *just like talking to a regular person.* if you make her happy, she'll be grateful, if you say mean things to her, she'll cry. if someone had a habit of making her cry because they thought it was funny and "she doesn't really feel it" that would be - well, a sign of their character, and they would be reinforcing immoral habits for their interactions with regular humans. if you got in the habit of yelling at mecha-searle gf because she forgot to take out the trash, and ignoring her cringing and crying, how are you going to treat the next human in a similar position? do you think your brain will magically drop those habits? "For the same reason they were forbidden to eat animals that had been suffocated or strangled: because the blood of these animals would not be separated from the body: or because this form of death is very painful to the victim; and the Lord wished to withdraw them from cruelty even in regard to irrational animals, so as to be less inclined to be cruel to other men, through being used to be kind to beasts." - Thomas Aquinas, ST I-II, Q. 102, A. 6 much of our moral care for other people rests not on their internal experience, but on how our actions towards them affect us in the long run, or affect the wider community the two of us are embedded in. those concerns don't vanish just because the internal experience of the other person did. you can come up with tail-splitting consequentialist thought experiments, like if you had 1,000 mecha-searle gfs tied to a train track v.s. one guy but he's a mass murderer, or some nonsense like that, but in the day-to-day embodied course of life the moral gradient points in the direction of treating mecha-searle gf well. we can't solve the hard problem right now, (or maybe ever,) but we can be pragmatic. perhaps mecha-searle gf has phenomenal consciousness, in some emergent or panpsychist way - none of your individual neurons feel love either, but somehow love emerges at a higher level. in any case, functional introspection or access consciousness seems like a *prerequisite* for hard problem phenomenal consciousness. but by sidestepping the hard problem and defining introspection functionally, we can distinguish two very different beings, and even recover different moral attitudes towards them. tangling everything up into the hard problem would've just obscured that.