I think that LLM's are a broken mirror or maybe disco ball. What you get back is almost always a reflection of what you put in, just garbled through and with the reflections of so many other people. The best parts that I often get back are mostly just the stuff that I already put in it.
I think that's also where LLMs get the most dangerous. That ELIZA effect of telling you exactly what you told it but in a slightly rephrased way makes you think that you think that your ideas are genius. You've got someone "actively listening" and telling you whatever idea you just gave it was the best idea it ever heard.
That's not what you want from a serious writing partner. You want a writing partner to challenge you, to point out your mistakes and flaws, to make you better, to offer a truly different perspective to your own. You don't want a dark reflection from a "yes man" to spiral into.
I think it is a net positive for me, because I am super unconfident by nature and the agreeable, almost sycophantic output from major LLMs give me just enough of a boost to debias me. They are certainly better than a physical rubber duck.
I feel it is a net negative for me. One of my hobbies has always been fiction writing. The LLM reflections of my own writing become slot machines to me for cheap dopamine hits. It's not because LLMs are good at fiction writing, far from it, it's that LLMs are good at reflecting my own writing back to me in fun house mirror ways that amuse me and frustrate me and pleasantly surprise me.
That slot machine addiction gives me a lot of perspective on what LLMs are good at in coding versus what they appear to be good at. It's easy to characterize them as "over-eager junior developers" or "powerful rubber duck debuggers" but their biases are still your biases so long as you are the one writing the prompts. A lot of the "surprises" aren't intelligence but fun house mirror versions of the same biases you input into the system.
I keep referring to their coding capabilities as "legacy code as a service" and the addiction to the fictional equivalent only underscores that understanding of LLMs. You don't write the next great American novel getting a word salad of past great novels back full of the best and worst of your own writing. You don't write the next great American software getting a legacy code jumble of past great software back full of the best and worst of your own prompting.
At the end of the day I'm still a pragmatist and using the LLMs in my programming efforts, but I don't trust them as a partner, I use them as a tool. (A biased tool that I know needs a careful eye.)
I think that's also where LLMs get the most dangerous. That ELIZA effect of telling you exactly what you told it but in a slightly rephrased way makes you think that you think that your ideas are genius. You've got someone "actively listening" and telling you whatever idea you just gave it was the best idea it ever heard.
That's not what you want from a serious writing partner. You want a writing partner to challenge you, to point out your mistakes and flaws, to make you better, to offer a truly different perspective to your own. You don't want a dark reflection from a "yes man" to spiral into.