I’ve been able to find a workflow where it’s mostly correct and can handle most of my gamedev related coding without making too many mistakes. I still have to actually read through the code and pay some attention to what it’s doing, and if it misses something and goes on a wild goose chase, it’s unusable and I have to start over (so someone who didn’t know what they are doing would be cooked), but whatever.
Sure, it does require a lot of looping adversarial reviews, and my average token cost is around 3000$ a month (we have unlimited budgets and a pretty accurate tracking, and also definitely cheaper than consumer prices per token with how large company it is), which is actually more than my monthly salary, but it’s just a job, for a company and on a product I don’t really care about, and I can 1) keep slacking in my job while doing my own coding stuff and projects, and keep seeing how absolutely unreasonable the prices are if you want to get at least semi-submitable results.
Is it worth it? Lol, no. The whole team is loosing codebase knowledge, we’re getting bottle-necked by pending PR reviews that are just stacking up and no one wants to do, so we’re not even more effective, the cost is absolutely absurd and in no way near sustainable.
And that’s while the whole industry is in the “Uber pricing” phase, so it will get a lot worse. But yeah, if you can burn 100-200$ per a simple implementation task, then it can have a pretty usable results. And that 3000$ a month does not include our CI review bot, that does additional rounds of multi-agent council reviews.
realitista@lemmus.org 2 weeks ago
I’m curious what you are using. The free versions of chatgpt have been like that for me, but even Gemini flash with extended thinking, also free for a while longer, is giving me pretty reliable results as long as there training data out there to derive an answer from.
tyler@programming.dev 2 weeks ago
Claude can one shot tasks until you get a larger system then it completely shits itself. These models are nothing more than autocomplete, and they can’t hold large systems in their heads. Anthropic literally tried to rewrite all of bun using Claude, they said they did it and yet it still hasn’t released six months later.
Not_mikey@lemmy.dbzer0.com 2 weeks ago
This has gotten a lot better for me by having a “send out scouts” skill that has a lower tier model agent search through the codebase before it starts to plan. Has handled my companies giant monolith pretty well and even can handle cross repo features as well.
Claude code has been using the new rust bun for ~5 months now and has been working fine, and bun 1.4 that released in August is using the rust rewrite
ThirdConsul@lemmy.zip 2 weeks ago
It is worth noting that the repo had multiple Anthropic employees commiting to it after the bun rewrite and before release, and that so far the only successful use cases that Anthropic or OpenAi showed are… rewrites from one language to another given sufficiently large test suites.
realitista@lemmus.org 2 weeks ago
Yeah I certainly wouldn’t advocate building a whole business around code it wrote. But for small personal tasks it hasn’t let me down. Building custom server applications, desktop applications, Firefox add-ons, upgrading my homeassistant 10 versions over a couple weeks without letting anything break. These sort of things it handles pretty easily.
treadful@lemmy.zip 2 weeks ago
I’ve not yet fucked with Claude. I don’t want to pay for it, and I really don’t like the surveillance aspect of these centralized systems. Mostly I’m using Gemini, whatever DDG had in their search results, and local models I’ve been fiddling with (like Qwen3.8 right now).
All more or less garbage once I get into the details of anything on the edge of my expertise.
SnotFlickerman@lemmy.blahaj.zone 2 weeks ago
I got a free year of perplexity.ai which gives me limited access to claude sonnet and yeah, its honestly pretty good.
RyanDownyJr@lemmy.world 2 weeks ago
I’ve been using OpenCode with whateverthefuck free models they have listed on there and they all seem to do fine with agentic tasks like building me scripts or executables to make my work tasks easier.
I used Gemini at the start with “Frontier Knowledge” and it seemed to do worse than the ones listed on OpenCode, but maybe that’s because i could only do like three prompts a week since I refuse to pay into an AI.
end of the day, its just LLMs writing code for me, but I cannot see how this would be useful for a large scale code base, but also #NotAProgrammer.
Doomsider@lemmy.world 2 weeks ago
More like #Youwillknowmorethanfutureprogrammers
realitista@lemmus.org 2 weeks ago
Are you using Gemini in flash extended thinking? (Not flash light) . I haven’t had many hallucinations other than cases where the training data it needs just doesn’t exist (cases where I can’t find the answers by googling either)
ThirdConsul@lemmy.zip 2 weeks ago
I mean as long as the coding tasks are simple, and seeded with enough context, then sure. As in “create tinder clone works”, but I still have to check the output in my company codebase. We are leveraging llms a lot for code writing, full agentic pipelines, loops, all the shiny new approaches.
The outcomes in all cases are still mediocre or below.
We have twice as many open prod bugs as a year ago.
ch00f@lemmy.world 2 weeks ago
How do you verify that everything it tells you is correct?
riskable@programming.dev 2 weeks ago
You check the links/references it gives you.
Gemini does a pretty good job of this because it doesn’t seem to have much built-in knowledge. Instead, it just searches the Internet on your behalf and returns summarized results with links to where it got that specific information.
I use it to search for scientific research all the time and the summaries often aren’t detailed enough so I actually click on those links. I’ve yet to encounter a situation where it fucked that up (invented links that don’t exist) but I have heard about it happening.
So far, the summaries have seemed to be pretty spot-on when it comes to biology papers 🤷
Psythik@lemmy.world 2 weeks ago
By writing instructions to insist that it double verifies every (non obvious) claim with a minimum of two sources. I also told mine to always assume that the initial prompt is missing crucial context, and to ask as many follow-up questions as necessary until it has enough information to provide the answer to the question I’m really asking. (For speed and efficiency you can even make it give you multiple choice options to click on.) Because sometimes the problem isn’t with the LLM, but with the user asking the wrong questions.
Using those two instructions alone, I’ve encountered considerably fewer hallucinations, and when I’m still not certain, I can simply click on the sources linked next to every single claim the AI makes.
ch00f@lemmy.world 2 weeks ago
Right, but that sounds like you implicitly trust the LLM and only verify statements when they seem off. So your intuition is the final arbiter if truth?
StabbingSky@lemmy.zip 2 weeks ago
It definitely has an element of garbage in, garbage out.
Frettchen@lemmy.world 2 weeks ago
I use it professionally. You are talking to a toddler or a professional and you never know. It’s like an interview where the person tries to get a job but you want to use it in production.