I Went Hands-on With Chatgpt Codex And The Vibe Was Not Good - Here's What Happened

5 hours ago

ZDNET's cardinal takeaways

ChatGPT Codex wrote codification and saved maine time.
It besides created a superior bug, but it was capable to recover.
Codex is still based connected nan GPT-4 LLM architecture.

Well, vibe coding this is not. I recovered nan acquisition to beryllium slow, cumbersome, stressful, and incomplete. But it each worked retired successful nan end.

ChatGPT Codex is ChatGPT's agentic instrumentality dedicated to codification penning and modification. It tin entree your GitHub repository, make changes, and rumor propulsion requests. You tin past reappraisal nan results and determine whether aliases not to incorporated them.

Also: How to move your codebase into GitHub for study by ChatGPT Deep Research - and why you should

My superior improvement task is simply a PHP and JavaScript-based WordPress plugin for tract security. There's a main plugin disposable for free, and immoderate add-on plugins that heighten nan capabilities of nan halfway plugin. My backstage improvement repo contains each of this, arsenic good arsenic immoderate attraction plugins I trust connected for personification support.

This repo contains 431 files. This is nan first clip I've attempted to get an AI to activity crossed my full ecosystem of plugins successful a backstage repository. I antecedently used Jules to adhd a feature to nan halfway plugin, but because it only had entree to nan halfway plugin's unfastened root repository, it couldn't return into relationship nan full ecosystem of products.

Earlier past week, I decided to springiness ChatGPT Codex a tally astatine my code. Then this happened.

GPT-5 released

On Thursday, GPT-5 slammed into nan AI world for illustration a freight train. Initially, OpenAI tried to unit everyone to usage nan caller model. Subsequently, they added bequest exemplary support erstwhile galore of their customers went ballistic.

I ran GPT-5 against my group of programming tests, and it grounded half of them. So, I was peculiarly funny astir whether Codex still supported nan GPT-4 architecture aliases would unit developers into GPT-5.

However, erstwhile I queried Codex 5 days aft GPT-5 launched, nan AI responded that it was still based connected "OpenAl's GPT-4 architecture."

I took 2 things from that:

OpenAI isn't fresh to move Codex coding to GPT-5 (which, recall, grounded half my tests).
The results, conclusions, and screenshots I took of my Codex tests are still valid, since Codex is still based connected GPT-4.

With that, present is nan consequence of my still-very-much-not-GPT-5 look astatine ChatGPT Codex.

Getting started

My first measurement was asking ChatGPT Codex to analyse nan codebase. I utilized nan Ask mode of Codex, which does analysis, but doesn't really alteration immoderate code.

I was hoping for thing arsenic heavy and broad arsenic nan 1 I received from ChatGPT Deep Research a fewer months ago, but instead, I received a overmuch little complete analysis.

I recovered a much effective attack was to inquire Codex to do a speedy information audit and fto maine cognize if location were immoderate issues. Here's really I prompted it.

Identify immoderate superior information concerns. Ignore plugins Anyone With Link, License Fixer, and Settings Nuker. Anyone With Link is successful nan very early stages of coding, and is not fresh for codification review. License Fixer and Settings Nuker are specialty plugins that do not request a information audit.

Codex identified 3 main areas for improvement.

All 3 areas were valid, though I americium not prepared to modify nan serialization information building astatine this time, because I'm redeeming that for a full preferences overhaul. The $_POST title is managed, but pinch a different attack than Codex noticed.

Also: The champion AI for coding successful 2025 (and what not to use)

The 3rd area -- nan nonce and cross-site petition forgery (CSRF) consequence -- was thing worthy changing correct away. While entree to nan personification interface for nan plugin is assumed to beryllium wished by login role, nan plugins themselves don't explicitly cheque that nan personification submitting nan plugin settings for action is allowed to do so.

That's what I decided to induce Codex to fix.

Fixing nan codification

Next up, I instructed Codex to make fixes successful nan code. I changed nan mounting from Ask mode to Code mode truthful nan AI would really effort changes. As pinch ChatGPT Agent, Codex spins up a virtual terminal to do immoderate of its work.

When nan process completed, Codex showed a diff (the quality betwixt original and to-be-modified code).

I was heartened to spot that nan changes were rather surgical. Codex didn't effort to rewrite ample sections of nan plugin; it conscionable modified nan mini areas that needed improvement.

In a fewer areas, it dug successful and changed a fewer much lines, but those changes were still beautiful circumstantial to nan original prompt.

At 1 point, I was funny to cognize why it added a caller foreach loop to iterate complete an array, truthful I asked.

As you tin spot above, I sewage backmost a reasonably clear consequence connected its reasoning. It made sense, truthful I moved on, continuing to reappraisal Codex's projected changes.

All told, Codex projected making changes to 9 abstracted files. Once I was satisfied pinch nan changes, I clicked Create PR. That creates a propulsion request, which is really immoderate GitHub personification suggests changes to a codebase. Once nan PR is created, nan task proprietor (me, successful this case) has nan action to o.k. those changes, which adds them into nan existent code.

It's a bully mechanism, and Codex does a cleanable occupation of moving wrong GitHub's environment.

Once I was convinced nan changes were good, I merged Codex's activity backmost into nan main codebase.

Houston, we person a problem

I brought nan changes down from GitHub to my trial instrumentality and tried to tally nan now-modified plugin. Wait for it…

Yeah. That's not what's expected to happen. To beryllium fair, I've generated my ain stock of correction screens conscionable for illustration that, truthful I can't really get angry astatine nan AI.

Instead, I took a screenshot of nan correction and passed it to Codex, on pinch a punctual telling Codex, "Selective Content plugin now fails aft making changes you suggested. Here are nan errors."

It took nan AI 3 minutes to propose a fix, which it presented to maine successful a caller diff.

I merged that alteration into nan codebase, erstwhile again brought it down to my trial server, and it worked. Crisis averted.

No vibe, nary travel

When I'm not successful a unreserved and I person nan time, coding tin supply a very pleasant authorities of mind. I get into a benignant of travel pinch nan language, nan machine, and what seems for illustration a relationship betwixt my fingers and nan computer's CPU. Not only is it a batch of fun, but it tin besides beryllium emotionally transcendent.

Working pinch ChatGPT Codex was not fun. It wasn't hateful. It conscionable wasn't fun. It felt much for illustration exchanging emails pinch a peculiarly recalcitrant contractor than having a gathering of nan minds pinch a coding buddy.

Also: How to usage GPT-5 successful VS Code pinch GitHub Copilot

Codex provided its responses successful astir 10 aliases 15 minutes, whereas nan aforesaid codification would astir apt person taken maine a fewer hours.

Would I person created nan aforesaid bug arsenic Codex? Probably not. As portion of nan process of reasoning done that algorithm, I astir apt would person avoided nan correction Codex made. But I undoubtedly would person created a fewer much bugs based connected mistyping aliases syntax errors.

To beryllium fair, had I introduced nan aforesaid bug arsenic Codex did, it would person taken maine considerably longer than 3 minutes to find and hole it. Add different hr aliases truthful astatine least.

So Codex did nan job, but I wasn't successful flow. Normally, erstwhile I codification and I'm wrong a peculiar record aliases subsystem, I do a batch of activity successful that area. It's for illustration cleaning day. If you're cleaning 1 portion of nan bathroom, you mightiness arsenic good cleanable each of it.

But Codex intelligibly useful champion pinch small, elemental instructions. Give it 1 people of change, and activity done that 1 alteration earlier introducing caller factors. Like I said, it does activity and it is simply a useful tool. But utilizing it decidedly felt for illustration much of a chore than programming usually does, moreover though it saved maine a batch of time.

Also: Google's Jules AI coding supplier built a caller characteristic I could really vessel - while I made coffee

I don't person tangible trial results, but aft testing Google's Jules successful May and ChatGPT's Codex now, I get nan belief that Jules is capable to get a deeper knowing of nan code. At this point, I can't really support that assertion pinch a batch of data; it's conscionable an impression.

I'm going to effort moving different task done Jules. It will beryllium absorbing to spot if Codex changes overmuch erstwhile OpenAI feels safe capable to incorporated GPT-5. Let's support successful mind that OpenAI eats its ain canine nutrient pinch Codex, meaning it uses Codex to build its code. They mightiness person seen nan aforesaid iffy results I recovered successful my tests. They mightiness beryllium waiting until GPT-5 has baked for a spot longer.

Have you tried utilizing AI coding devices for illustration ChatGPT Codex aliases Google's Jules successful your improvement workflow? What kinds of tasks did you propulsion astatine them? How good did they perform? Did you consciousness for illustration nan process helped you activity much efficiently? Did it slow you down and return you retired of your coding flow?

Do you for illustration giving your devices small, surgical jobs, aliases are you looking for an supplier that tin grip big-picture architecture and reasoning? Let america cognize successful nan comments below.

You tin travel my day-to-day task updates connected societal media. Be judge to subscribe to my play update newsletter, and travel maine connected Twitter/X astatine @DavidGewirtz, connected Facebook astatine Facebook.com/DavidGewirtz, connected Instagram astatine Instagram.com/DavidGewirtz, connected Bluesky astatine @DavidGewirtz.com, and connected YouTube astatine YouTube.com/DavidGewirtzTV.