Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly.
If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here.
> Have experience shipping production full-stack products across modern web frontends and backend services.
I'm amazed that the requirements are so low (or at least this vague) for jobs at companies like these.
Has anyone else had the experience of going to an interview and feeling like you were never asked any qualifying questions?
All the questions were easy, your answers were straightforward, you "got them right", but then were not chosen?
I find on the other side, they're also left with dozens of people who "passed" and then it comes down to a pretty arbitrary decision on who gets hired (if we are talking external, no referral, etc.)
I wonder if they can make job descriptions highly specific to filter the shortlist faster and more effectively (to actually get a shortlist).
How does it deal with the current guardrails 5.6 Sol has on finding vulnerabilities? When I use it in the Codex app it would sometimes say it found a vulnerability, but it cannot tell me what it is.
The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration.
We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build.
Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.
I tried it, it started a scan but stopped after hitting the rate-limit of my account. It gave up after just a minute of retrying (rate limits are tokens per minute, so... :P).
It said "Partial output was kept at <...>", but I dont see a obvious way of picking it up in a new scan? (The failed run cost me ~$13)
We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.
> co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
> Thanks for checking this out and for flagging the auth issues.
Offtopic, but this right here is why I don't believe any marketing around "great amazing models that one-shot everything and programmers are no longer needed".
You just have to look at what these labs routinely produce, and their own products.
Edit to respond to @simonw whose comment I saw before he retracted it ;)
This comment is tied directly to consistent continuous claims by the LLM labs. Their own products disprove their own claims, and it would indeed be nice if fewer people believed them :)
Just ran it on a small repo. It ran for almost an hour and then got interrupted. It drained half my weekly usage on a Pro plan.
npx codex-security scan .
[00:00] Preparing scan
[00:00] Authentication: stored Codex credentials.
[00:03] Preparing scan
[01:20] Running scan
[01:20] Preflight: worker delegation supported (up to 8 worker slots).
[52:47] Running scan
codex-security: Could not save the Codex Security scan: Repository HEAD changed while the scan was running. Start a new scan.
codex-security: Partial output was kept at ...
Some of approaches there could be useful in other contexts. OAI has the compute to experiment with different prompts and I'd expect these to be somewhat optimized.
security tools from AI companies feel like fire departments run by arsonists. useful, sure, but you can't help noticing who benefits from all the fires
Update: As far as I understand, this was already available as a Codex plugin. The main news is that OpenAI has now open-sourced it, and development is still moving quickly.
I seem to have gotten a bunch of you are trying to stuff we don't allow errors.. very annoying.
Can they explain what types of projects it works on and how does it check I own it? Like will it just not work on Linux kernel even on my own patches to it?
Looks great but the CLI output is not particularly interesting while the scan is running. I wish it could show token usage, some kind of progress, etc.
Yes, if only due to the business models being entirely different. AWS sells compute. OpenAI sells models. One of these things benefits from training on your data significantly more than the other.
I don't understand why Snyk is IN business in any way. Who really wants to upload his own code to a company that is specialized at searching security issues?
How can I trust that they show me all findings they have instead of selling the best ones to some three letter organisations?
I was actually discussing solutions for this with my coworkers—building white-hat security agents. It seems like openai/codex-security could simplify a lot of that, or at least provide a version of Codex that's purpose-built for security workflows. Really exciting news!
The scanner is the least interesting part of this. The harness around it is the product: dedup across runs, false-positive tracking, budget controls, CI gating. That is the layer where we'll see most interesting innovations in my opinion.
I'm building AQ, a coding harness for teams and the pattern is identical. For a while, I thought the raw model is the answer and quickly changed my mind. Purpose built harnesses are way more powerful than it sounds.
Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly.
If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here.
CLI docs: https://learn.chatgpt.com/docs/security/cli
EDIT: If you'd like to help make this better, we're hiring: https://openai.com/careers/full-stack-software-engineer-cybe...
> Have experience shipping production full-stack products across modern web frontends and backend services.
I'm amazed that the requirements are so low (or at least this vague) for jobs at companies like these.
Has anyone else had the experience of going to an interview and feeling like you were never asked any qualifying questions?
All the questions were easy, your answers were straightforward, you "got them right", but then were not chosen?
I find on the other side, they're also left with dozens of people who "passed" and then it comes down to a pretty arbitrary decision on who gets hired (if we are talking external, no referral, etc.)
I wonder if they can make job descriptions highly specific to filter the shortlist faster and more effectively (to actually get a shortlist).
Anyway end rant. Cool job, hope you fill it.
This sounds like it’s just an app development role, not a security analysis position, so I’m not sure what your complaint is.
This looks great, thanks for open-sourcing it!
How does it deal with the current guardrails 5.6 Sol has on finding vulnerabilities? When I use it in the Codex app it would sometimes say it found a vulnerability, but it cannot tell me what it is.
When would I use this over the plugin in codex? Which I think can be invoked from cli as well
The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration.
We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build.
Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.
I tried it, it started a scan but stopped after hitting the rate-limit of my account. It gave up after just a minute of retrying (rate limits are tokens per minute, so... :P).
It said "Partial output was kept at <...>", but I dont see a obvious way of picking it up in a new scan? (The failed run cost me ~$13)
Does it require hitting OpenAI's APIs or can one also stand up a local OpenAI compatible LLM endpoint?
We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.
Exciting! Is there any open GitHub issue we can track?
Been watching your progress for a while, glad OpenAI have looked after you and the team and you still get to ship!
How does it fare against its own codebase?
> co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
> Thanks for checking this out and for flagging the auth issues.
Offtopic, but this right here is why I don't believe any marketing around "great amazing models that one-shot everything and programmers are no longer needed".
You just have to look at what these labs routinely produce, and their own products.
Edit to respond to @simonw whose comment I saw before he retracted it ;)
This comment is tied directly to consistent continuous claims by the LLM labs. Their own products disprove their own claims, and it would indeed be nice if fewer people believed them :)
Just ran it on a small repo. It ran for almost an hour and then got interrupted. It drained half my weekly usage on a Pro plan.
Working as intended
I plan to hack it to use openrouter and Kimi K3 or GLM 5.2 to keep expenses reasonable.
For context can you share the line count?
FYI: Kimi K3 is relatively expensive on open router API pricing for agentic tasks, or at least that's been my experience playing around with it.
It's interesting how much of the value here is providing the english Skill definitions that tell the LLM what to do: https://github.com/openai/codex-security/tree/main/sdk/types...
Some of approaches there could be useful in other contexts. OAI has the compute to experiment with different prompts and I'd expect these to be somewhat optimized.
from plugin to main focus, wow
security tools from AI companies feel like fire departments run by arsonists. useful, sure, but you can't help noticing who benefits from all the fires
comment feels like someone complaining about being offered a fireproofing solution in the age of flamethrowers.
If you’re the one vibe coding you’re the arsonist.
Update: As far as I understand, this was already available as a Codex plugin. The main news is that OpenAI has now open-sourced it, and development is still moving quickly.
I got my agent to analyze it and do a write up here:
https://wasnotwas.com/writing/inside-openai-codex-security/
Alibaba just open sourced their version of a CLI code review tool too.
https://github.com/alibaba/open-code-review
They are entirely different products
I seem to have gotten a bunch of you are trying to stuff we don't allow errors.. very annoying.
Can they explain what types of projects it works on and how does it check I own it? Like will it just not work on Linux kernel even on my own patches to it?
https://news.ycombinator.com/item?id=49090181
Looks great but the CLI output is not particularly interesting while the scan is running. I wish it could show token usage, some kind of progress, etc.
How does it work? Does the tool upload code to ChatGPT for analysis? That may not be allowed for some corporate projects.
Yes, I suspect companies that don’t allow ChatGPT will not be able to use the ChatGPT security analysis tool.
Amazon bedrock is an option for gpt models that does not send your data to openai.
Does Amazon offer better privacy guarantees than OpenAI?
Yes, if only due to the business models being entirely different. AWS sells compute. OpenAI sells models. One of these things benefits from training on your data significantly more than the other.
I wonder if tools like this will put companies like snyk out of business. We use snyk at work and I have not been satisfied.
I like to think it just upped the bar, but good durable expertise will need to rise with it.
Why would a few code snippets put Snyk out of business?
I don't understand why Snyk is IN business in any way. Who really wants to upload his own code to a company that is specialized at searching security issues?
How can I trust that they show me all findings they have instead of selling the best ones to some three letter organisations?
be careful , your code will go to the cloud/ai using this
Yeah, this isn’t exactly new for us :d
Just getting auth issues so far...
https://news.ycombinator.com/item?id=49090181
yeah same here
Allow only OpenAi key? Requires Cyber registration? Yes. Yes. Useless.
hey
I don't think there's much to this other than it being a convenient CI wrapper around their existing models?
Edit: there's a little bit more meat here: https://github.com/openai/codex-security/tree/main/sdk/types...
All of codex is a wrapper around their models. There’s still value in a purpose-built harness.
Yeah, I think so too.
Yeah but management loved the idea.
would love it to see it h2h against https://github.com/usestrix/strix (45k stars)
They are entirely different products
yeah you mean because OAI is only whitebox? or expand on that a bit, haven't played around a ton w the oss codex sec
I was actually discussing solutions for this with my coworkers—building white-hat security agents. It seems like openai/codex-security could simplify a lot of that, or at least provide a version of Codex that's purpose-built for security workflows. Really exciting news!
The scanner is the least interesting part of this. The harness around it is the product: dedup across runs, false-positive tracking, budget controls, CI gating. That is the layer where we'll see most interesting innovations in my opinion.
I'm building AQ, a coding harness for teams and the pattern is identical. For a while, I thought the raw model is the answer and quickly changed my mind. Purpose built harnesses are way more powerful than it sounds.