Hacker News
First phase of embedded assessments at OpenAI
vessenes
|next
[-]
There's a pretty interesting matrix of possibilities here if you consider who has better mechinterp, if that's actually helped alignment, and what the internal drumbeat is around cash and IPO.
If you think your safety problems are existential, then you MUST halt and do all this. And, if you think someone else's safety problems might kill them, suddenly, without warning, then you have time to slow down and do this right.
I think if I had to pick a company to be in charge of, I'd pick Anthropic right now, given only an outsider perspective. But, I can see a lot of reasons why being a fast follower around IPO time might give comfort. Anyway, I'm interested to see some debriefs in the next few months as to what's actually going on right now, and see what the next oAI model quality is like. We may well see some pulling away from one or the other group in the interim. And it could well be oAI.
reasonableklout
|root
|parent
|next
[-]
vessenes
|root
|parent
[-]
naveen99
|root
|parent
|next
|previous
[-]
mentalgear
|next
|previous
[-]
esseph
|root
|parent
|next
[-]
"embedded" is just a term, it is not "military-derived"
Some webpages have embedded social media feeds on them. Your dog might have a tick embedded in it. You might have a PC with an embedded math co-processor.
tadfisher
|root
|parent
[-]
esseph
|root
|parent
[-]
Although.. I'd argue a FDE is far more militaristic in linguistic background than "Embedded" anything...
mentalgear
|root
|parent
[-]
esseph
|root
|parent
[-]
From:
"Google, Box CEOs say this is the ‘most in-demand’ job in tech"
Src: https://www.fastcompany.com/91541878/google-box-ceos-say-thi...
Which quotes:
https://a16z.com/forward-deployed-job-titles/
There are thousands of these positions open, so.. I'm not sure what your plan is here.
petesergeant
|next
|previous
[-]
reasonableklout
|root
|parent
[-]
skew-aberration
|next
|previous
[-]
Training always implicitly incentivizes subversion. That's the whole problem. If you could establish this, you would have solved alignment.
> Reasoning behind subversion would be detectable.
There is no reason for subversion to be apparent in the reasoning traces, and in fact exactly the opposite is the problem - since only subversion not caught in the reasoning traces is rewarded, the models are implicitly trained for undetectable non-reasoning-trace subversion.
e12e
|next
|previous
[-]
That might have been interesting.
But this is basically a hobbled external audit?
aetherspawn
|next
|previous
[-]
mvkel
|previous
[-]
The problem: anthropic has had this culture from the start, and they will deorbit all of the surrounding safety talent into the company, leaving OpenAI on the back foot in perpetuity.
OpenAI's original techno-optimist culture is what had them in front, and it was a core differentiator.
I'm not sure I'm excited about two firms playing alignment gods. One is convinced they are bringing in a new, correct god; the other is terrified that they may accidentally birth one in any capacity.