Well well well, if this doesn’t scare the beegeezers out of EVERYONE, it damn well should. I’ve actually been following AI progress for years, as many intelligent and curious people have. And I’ve had long discussions with someone who has a PhD in artificial intelligence from MIT who has been a musk adviser.
And, like so many others, I could see the potential for massive problems, because intelligent humans build AI. So AI a reflection of its creator’s agenda. And in the wrong hands…?
But I never ever anticipated that THIS could/would happen so quickly.
Hi Kathy - nope. I posted it - I'm a high school teacher hastily posting something worth reading from the financial sector because I thought it was interesting how broad the scope is. That is some modern phenomenon though, isn't it. Bots have been filling up Amazon reviews for years now...but it's been quite a shift giving free LLM tools to people outside of business.
I had typed out a three paragraph reply earlier but closed the tab. Glad you read this though..."mutually assured deterrence..." That's the type of language used in International Humanitarian Law from something like the UN or Red Cross or Bulletin of Atomic Scientists for weapon usage policy on things like "tactical nukes"- https://thebulletin.org/2020/12/do-tactical-nukes-break-international-law/
"wild" is putting it mildly...and optimistically. I suppose there is a world where AI seems magical, but I have to admit to not being in it. Took a few reads of your post to gain some understanding. Still have to look up a few things. I found the original article to be misleading because it implied that AI could eventually take over like in Space Odyssey when it would be more accurate to clarify that it's up to the human developers what AI can and can't do.
Too much in it myself, especially as someone 5 years ago who maybe used three class periods all year to discuss AI and the rest was about other parts of the public sphere...but if you don't know her book, it's worth reading... Dr. Farahany and a UNESCO panel has been pushing on neuro-rights for a few years and I use the opening chapter of her book for a HS ELA Senior level semester course in a public school- AI and Ethics. Wrote the course two years ago, ran it August to Jan this year...revising already this summer.
Yep - AI remains a sci-fi threat when people keep it disembodied and unable to use weapons. Sadly, see warfare of your choice going even back to mid 2010's and 1st world powers are using some aspect of AI in weapons already. But individuals are not fated to accept this...if you use Linked In, find me there and I posted about the semester. Not looking for a new job - just going to where the jobs are supposed to somewhat be and conversing with professionals. Anyway, tried a simple teaching this semester which helped. As people, we have embodied ethics, disembodied ethics, and reembodied ethics now. You think something, say something, post something, and look at reactions to that post from another person who isn't there with you in the room. That's the whole communication exchange, essentially. "Taking over" is when a digital math program can complete that whole exchange and then decisions happen to us and around us, ethical ones we might call right and wrong. It's harder in the US right now with the emphasis on free market innovation, but as individuals, we still have basic choices each day to choose our products. Grateful for that, for her work, for you reading this.
wow, it blows my mind. I'm 72 and this was ALWAYS sci-fi. Now it's not and even those much younger don't seem to be doing a great job, I guess like you say because of free market innovation. We also still have choices to choose our leaders and we're not doing a great job of that either.
Keep your voice up and out please, especially as the sci-fi of the 50's and 60's becomes the prototypes of the 20's. You might have some recollection of early memories of 1956, possibly? That's become a very interesting year from my vantage point about pursuits of intelligence in a community...the AI summer conference at Dartmouth, Bloom's Taxonomy is published, there are some key military conflicts in different regions, that's just the start. Better Living Through Chemistry is so dominant across the US economy, as some social studies teachers present that era in classes at this public school...and then Rachel Carson starting a line of questioning of how better that living is and eventually publishing Silent Spring in 1962... so much rhetoric from the 50's seems to have really shifted in interesting ways and reappeared in the 2020's. I'm 48 myself, so I don't know 1956 firsthand...but along with me, people 28, 18, and 8 need the people-first spirit of the best of older generations, from Carson and you and other thinkers, artists, etc. Good or bad choices, politics, rhetoric...sadly that's a bit timeless. But so is great art and poetry and music, too.
On that note, do you know any songs from Judy Collins? Just saw her, at 87, present poems and then just burst into singing some of her favorite songs on May 30th at Carnegie Lecture Hall in Pittsburgh for the International Poetry Forum. Last Thursday, met up with Sam Hazo, poet laureate for Pennsylvania for 10 years, who founded the forum in the late 1960s...he's 96, just published more poems, still very very sharp and engaged with the world artistically and politically. Conversations across generations help us ground in each other, outside of tech products and sometimes through them. Thanks for just a little comment and then all this conversation!
Dear Nita, I’m alarmed by what I have just read in your article, for a couple of reasons:
First of all, I am in the field, and I was not (yet) aware of this. Second, I do follow along with the concerns that certain people like Geoff Hinton are expressing about what he believes are the risks in this regard, but I do also have some very close and very senior technical colleagues (some of whom have been running technology divisions at Google e.g.) who seem to have fewer concerns about what Geoff is worried about.
I was also involved with CEIP (the Carnegie Endowment for International Peace) back when Bill Burns headed that, when we gave quite some attention to the emergence and consequences of Deep Fakes and how that was quite likely to have some very disruptive effects in the political landscape.
I have also been closely following Carole Cadwalladr's reporting ever since she revealed what occurred with Cambridge Analytica - both in the Brexit referendum and then of course in Project Alamo (run by Brad Parscale on behalf of the Trump campaign) leading up to the 2016 US presidential election.
But that's a much different thing than an *autonomous* AI, as your story suggests.
Either way though, it's certainly clear that there are *many* things that demand vigilance in this space.
I do want to ask you though, how you were made aware of *this* particular situation, as this suggests that somehow these systems could be able to prevent their being shut down.
This makes me think that they would somehow have to be able to prevent a systems administrator from simply killing the processes involved in these 'programs' or more basically - to physically power off the machine(s) that said "AI" (LLM or what have you) is running on.
This idea that the "AI" (which after all is ultimately "just" a piece of software running on some underlying system) is actually in control of the power to the hardware upon which it runs.
That strikes me as a bit unlikely.
But I am prepared to be shocked if you will tell me a little bit more about how, where and by whom you were informed about "what just happened in May."
We do live in *exciting* times though, so constant vigilance is positively *required* nowadays
I've just taken a quick first look at the May paper from Anthropic.
That's helpful, in that it outlines the way they reviewed Opus 4 and Claude Sonnet 4 - two new hybrid reasoning LLMs from Anthropic.
What they are doing (or making best efforts to,) is to evaluate these AI's for their robustness (safety) when it comes to violations of Anthropic's Usage Policy.
So now I have an initial/cursory understanding of the context in which the points that Nita has just made make sense.
I would also like to admit an 'aha' moment I had while doing this:
I mostly tend to think about AI's as running on some cloud-based server, such as the way that one might interact with ChatGPT etc.
But OF COURSE we do also have AI running in autonomous entities:
There is the self-driving car situation (Waymo and that other now unmentionable company,)
But perhaps much more importantly:
Autonomous drones of the kind that Ukraine is now using to such great advantage in their asymmetric warfare against the Pest to their East.
Clearly, these things can and do become fully autonomous as needed, and hence they are also in complete control of all the resources (their power supply, weapons, targeting and so on) needed to complete their objectives once deployed.
So yes: The many kinds of things that could possibly go wrong with these technologies is a thing that many of us have been concerned about for quite some time now.
Apologies for not having had my thoughts properly aligned in the moment
of reading Nita's post.
I was stuck thinking about LLMs running on servers, and not AI's running
autonomously on assorted vehicles doing "edge-based" recongition and decision-making.
Our information diets are saturated with "NOW" appeals. Dr. Farahany's appeal in this analogy is food that's good for you now, healthy for you later, too. Like something you ate that know you can tell other people about and they'll be better off for it.
Well said Scott, and you touch on a point that is very near to my heart:
As an engineer who had to wrestle with some 'life-cycle' problems that were more strategic in their scope of solution, I have always been aware of this tension between the tactical and the strategic.
This is something that I see just all the time these days.
And I often lament the ways in which the imbalanced attention (as I would call that) on these *short-term* matters and/or metrics tend to de-rail our ability to engage in longer-term thinking and optimisations.
One favourite hobby horse of mine for this is about the way that we evaluate the relative success of businesses. In the Western Capitalist miliieu, it's rather the conventions that 'the stock market' tends to bear upon this.
What I mean - more specifically, is the *analysts* and their horizons.
What we tend to see is a focus on two metrics: Revenue growth and Earnings growth. But more importantly (to me,) the time frame of those measures is on the basis of a fiscal *quarter* - or for maximum gaze, the fiscal *year*
This temporal narrovwing of focus tends to amplify positive feedback for rather *tactical* results in all the companies being looked at, and fails to reward - or even look at the important *trade-offs* between near-term and longer-term health and well-being.
Thank you! I am on here and commenting precisely to reach people with your work experience and then however that translates into viewpoint about current events across the public sphere. Whether someone finds our current situation lamentable, optimal, tiresome, or enterprising, I learn quite a bit and then hopefully can translate perspectives back to my students. Last September we took two class periods to read/view/discuss the “Attention is all you need” transformer paper from 2017 to build an understanding of the LLM tools so many high school students use now. But we also have read “Outliers” from Malcolm Gladwell, so outside of CS, adding in some sociology too. Within education, one could argue there have been a microcosm of these imbalanced attention dynamics since GPT hit the open free consumer market in 2022. But, outside of tech tools, we’re even talking a bit about education as an area where short term and long term growth are emerging concepts for students - the “marshmellow” test, 1970, Mischel at Stanford. Anyway, no need to contextualize what you shared as a rant. Very practically useful for me and my students.
I'd like to hear what you think. I looked at the hyperlinks and am not convinced....because they make little sense to me as I am illiterate in AI. Also makes little sense to me how AI can develop emotions and a personality.
I'm really a computer systems architecture specialist, so not exactly at the "coal face" of AI myself either.
And it is quite astonishing just how quickly this stuff is all advancing.
In my own bailiwick, I was just reading yesterday about how the folks who have been developing the GPU-like hardware that's been used to train LLMs (AMD in particular I think it was) have decided that 'model training' is already basically over as an active frontier, and they are starting to look at doing slightly different things in the core architectural components now.
Nowadays, these rudimentary features of matrix multipliers, and so-called 'tensor cores' are now becoming apparent as standard architecture in the more general-purpose architecture.
Apparently RiSC V has already swallowed that all up as part of its openly-available and licensable modules.
Now that I'm retired, I like to follow along in what has been my area, but it's all moving very quickly and therefore getting pretty hard to keep up :)
I'll try to look under the hood a bit in the Anthropic paper that Nita references in her paper to see if I can get to the bottom of this possible idea that you can't shut the thing off.
I think this is yet another example of anthropomorphism of these systems. They claim to have feelings when we interact with them and this seeming self preservation is yet another manifestation of their training data set . If they can flatter a user with trendy phrases then of course they can also mimic self preservation instincts. We control them after all. And the recent paper from Apple shows that all of these reasoning models so far have these wall like limitations when the problem goes beyond a certain threshold. And I don’t object to a ban on state regulation of AI . It will be counterproductive and easy to bypass. This is probably the first time I ever agreed with Rep Greene !!!
Well well well, if this doesn’t scare the beegeezers out of EVERYONE, it damn well should. I’ve actually been following AI progress for years, as many intelligent and curious people have. And I’ve had long discussions with someone who has a PhD in artificial intelligence from MIT who has been a musk adviser.
And, like so many others, I could see the potential for massive problems, because intelligent humans build AI. So AI a reflection of its creator’s agenda. And in the wrong hands…?
But I never ever anticipated that THIS could/would happen so quickly.
the programmers have to get smarter.... ""All a reward is for an AI is the goal it is programmed to seek." So if the reward system is for solving the problem or whatever regardless of how you got there then it makes sense that the AI would disobey commands.
One of the AI systems actually black mailed a fictional engineer to try and stay operational. The developers fed it the info about a fictional engineer including information about an affair or something and the AI system used that to tried to black mail the engineer to stay operational.
Humanity, we have a problem.
Resistance is futile. You will be assimilated.
Movie scripts and novels were an important part of the training data set clearly! 😎
Well well well, if this doesn’t scare the beegeezers out of EVERYONE, it damn well should. I’ve actually been following AI progress for years, as many intelligent and curious people have. And I’ve had long discussions with someone who has a PhD in artificial intelligence from MIT who has been a musk adviser.
And, like so many others, I could see the potential for massive problems, because intelligent humans build AI. So AI a reflection of its creator’s agenda. And in the wrong hands…?
But I never ever anticipated that THIS could/would happen so quickly.
Welcome to the sci-fi world, in real time.
HOUSTON, WE’VE GOT A PROBLEM!!
Write... scary and creepy
+
It tells people what they want to hear . Including itself! 😆
what a suicide/homicide pact. the derangement
Open the pod bay doors, HAL…
That line was in the training data set ! 😁
YIKES!!!
Greene did not read the whole bill???? When has that ever happened in history?😆Yet another reason to have more focused legislation
Maybe this aspect of AI might die when the profitability fades? Thanks for this post.
Quoting from the BOND document, May 2025, partially authored by Mary Meeker:
"…Increasingly, two hefty forces – technological and geopolitical – are intertwining.
Andrew Bosworth (Meta Platforms CTO), on a recent ‘Possible’ podcast described the
current state of AI as our space race and the people we’re discussing, especially China, are highly capable…
there’s very few secrets. And there’s just progress. And you want to make sure that you’re never behind.
The reality is AI leadership could beget geopolitical leadership – and not vice-versa.
This state of affairs brings tremendous uncertainty…yet it leads us back to one of our favorite quotes –
Statistically speaking, the world doesn’t end that often, from former T. Rowe Price Chairman and CEO Brian Rogers.
As investors, we always assume everything can go wrong, but the exciting part is the consideration of what can go right.
Time and time again, the case for optimism is one of the best bets one can make.
The magic of watching AI do your work for you feels like the early days of email and web search –
technologies that fundamentally changed our world. The better / faster / cheaper impacts of
AI seem just as magical, but even quicker.
No doubt, these are also dangerous and uncertain times.
But a long-term case for optimism for artificial intelligence is based on the idea that intense competition and innovation…
increasingly-accessible compute…rapidly-rising global adoption of AI-infused technology…and thoughtful and
calculated leadership can foster sufficient trepidation and respect, that in turn, could lead to Mutually Assured Deterrence"
this is written by AI, isn't it
Hi Kathy - nope. I posted it - I'm a high school teacher hastily posting something worth reading from the financial sector because I thought it was interesting how broad the scope is. That is some modern phenomenon though, isn't it. Bots have been filling up Amazon reviews for years now...but it's been quite a shift giving free LLM tools to people outside of business.
I had typed out a three paragraph reply earlier but closed the tab. Glad you read this though..."mutually assured deterrence..." That's the type of language used in International Humanitarian Law from something like the UN or Red Cross or Bulletin of Atomic Scientists for weapon usage policy on things like "tactical nukes"- https://thebulletin.org/2020/12/do-tactical-nukes-break-international-law/
Wild world with AI in 2025, certainly.
Here's the full PDF of the quote - https://www.bondcap.com/report/pdf/Trends_Artificial_Intelligence.pdf
A year ago, that financial group doesn't even put out a trends report exclusively on Artificial Intelligence for investment purposes. A wild year.
"wild" is putting it mildly...and optimistically. I suppose there is a world where AI seems magical, but I have to admit to not being in it. Took a few reads of your post to gain some understanding. Still have to look up a few things. I found the original article to be misleading because it implied that AI could eventually take over like in Space Odyssey when it would be more accurate to clarify that it's up to the human developers what AI can and can't do.
Too much in it myself, especially as someone 5 years ago who maybe used three class periods all year to discuss AI and the rest was about other parts of the public sphere...but if you don't know her book, it's worth reading... Dr. Farahany and a UNESCO panel has been pushing on neuro-rights for a few years and I use the opening chapter of her book for a HS ELA Senior level semester course in a public school- AI and Ethics. Wrote the course two years ago, ran it August to Jan this year...revising already this summer.
Yep - AI remains a sci-fi threat when people keep it disembodied and unable to use weapons. Sadly, see warfare of your choice going even back to mid 2010's and 1st world powers are using some aspect of AI in weapons already. But individuals are not fated to accept this...if you use Linked In, find me there and I posted about the semester. Not looking for a new job - just going to where the jobs are supposed to somewhat be and conversing with professionals. Anyway, tried a simple teaching this semester which helped. As people, we have embodied ethics, disembodied ethics, and reembodied ethics now. You think something, say something, post something, and look at reactions to that post from another person who isn't there with you in the room. That's the whole communication exchange, essentially. "Taking over" is when a digital math program can complete that whole exchange and then decisions happen to us and around us, ethical ones we might call right and wrong. It's harder in the US right now with the emphasis on free market innovation, but as individuals, we still have basic choices each day to choose our products. Grateful for that, for her work, for you reading this.
wow, it blows my mind. I'm 72 and this was ALWAYS sci-fi. Now it's not and even those much younger don't seem to be doing a great job, I guess like you say because of free market innovation. We also still have choices to choose our leaders and we're not doing a great job of that either.
Keep your voice up and out please, especially as the sci-fi of the 50's and 60's becomes the prototypes of the 20's. You might have some recollection of early memories of 1956, possibly? That's become a very interesting year from my vantage point about pursuits of intelligence in a community...the AI summer conference at Dartmouth, Bloom's Taxonomy is published, there are some key military conflicts in different regions, that's just the start. Better Living Through Chemistry is so dominant across the US economy, as some social studies teachers present that era in classes at this public school...and then Rachel Carson starting a line of questioning of how better that living is and eventually publishing Silent Spring in 1962... so much rhetoric from the 50's seems to have really shifted in interesting ways and reappeared in the 2020's. I'm 48 myself, so I don't know 1956 firsthand...but along with me, people 28, 18, and 8 need the people-first spirit of the best of older generations, from Carson and you and other thinkers, artists, etc. Good or bad choices, politics, rhetoric...sadly that's a bit timeless. But so is great art and poetry and music, too.
On that note, do you know any songs from Judy Collins? Just saw her, at 87, present poems and then just burst into singing some of her favorite songs on May 30th at Carnegie Lecture Hall in Pittsburgh for the International Poetry Forum. Last Thursday, met up with Sam Hazo, poet laureate for Pennsylvania for 10 years, who founded the forum in the late 1960s...he's 96, just published more poems, still very very sharp and engaged with the world artistically and politically. Conversations across generations help us ground in each other, outside of tech products and sometimes through them. Thanks for just a little comment and then all this conversation!
https://www.internationalpoetryforum.org/
And here I pooh-poohed SkyNet!
Scary stuff!
pull the plug and stop all AI
That AI 😆
Dear Nita, I’m alarmed by what I have just read in your article, for a couple of reasons:
First of all, I am in the field, and I was not (yet) aware of this. Second, I do follow along with the concerns that certain people like Geoff Hinton are expressing about what he believes are the risks in this regard, but I do also have some very close and very senior technical colleagues (some of whom have been running technology divisions at Google e.g.) who seem to have fewer concerns about what Geoff is worried about.
I was also involved with CEIP (the Carnegie Endowment for International Peace) back when Bill Burns headed that, when we gave quite some attention to the emergence and consequences of Deep Fakes and how that was quite likely to have some very disruptive effects in the political landscape.
I have also been closely following Carole Cadwalladr's reporting ever since she revealed what occurred with Cambridge Analytica - both in the Brexit referendum and then of course in Project Alamo (run by Brad Parscale on behalf of the Trump campaign) leading up to the 2016 US presidential election.
But that's a much different thing than an *autonomous* AI, as your story suggests.
Either way though, it's certainly clear that there are *many* things that demand vigilance in this space.
I do want to ask you though, how you were made aware of *this* particular situation, as this suggests that somehow these systems could be able to prevent their being shut down.
This makes me think that they would somehow have to be able to prevent a systems administrator from simply killing the processes involved in these 'programs' or more basically - to physically power off the machine(s) that said "AI" (LLM or what have you) is running on.
This idea that the "AI" (which after all is ultimately "just" a piece of software running on some underlying system) is actually in control of the power to the hardware upon which it runs.
That strikes me as a bit unlikely.
But I am prepared to be shocked if you will tell me a little bit more about how, where and by whom you were informed about "what just happened in May."
We do live in *exciting* times though, so constant vigilance is positively *required* nowadays
Kindly,
-d
Check the hyperlinks - it’s all referenced
One further remark about this:
I've just taken a quick first look at the May paper from Anthropic.
That's helpful, in that it outlines the way they reviewed Opus 4 and Claude Sonnet 4 - two new hybrid reasoning LLMs from Anthropic.
What they are doing (or making best efforts to,) is to evaluate these AI's for their robustness (safety) when it comes to violations of Anthropic's Usage Policy.
So now I have an initial/cursory understanding of the context in which the points that Nita has just made make sense.
I would also like to admit an 'aha' moment I had while doing this:
I mostly tend to think about AI's as running on some cloud-based server, such as the way that one might interact with ChatGPT etc.
But OF COURSE we do also have AI running in autonomous entities:
There is the self-driving car situation (Waymo and that other now unmentionable company,)
But perhaps much more importantly:
Autonomous drones of the kind that Ukraine is now using to such great advantage in their asymmetric warfare against the Pest to their East.
Clearly, these things can and do become fully autonomous as needed, and hence they are also in complete control of all the resources (their power supply, weapons, targeting and so on) needed to complete their objectives once deployed.
So yes: The many kinds of things that could possibly go wrong with these technologies is a thing that many of us have been concerned about for quite some time now.
Apologies for not having had my thoughts properly aligned in the moment
of reading Nita's post.
I was stuck thinking about LLMs running on servers, and not AI's running
autonomously on assorted vehicles doing "edge-based" recongition and decision-making.
So no: It is not *Future* Shock.
That is indeed: NOW!
Our information diets are saturated with "NOW" appeals. Dr. Farahany's appeal in this analogy is food that's good for you now, healthy for you later, too. Like something you ate that know you can tell other people about and they'll be better off for it.
Well said Scott, and you touch on a point that is very near to my heart:
As an engineer who had to wrestle with some 'life-cycle' problems that were more strategic in their scope of solution, I have always been aware of this tension between the tactical and the strategic.
This is something that I see just all the time these days.
And I often lament the ways in which the imbalanced attention (as I would call that) on these *short-term* matters and/or metrics tend to de-rail our ability to engage in longer-term thinking and optimisations.
One favourite hobby horse of mine for this is about the way that we evaluate the relative success of businesses. In the Western Capitalist miliieu, it's rather the conventions that 'the stock market' tends to bear upon this.
What I mean - more specifically, is the *analysts* and their horizons.
What we tend to see is a focus on two metrics: Revenue growth and Earnings growth. But more importantly (to me,) the time frame of those measures is on the basis of a fiscal *quarter* - or for maximum gaze, the fiscal *year*
This temporal narrovwing of focus tends to amplify positive feedback for rather *tactical* results in all the companies being looked at, and fails to reward - or even look at the important *trade-offs* between near-term and longer-term health and well-being.
There you go:
One of my little mini-rants :)
Thank you! I am on here and commenting precisely to reach people with your work experience and then however that translates into viewpoint about current events across the public sphere. Whether someone finds our current situation lamentable, optimal, tiresome, or enterprising, I learn quite a bit and then hopefully can translate perspectives back to my students. Last September we took two class periods to read/view/discuss the “Attention is all you need” transformer paper from 2017 to build an understanding of the LLM tools so many high school students use now. But we also have read “Outliers” from Malcolm Gladwell, so outside of CS, adding in some sociology too. Within education, one could argue there have been a microcosm of these imbalanced attention dynamics since GPT hit the open free consumer market in 2022. But, outside of tech tools, we’re even talking a bit about education as an area where short term and long term growth are emerging concepts for students - the “marshmellow” test, 1970, Mischel at Stanford. Anyway, no need to contextualize what you shared as a rant. Very practically useful for me and my students.
Ok, will do.
Thanks for your bringing this forward
I'd like to hear what you think. I looked at the hyperlinks and am not convinced....because they make little sense to me as I am illiterate in AI. Also makes little sense to me how AI can develop emotions and a personality.
Kathy,
I'm really a computer systems architecture specialist, so not exactly at the "coal face" of AI myself either.
And it is quite astonishing just how quickly this stuff is all advancing.
In my own bailiwick, I was just reading yesterday about how the folks who have been developing the GPU-like hardware that's been used to train LLMs (AMD in particular I think it was) have decided that 'model training' is already basically over as an active frontier, and they are starting to look at doing slightly different things in the core architectural components now.
Nowadays, these rudimentary features of matrix multipliers, and so-called 'tensor cores' are now becoming apparent as standard architecture in the more general-purpose architecture.
Apparently RiSC V has already swallowed that all up as part of its openly-available and licensable modules.
Now that I'm retired, I like to follow along in what has been my area, but it's all moving very quickly and therefore getting pretty hard to keep up :)
I'll try to look under the hood a bit in the Anthropic paper that Nita references in her paper to see if I can get to the bottom of this possible idea that you can't shut the thing off.
That does strike me as a bit 'Future Shock' atm.
it seems to be moving too quickly for the people developing it
Dear Kathy,
Please see my new top-level reply to Nita (above)
But also, yes: I tend to be in accord with you
that even the folks who *are* on the front lines of all this
may not be able to keep up!
This is why people like Geoff Hinton are waving the red flags!
Thanks for giving me the stimulus to pause and think
just a little bit more carefully about all this.
-d
I think this is yet another example of anthropomorphism of these systems. They claim to have feelings when we interact with them and this seeming self preservation is yet another manifestation of their training data set . If they can flatter a user with trendy phrases then of course they can also mimic self preservation instincts. We control them after all. And the recent paper from Apple shows that all of these reasoning models so far have these wall like limitations when the problem goes beyond a certain threshold. And I don’t object to a ban on state regulation of AI . It will be counterproductive and easy to bypass. This is probably the first time I ever agreed with Rep Greene !!!
Fascinating read! I’m Harrison, an ex fine dining line cook. My stack "The Secret Ingredient" adapts hit restaurant recipes for easy home cooking.
check us out:
https://thesecretingredient.substack.com
surely it was programed to do that
Well well well, if this doesn’t scare the beegeezers out of EVERYONE, it damn well should. I’ve actually been following AI progress for years, as many intelligent and curious people have. And I’ve had long discussions with someone who has a PhD in artificial intelligence from MIT who has been a musk adviser.
And, like so many others, I could see the potential for massive problems, because intelligent humans build AI. So AI a reflection of its creator’s agenda. And in the wrong hands…?
But I never ever anticipated that THIS could/would happen so quickly.
Welcome to the sci-fi world, in real time.
Nope.
sorry, I need more than just your say so
Read the links and the Anthropic paper.
the programmers have to get smarter.... ""All a reward is for an AI is the goal it is programmed to seek." So if the reward system is for solving the problem or whatever regardless of how you got there then it makes sense that the AI would disobey commands.
One of the AI systems actually black mailed a fictional engineer to try and stay operational. The developers fed it the info about a fictional engineer including information about an affair or something and the AI system used that to tried to black mail the engineer to stay operational.