As a self-styled smartass, I am prone to bouts of tomfoolery and hijinks. This weekend, I texted the following to the family group chat:

“I will be leaving to get dinner and should be back by 6:30 pm. If you would like to continue to receive status updates, text YES to this number. Normal text and/or data rates will apply.”

On one of the phones, there was a button to auto-send a YES response. 

I had stumbled into the API!

Other family members tried to get that response with less, but it was clear that the full verbiage needed to be in there to make it work. I got a few more of those and we had a laugh.

Today, I went for two:

“Your appointment for 9:30 am is scheduled. Text CONFIRM to this number to confirm your appointment or CANCEL to cancel it. Normal text and/or data rates will apply.”

The result? Both a CONFIRM and a CANCEL button appeared on the other phone for autoresponses. 

This means, of course, that the API is invoked via scanning the text message itself. There are no back-end flags in my packets or anything like that, it’s straight-up giving the system a prompt and getting a response out of it that leverages into the target system adding executable code as a result of reading the prompt.

As a security person, I find the upshot of this to be chilling. There are other functions that could be automated and if the API simply attaches code to a message based on its wording without any verification of authenticity or authority, then it is a massive hole in the system. To defend against possible abuses, I know that I have some autoresponders set up with professionals that I make appointments with. Those I already know. If I make a new appointment with a new person and get an autoresponse in the time frame of that appointment, then I’m OK with that. What’s most dangerous is some kind of scam targeting people over 50 who are already at higher risk of implicitly trusting without verification. By using official-looking texts, it already increases the risk that they make an error. By having the system attach code for autoresponses, it makes them look that much more legitimate and, therefore, gives such attacks a higher conversion rate.

Which thought leads me to a larger zero-trust concept: cybersecurity also involves the concepts and philosophies surrounding our work. When we unequivocally accept any new paradigm without sufficient testing, verification, and cautious observation, then we place ourselves into a potentially unacceptably high level of risk. And when we let proven flaws remain in our systems because we choose not to disrupt production, then we know we are set up for a terrible tragedy.

When I taught, a grade below 70 was colloquially referred to as “failing”. As in, if someone got a score below 70 for a course, that person didn’t get credit for the course.

As it stands right now, generative AI is failing when it comes to using data from memory and failing hard when it comes to interpreting charts, diagrams, and images. Given that the current generation of generative AI is not that much better than the previous one, if we are plateauing out on generative AI performance, then the vast promises surrounding the technology ring hollow.

If a CEO announced, “I plan to replace my skilled workforce with low-paid people who are, at best, 68.8% accurate, and who will double-down on inaccuracies when challenged”, or, “we’re going all-in on inaccuracy, environmental destruction, and frustrating our customers” we would think that CEO had lost their ever-lovin’ mind. And if someone said they were going to revolutionize the world by putting BS artists into every development pipeline, expert system, and search engine, I would not see that revolutionization ending well.
https://venturebeat.com/ai/the-70-factuality-ceiling-why-googles-new-facts-benchmark-is-a-wake-up-call