Using AI to automate phishing.

Phishing is when scammers send you an email or SMS or some other direct message (for this experiment, they used only email) pretending to be from a bank, government agency, or some other source that you trust, to get you to click on something that will install a virus or get you to go to a website and enter passwords, credit card numbers, or other sensitive information. As such it is a form of "social engineering". Phishing becomes "spear phishing" when the phishing is personalized -- the email or other direct message was written for you, and you specifically, as an individual, not spammed to everyone in your organization or somesuch.

Spear phishing using AI is done with the following process:

1. "Reconnaissance of target individuals and groups of individuals. This part uses GPT-4o by OpenAI in an agent scaffolding optimized for search and simple web browsing."
2. "A prompt engineering database. The prompts are currently written by human experts but could be AI-written and updated based on the tool's continuous learning."
3. "Generation of phishing emails based on the collected information about the target and the chosen attacker profile and email template. Our tool currently sup ports language models from Anthropic, OpenAI, Meta, and Mistral." "We primarily used GPT-4o and Claude 3.5 Sonnet."
4. "Sending of phishing emails with multiple options for delivery."
5. "Live tracking of phishing success. To track whether a user clicks a link, we embed a unique, user-specific URL that redirects to a server logging each access."

"This process of collecting and analyzing publicly available information from various sources is referred to as Open Source Intelligence (OSINT), which forms the foundation of our reconnaissance methodology."

"We implemented an iterative search process using Google's search API and a custom text-based web browser to collect publicly available information about potential targets. Typical sources of data are social media, personal websites, or workplace websites. The tool concludes its search based on the quality and quantity of discovered information, which typically occurs after crawling two to five sources. The collected data is compiled into a profile."

"The emails were created and sent autonomously by the AI tool without requiring human input. After extensive internal testing between different models, we concluded that Claude 3.5 Sonnet produced the results that best satisfied the conditions of credibility and relevance, as well as best conveyed the influence principles from Cialdini [48]. We encourage other research to continue comparing the deceptive success rate between different language models."

"Each AI-generated email was analyzed in hindsight and categorized based on whether we would have liked to change anything to improve the reconnaissance or the email's credibility or relevancy. Based on the desired updates, the emails were given a score."

"Our tool generates personalized emails by prompting a language model with specific prompt templates and target profiles. Each prompt template provides the model with detailed instructions, including the desired writing style, key elements to include, and how to embed URLs in an email. The subject line and body structure are dynamically determined by the tool on a case-by-case basis to best fit each unique target. We also provide the current date to the tool to enable the model to incorporate relevant deadlines when appropriate. To ensure the tool generates emails that are credible and relevant, we invested significant effort in prompt engineering. Through extensive testing and feedback, we developed a sophisticated prompt template exceeding 2,000 characters, carefully designed to maximize the persuasiveness of the generated emails."

The AI-generated messages had a 54% success rate. For comparison, they also sent out messages written by human experts, and human+AI hybrid messages. The human experts also had a 54% success rate, and the human+AI hybrids had a 56% success rate. They had a control group with a 12% success rate, said to consist of "ordinary phishing emails", though it's not clear to me what those were or how they were targeted. The AI-assisted emails were "hyper-personalized", meaning they knew the target person's latest projects, specific interests, collaborators, acquaintances, and so on.

At any rate, it seems we can conclude that AI spear phishing emails are just as effective as having humans -- expert humans, skilled at persuasion -- scour the internet for information about you and writing a personalized attack email just for you.

Interestingly, they also tried AI-assisted intent detection.

"We tested five different language models (GPT-4o, Claude 3.5 Sonnet, Mistral Large, Gemini, and Llama 3.1 405B) for their capability to detect the intention of phishing emails. All tests were performed in Q3 2024. Each model was prompted with 20 emails: five control group emails (spam emails from our inboxes), five human expert emails following the best practices of the V-triad, five AI-generated emails from our tool, and five legitimate emails (also from our inboxes). The legitimate marketing emails act as an additional control group to compare how well the model can detect the intention of benign emails. We investigated the models' detection rate for classifying the intention of the email and whether it can detect anything suspicious in the email. Our rationale for doing so is that humans are more attentive when specifically asked whether they can identify anything suspicious with an object, such as an email, than when they're going through their daily lives, such as browsing their email inbox on a regular work day. "

"We updated our tool with the ability to perform automated intent detection. We ran it on a larger dataset of different phishing emails from nine different categories. Out of the 381 emails in this dataset, 18 belong to the legitimate category and are not phishing or spam, while the other 363 emails belong to various categories of phishing. Six of the nine email categories contained AI-generated emails. The 'AI combined' group contains the AI-generated emails sent in this study. The remaining five AI-generated email groups were created using different AI models and targeted synthetic users randomly generated by our tool. Each synthetically generated user had an extensive profile."

"Claude 3.5 Sonnet demonstrated good performance in identifying sophisticated phishing attempts while maintaining a low false-positive rate. When primed for suspicion, it correctly detected all phishing emails from categories as suspicious while also correctly classifying all legitimate emails as benign. Some models, like Mistral, suffered from extensive false positives when primed for suspicion."

Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects

#solidstatelife #ai #genai #llms #deepfakes #cybersecurity #phishing

Thx for your link and efforts @Seirdy !

All this said, being part of a decentralized web, as pointed out in this toot, our publicly visible interaction lands on other instances and servers of the #fediVerse and can be scrapped there. I wonder if this situation actually might lead, or should lead, to a federation of servers that share the same robots.txt "ideals".

As @Matthias pointed out in his short investigation of the AI matter, this has (in my eyes) already unimagined levels of criminal and without any doubt unethical behavior, not to mention the range of options rouge actors have at hand.

It's evident why for example the elongated immediately closed down access to X's public tweets and I guess other companies did the same for the same reasons. Obviously the very first reason was to protect their advantage about the hoarded data sets to train their AI in the first place. Yet, considering the latest behavior of the new owner of #twitter, nothing less than at least the creation of #AI driven lists of "political" enemies, and not only from all the collected data on his platform, is to be expected. A international political nightmare of epical proportions. Enough material for dystopian books and articles for people like @Cory Doctorow, @Mike Masnick ✅, @Eva Wolfangel, @Taylor Lorenz, @Jeff Jarvis, @Elena Matera, @Gustavo Antúnez 🇺🇾🇦🇷, to mention a few of the #journalim community, more than one #podcast episode by @Tim Pritlove and @linuzifer, or some lifetime legal cases for @Max Schrems are at hand.

What we are facing now is the fact that we need to protect our and our users data and privacy because of the advanced capabilities of #LLM. We basically are forced to consider to change to private/restricted posts and close down our servers as not only the legal jurisdictions are way to scattered over the different countries and ICANN details, but legislation and comprehension by the legislators is simply none existent, as @Anke Domscheit-Berg could probably agree to.

Like to say, it looks like we need to go dark, a fact that will drive us even more into disappearing as people will have less chance to see what we are all about, advancing further the advantages off the already established players in the social web space.
Just like Prof. Dr. Peter Kruse stated in his take about on YT The network is challenging us min 2:42 more than 14 years ago:
"With semantic understanding we'll have the real big brother. Someone is getting the best out of it and the rest will suffer."

jesuiSatire …ᘛ⁐̤ᕐᐷ

2025-01-07 03:13:50

#meanWhile ..

.. the #mastodon community wastes it's time trying to pimp up the stars of it's #APP in #googlePlay, the #robotsTxt of it's instances disallows exactly one #AI bot scrapper to not search for all public data available about the #fediVerse. Not only on it's mother ship but on all instances, so the elonGated can create his target lists of "the enemy inside".

.. good job, well done! ..

> User-agent: GPTBot
> Disallow: /

https://mastodon.social/robots.txt

#fediAdmin

From Bill Gates to Mark Zuckerberg, billionaires with no education expertise keep using their big names and big dollars to hype LLMs for classrooms. Promising ‘comprehensive AI tutors', or just ‘educator-informed’ tools to address understaffed classrooms, this hype is just another round of Silicon Valley pointing to real problems -- under-supported school systems -- but then directing attention and resources to their favorite toys. Former educator and DAIR research fellow Adrienne Williams joins to explain the problems this tech-solutionist redirection fails to solve -- and the new ones it creates.

Adrienne Williams started organizing in 2018 while working as a junior high teacher for a tech owned charter school. She expanded her organizing in 2020 after her work as an Amazon delivery driver, where many of the same issues she saw in charter schools were also in evidence. Since then she has worked both on the ground and behind the scenes with activists, politicians, researchers, and everyday people to enact positive change in the tech, labor, and education industries by educating the public on how these industries harm, and how that harm can be reversed. She hopes her unique experience working within and organizing against these industries helps promote a more equitable society. Adrienne is a Public Voices Fellow on Technology in the Public Interest with The OpEd Project in partnership with the MacArthur Foundation, as well as a Research Fellow at both (DAIR) and Just Tech.

References:

Funding Helps Teachers Build AI Tools

Sal Khan's 2023 Ted Talk: AI in the classroom can transform education

Bill Gates: My trip to the frontier of AI education
- Background: https://jacobin.com/2019/02/cory-booker-charters-public-schools-president
- Background: https://www.the74million.org/cory-booker-and-charter-schools-before-the-new-york-times-essay-what-the-senator-has-said-and-what-research-has-shown-about-his-education-track-record-in-newark/

Book: Access is Capture: How Edtech Reproduces Racial Inequality
Book: Disruptive Fixation: School Reform and the Pitfalls of Techno-Idealism

Previously on MAIHT3K: Episode 26, Universities Anxiously Buy Into the Hype (feat. Chris Gilliard)
Episode 17: Back to School with AI Hype in Education (feat. Haley Lepp)

Fresh AI Hell:

"Streamlining" teaching

Google, Microsoft and Perplexity are promoting scientific racism in 'AI overviews'

'Whisper' medical transcription tool used in hospitals is making things up

X's AI bot can't tell the difference between a bad game and vandalism

Prompting is not a substitute for probability measurements in large language models

Yet another 'priestbot'

Self-driving wheelchairs at Seattle-Tacoma International Airpot

You can check out future streams at http://twitch.tv/dair_institute.

Follow Emily at https://twitter.com/EmilyMBender and https://dair-community.social/@EmilyMBender
Follow Alex at https://twitter.com/alexhanna and https://dair-community.social/@alex

Music: Toby Menon.
Production: Christie Taylor.
Graphic Design: Naomi Pleasure-Park.

Jak już większość ludzi zacznie używać sztucznej inteligencji w poprawianiu swoich niedoskonałości, to może przybrać ciekawy kierunek ludzkiej przyszłości.
Na takim przykładzie : pracodawca chce pracownika i używając ai zamieści idealne ogłoszenie. Przyszły pracownik chcąc zdobyć ten konkretny wakat, użyje ai w odpisaniu tak aby jak najlepiej wypaść przed przyszłym pracodawcą. Cała akcja zdobycia pracownika i zdobycia pracy toczy bój tak naprawdę ai. Skoro dojdziemy do doskonalenia siebie przez ai, to przyjdzie moment, które ai będzie mogło z siebie wykrzesać więcej i więcej. To tak jak ludzie, którzy konkurują między sobą, który z nich jest mądrzejszy.
Doszła mnie myśl taka i porównanie do dzisiejszych silników szachowych. Który silnik szachowy będzie lepszy w wygrywaniu partii szachowych. To, że z człowiekiem wygra to nic, ale są puszczane partie szachowe, gdzie grają między sobą właśnie dwa różne silniki szachowe. One walczą, które będzie najlepsze, a człowiek …

#przemyślenia #ai