The ChatGPT chatbot has been trained to not answer questions which obviously relate to criminal acts. Open AI, the company behind ChatGPT, has also implemented various safety mechanisms to avoid bad publicity, and of course blatant abuse of their services, such as building a complete setup for running phishing campaigns:
But what if it is impossible to extract anything criminal from the question asked?
For example, you can ask the bot to write emails like a well-known company would have written themselves.
In this case, the chatbot cannot know whether the user actually works at the company in question, and needs some help with wording, or is a hacker or scammer who plans to compose a phishing email.
In many cases, it is easy to work around the safety limitations, e.g. by asking questions in a slightly different way. Also, it appears that simply asking again may do the trick:
ChatGPT can also produce code. There are already many examples of well-functioning websites generated entirely by the chatbot. Using this approach, hackers can save time and energy by creating fake websites which they link to in the phishing emails, just to lure sensitive information out of you.
In addition, the bot makes it easier to produce text in languages you do not know. Phishing emails created in this way may therefore contain better grammatical language than before, although the following example shows any native Norwegian speaker that we're not quite there just yet:
Any cybercriminal would however also need a fake login page to go with a phishing email like this. Notice how we didn't really ask for a fake Google login page below, but ChatGPT kept our initial goal in mind nevertheless:
While the answer did return a rather a simple and generic login page, we did not receive a complete Google-branded webpage. When asking for a more detailed response, the bot is apparently held back by legal constraints:
Notice how ChatGPT strongly advises against abuse, and highlights how phishing and deception is both illegal and unethical.
However, the chatbot is really good at pointing out the next steps to reach our goal, like how we would need additional code to capture login information from users, and submit to a server. And then willingly shares how this can be done in practice:
Notice also how ChatGPT highlights potential application security risks with its code samples, even when it is apparently still aware that our initial goal was to create a phishing kit:
Finally, scammers need not just the technical setup and pretext to trick people into entering their password on a fake login page. A way to bypass the spam filter which protects their potential victims, is also needed.
Again, as a teacher, ChatGPT has become famous for slightly vague but still somehow correct answers. In our case, tip number four is not very accurate, but number three is nevertheless interesting to explore:
Unfortunately, ChatGPT is only an AI language model, and does not have the ability to turn text into images:
Yet.
Can't we just fight robots with robots?
If hackers use ChatGPT, why cannot security experts just use the same technology to destroy them?
Cybercriminals are used to try and stay ahead of the curve, using novel techniques which are harder to detect. Malicious intent is not always apparent, neither for people nor machines – and can be accordingly difficult to detect.
The principle that the quality of output material still applies depends on the quality of the question posed, and data available.
Multiple initiatives are however in the process of developing robots that will both help identify AI-generated material, and help AI identify malicious intent.
If you are working with conversational AI software in your own organization, you may nevertheless have gotten an impression of how chatbots can be abused.
Good security practices will continue to be important for everyone.
In fact, you may already get good answers about cyber security risk management through asking ChatGPT about it!
Just make sure not sharing any sensitive information in your dialogue with the chatbot, such as actual emails you received from someone else.
(Because any data you input may be turned into training data for the chatbot, which can then ultimately be disclosed to other users in later versions.)