If you don’t want to take advice from a parrot, don’t listen to ChatGPT: putting the tool to the test

ChatGPT has taken the world by storm since OpenAI revealed the beta version of its advanced chatbot. OpenAI also released a free ChatGPT app for iPhone and iPad, putting the tool directly into the hands of consumers. The chatbot and others Generative AI The tools flooding the tech scene have stunned and frightened many users due to their human-like responses and almost instant answers to questions.

People don’t realize that even though these chatbots provide responses that seem “human,” what they’re missing is fundamental understanding. ChatGPT was trained on a plethora of Internet data – billions of pages of text – and draws its answers from that information alone.

The data ChatGPT is formed from, called Common Crawl, is about as good as it gets when it comes to training data. Yet we never really know Why Or how the bot arrives at certain answers. And if he generates inaccurate information, he will say so with confidence; he doesn’t know it’s wrong. Even with deliberate, wordy prompts and premises, it can produce both correct and incorrect information.

The costly consequences of blindly following ChatGPT advice

We can compare the AI ​​generation to a parrot that imitates human language. While it’s good that this tool doesn’t have any unique thoughts or understandings, too many people mindlessly listen and follow its advice. When a parrot talks, you know it’s repeating words it’s heard, so you take it with a grain of salt. Users must process natural language models with the same dose of skepticism. The consequences of blindly following a chatbot’s “advice” could be costly.

A recent study by researchers at Stanford University, “How does ChatGPT behavior change over time?» found that the robot’s accuracy in solving a simple math problem was 98% in March 2023, but dropped significantly to just 2% in June 2023. This highlights its unreliability. Keep in mind that this search was for a basic math problem – imagine if the math or topic is more complex and a user can’t easily validate that it’s wrong.

• What if it’s code and it contains critical bugs?
• What about predictions about whether a group of X-rays is cancerous?
• How about a machine predicting your value to society?

If someone asks ChatGPT a question, they are likely not a subject matter expert and therefore do not know the difference between correct and incorrect information. Users might not spend time checking answers and make decisions based on incorrect data.

Picking ChatGPT’s Cybersecurity Resilience “Brain”

I asked ChatGPT for proposed solutions and tactical steps to build cybersecurity resilience against bad actors – a topic I am very familiar with. He provided some helpful advice and some bad advice. Based on my years of experience in the cybersecurity field, it was immediately obvious to me that the advice was questionable, but someone who is not an expert on the subject would probably not understand what answers were useful or harmful. Each of the boards emphasized the need for the human element when evaluating a robot’s boards.

ChatGPT: “Train your staff: Your staff can be your first line of defense against bad actors. It is important to train them on data security best practices and make them aware of potential threats.

• My opinion: It is essential to keep considerations such as experience level and areas of expertise in mind, as audience knowledge informs the educational approach. Likewise, training should be anchored in an organization’s specific cybersecurity needs and goals. The most valuable training is practical and based on employees’ daily activities, such as using strong, unique passwords to protect their accounts. As a bot, ChatGPT does not have this context unless you, the requester, provide it. And even with overly wordy and specific prompts, he can still share bad advice.

The verdict: It’s good advice, but it’s missing important details on how to train and educate employees.

ChatGPT: “Collaborate with other companies and organizations: Collaboration is key to building resilience against bad actors. By working with other companies and organizations, you can share best practices and information about potential threats.“

• My opinion : This is good advice when taken in context, especially when public and private sector organizations collaborate to learn from each other and adopt best practices. However, ChatGPT did not provide such context. The companies come together after one of them is the victim of an attack and discussing the details of the ransomware attack or payment, for example, could be incredibly dangerous. When a breach occurs, the primary focus should not be on collaboration but rather on triage, response, forensic analysis, and collaboration with law enforcement.

The Verdict: You need the human element to effectively evaluate information from natural language processing (NLP) models.

ChatGPT: “Implement strict security measures: One of the most important steps to building resilience against bad actors is implementing strong security measures for your AI systems. This includes things like strong authentication mechanisms, secure data storage, and encryption of sensitive data.

• My opinion : While this is good, high-level (albeit common sense) advice, “strict security measures” differ depending on the organization’s security maturity journey. For example, a 15-person startup guarantees different security measures than a Fortune 100 global bank. And while AI can give better advice with better prompts, operators aren’t trained on what questions to ask nor the warnings to be provided. For example, if you say the tips are for a small business without a security budget, you’ll undoubtedly get a very different answer.

ChatGPT: “Monitor and analyze data: By monitoring and analyzing data, you can identify patterns and trends that may indicate a potential threat. This can help you take action before the threat becomes serious.

• My opinion : Technical and security teams use AI to establish a behavioral baseline, which can be a robust and useful tool for defenders. The AI ​​finds atypical things to observe; however, he should not make decisions. For example, suppose an organization has a server that has been running a function daily for the past six months and suddenly it is downloading large amounts of data. The AI ​​could flag this anomaly as a threat. However, the human element remains crucial for the analysis, i.e. to see if the problem was an anomaly or something common like a burst of software updates on Patch Tuesday. The human element is necessary to determine whether abnormal behavior is truly malicious.

Advice as good (and recent) as training data

Like any learning model, ChatGPT draws its “knowledge” from Internet data. Skewed or incomplete training data impacts the information it shares, which can cause these tools to produce unexpected or distorted results. Additionally, the advice given by the AI ​​is as old as its training data. In the case of ChatGPT, anything based on information after 2021 is not considered. This is a major consideration for an industry such as cyber securityconstantly evolving and incredibly dynamic.

For example, Google recently made the .zip top-level domain public, allowing users to register .zip domains. But cybercriminals are already using .zip domains in their phishing campaigns. Now, users need new strategies to identify and avoid these types of phishing attempts.

But since this is new, to be effective in identifying these attempts, an AI tool would need to be trained on additional data on top of the Common Crawl. Building a new dataset like the one we have is nearly impossible due to the amount of text generated, and we know that using a machine to teach it is a recipe for disaster. This amplifies biases in the data and reinforces incorrect elements.

Not only should people be wary of ChatGPT’s advice, but the industry must evolve to combat how cybercriminals use it. Bad actors are already creating more credible phishing emails and scams, and that’s just the tip of the iceberg. Tech giants must work together to ensure ethical users are careful, responsible, and stay ahead in the AI ​​arms race.

Zane Bond is a cybersecurity expert and product manager at Security guard.