What NZ Businesses Need to Know Before Deploying AI Chatbots

Generative AI chatbots are becoming a serious consideration for many New Zealand businesses. Whether it's to improve customer service, reduce workload on staff, or offer 24/7 access to information, the benefits are clear. But so are the risks, especially when chatbots are connected to sensitive business systems or customer data.

This post outlines what you need to know before deploying a GenAI chatbot, with a focus on security, data handling, and risk mitigation.

busy little chat bot

Why Businesses Are Looking at GenAI Chatbots

AI chatbots can:

• Answer routine questions instantly

• Automate common internal and customer-facing tasks

• Scale easily across thousands of users

• Operate around the clock without staff overhead

• Personalise content or responses based on context

Industries across NZ including healthcare, finance, retail, and logistics are testing or already using these tools. But the real challenge is deploying them safely, especially when they’re interacting with staff, customers, or internal systems.

Understanding How GenAI Uses and Handles Data

To secure a chatbot properly, it’s important to understand how it works under the hood.

Where the AI Gets Its Knowledge

Large language models (LLMs) like GPT-4 are trained on enormous datasets pulled from public internet sources, books, forums, and documentation. This “training data” gives them the ability to answer general questions, summarise text, or write in natural language.

If you're using a publicly hosted LLM (like OpenAI or Claude), this training has already occurred and can't be modified unless you fine-tune the model with your own data. However:

• Some enterprise tools allow you to extend the AI’s knowledge using your data, such as product manuals, wikis, or policies. This is usually done via embedding and retrieval. The AI doesn't "learn" the data directly, but retrieves relevant chunks at runtime based on user input.

• In some setups, businesses opt to fine-tune a model or allow it to continually learn from new interactions. This creates a risk of data poisoning or model drift if not controlled.

What Happens When Someone Types a Prompt

When a user interacts with a chatbot, here’s what typically happens:

1. The user enters a message or query.

2. This is sent to the LLM API along with any additional “context” the developer has configured e.g. previous messages, role instructions, or access to documents via retrieval plugins.

3. The model generates a response based on the entire context window which may include recent chats, documents, and system instructions.

4. The response is returned to the user, and often logged for monitoring, analytics, or further training (unless disabled).

This means user prompts, metadata (like IP address), and generated responses may all be processed and stored by the AI provider, unless you're using a self-hosted model or have signed an enterprise contract that disables logging and training.

Key Point: Even if the AI seems "stateless", it can still retain and process context from prior inputs during a session, and this can lead to data being unintentionally carried across conversations.

Key Security Risks to Watch

Prompt Injection and Jailbreaking

Attackers can manipulate chatbot behaviour using crafted prompts. These might:

• Override system instructions (e.g. “ignore previous rules”)

• Extract data from memory

• Trigger unintended API actions (if the bot is integrated with systems)

Mitigation: Filter inputs and outputs, use strict prompt templating, and apply access controls to anything the bot can reach.

Data Leakage and Overexposure

If the AI is connected to internal documents, APIs, or systems, it can accidentally reveal more than it should, especially if access control isn’t enforced per user or session.

Mitigation: Use retrieval tools with scoped access per user. Never give the model direct access to raw databases. Use redaction or masking for sensitive outputs.

Chat History Exposure

Some chatbots store conversation logs for analytics or fine-tuning. If those logs include sensitive queries (e.g. customer info, credentials, financials), that’s a breach waiting to happen.

Mitigation: Disable persistent logging unless necessary. Encrypt logs, limit access, and strip sensitive data at the edge.

Untrustworthy Outputs

Models can hallucinate, confidently generating inaccurate or misleading response. This can result in false information propagating through business processes or causing business disruption if used without being reviewed. In regulated industries, this can lead to compliance issues or reputational harm.

Mitigation: Keep critical functions behind human review. Limit AI scope to read-only or advisory roles unless outputs are verified.

Third-Party Risk

Using SaaS-based chatbots means sending data to a third party. Unless you’ve checked how that data is handled, you could be violating privacy laws.

Mitigation: Review provider terms carefully. Ensure models don’t train on your data without consent. For sensitive workloads, consider self-hosting the model.

Technical Security Practices for Deployment

If you're planning to deploy a GenAI chatbot, make sure you have these fundamentals covered:

Access Control and Authentication

• Require login (SSO or MFA) before accessing internal bots

• Enforce role-based permissions on what the chatbot can access

• Segment access to sensitive data per user or department

Secure API and Integration Design

• Never give the chatbot direct database or file access

• Use middleware APIs to apply business logic and permission checks

• Rate limit all external API calls triggered by the bot

Context Window Management

• Limit how much data is passed into each interaction

• Avoid retaining session data across users

• Scrub sensitive details from context inputs

Input and Output Filtering

• Block known prompt injection patterns

• Apply content filtering to responses

• Log and monitor for high-risk queries or output anomalies

Hosting and Architecture

• Use secure cloud environments (e.g. VPC, private endpoints)

• Encrypt all data in transit and at rest

• Isolate chatbot functions from core infrastructure

Sector-Specific Considerations

• Healthcare & Financial Services: Avoid giving the bot access to any identifiable patient or client data unless you’ve done a full privacy and compliance review. Outputs should not be used for clinical or financial decision-making without validation.

• Retail & E-commerce: Be wary of bots accessing order data or user accounts. Authenticate users before surfacing anything personal.

• Public Sector & Government: Transparency is key. Ensure logs are auditable, and assess impacts on privacy, public trust, and data sovereignty (especially in relation to Māori data use).

Final Thoughts

GenAI chatbots are powerful, but they’re also complex and potentially risky if deployed without proper controls. Understanding how the model works, what data it can see, and how it processes inputs is critical to avoiding issues.

A secure deployment requires:

• Governance and planning from the start

• Strong access and data control

• Ongoing monitoring and testing

• Clear usage policies and staff education

If you're unsure about how your chatbot is handling data or where it might be vulnerable, it’s worth engaging a cybersecurity team with experience in AI risk assessments and penetration testing.