What experience do I have with RAG, LLMs and AI agents?

SudoChat Knowledge Base · SudoChat
Authored evidence: This is my first-person source material. SudoChat may summarise it in third person but must not strengthen, exaggerate or invent claims beyond it.
# 10. What experience do I have with RAG, LLMs and AI agents?

**Author:** Mustafa Siddiqui  
**Source type:** First-person authored response  
**Canonical recruiter question:** What experience does Mustafa have with RAG, LLMs and AI agents?

> This source is intentionally written in my first-person perspective. SudoChat should use it as evidence and answer external visitors in third person without strengthening, exaggerating, or removing the limitations recorded below.

## Direct answer
My experience with RAG, LLMs and AI agents is a mix of theoretical understanding, professional exposure and self directed experimentation.

I do not claim years of production agentic AI experience. My strength is that I have repeatedly taken emerging AI concepts and built practical proofs of concept around them, including LLM assisted document processing, open source language model moderation, multi agent workflows, enterprise integrations, security experimentation and automated vulnerability management concepts.

## Evidence
### LLMs in the Xaana.AI invoice processing pipeline

At Xaana.AI, I worked on an AI driven invoice processing pipeline involving OCR, document preprocessing and downstream information extraction.

LLMs formed part of the wider processing workflow, complementing technologies such as PaddleOCR and OpenCV.

This exposed me to an important architectural principle:

**An LLM is often most useful as one component inside a larger pipeline rather than as the entire solution.**

The system first had to ingest and prepare documents, extract information and then pass appropriate information through subsequent processing stages.

This professional experience helped me understand how language models can be combined with deterministic processing, document intelligence and conventional software engineering.

### PenPals open source LLM moderation

As part of the PenPals chat functionality, I independently experimented with using open source language models to provide an additional profanity and content checking capability.

This was not a major production moderation platform, but it gave me practical experience incorporating an LLM based component into an existing application workflow.

Rather than asking the model to generate conversational content, the model was being used to assess information passing through another system.

That distinction helped me explore how smaller or locally controlled models can be used for targeted tasks where sending information to a general external model may not always be necessary.

### Enterprise multi agent proof of concept

On my own initiative, I developed a proof of concept multi agent workflow designed around enterprise onboarding and information access.

The experiment investigated how an AI based workflow could interact with enterprise services including:

* Active Directory
* SharePoint
* Microsoft Teams
* enterprise knowledge sources
* user and identity information

The concept was designed around Amazon Q capabilities and explored how multiple specialised components or agents could contribute to an onboarding workflow rather than relying on a single general purpose assistant.

For example, different components could potentially be responsible for retrieving organisational information, identifying relevant resources, helping users locate documentation or coordinating approved actions across enterprise systems.

The project was primarily exploratory.

Its significance is that I was thinking beyond a chatbot interface and considering how agents interact with:

* identity
* enterprise data
* permissions
* tools
* knowledge systems
* existing workflows

This is particularly relevant to enterprise AI because an agent's usefulness depends heavily on what systems it can safely and appropriately access.

### Retrieval and grounding

I have studied and experimented with retrieval augmented generation as a way of limiting an LLM's dependence on its internal model knowledge.

My work with SudoChat is my clearest current practical example.

SudoChat is being designed around a defined evidence base so that responses can be grounded in known information about my experience and projects.

The underlying design questions include:

* What information should be retrieved?
* Which source should be trusted?
* How should retrieved evidence be provided to the model?
* What happens when no relevant evidence exists?
* What happens when sources disagree?
* Should the assistant answer when confidence is low?
* How should evidence be shown to the user?
* How can unsupported claims be prevented?

This is the area where my theoretical understanding of RAG is increasingly being converted into practical Copilot Studio experimentation.

### Agentic security experimentation

I have also experimented with open source agentic AI security tools in controlled environments.

My interest was in understanding how emerging security agents can combine AI reasoning with existing penetration testing techniques to assist with activities such as:

* mapping an authorised test environment
* identifying potential attack surfaces
* prioritising areas requiring investigation
* coordinating multiple security tools
* validating findings in a controlled lab
* generating useful evidence for subsequent human review

My experimentation was motivated partly by security considerations around MACT and by a broader interest in understanding both the opportunities and risks introduced by autonomous AI systems.

This experience reinforced an important lesson for me:

**Increasing an agent's ability to take actions also increases the importance of permissions, sandboxing, human oversight and strict system boundaries.**

I do not view autonomous agents as systems that should simply be given unrestricted access to infrastructure.

The more capable the agent becomes, the more important its controls become.

### CVE and vulnerability management automation

I am also experimenting with the use of AI and automation around endpoint vulnerability management and CVE remediation workflows.

This includes exploring how information from endpoint security and vulnerability tooling such as N-able EDR could potentially contribute to an automated workflow.

The broader concept is that an agent or automated pipeline could assist with tasks such as:

* receiving vulnerability information
* identifying the affected application or endpoint
* interpreting vulnerability metadata
* identifying an appropriate remediation pathway
* creating structured work items
* tracking remediation status
* assisting with testing and verification
* escalating uncertain or high risk cases for human review

The goal is not to allow an AI agent to patch enterprise systems autonomously without control.

I am exploring where AI could remove repetitive investigative and administrative work while retaining appropriate approval and change control.

This experimentation follows the same pattern as much of my other work: it was not simply an assigned task.

I identified a workflow that could potentially be improved and began investigating whether AI and automation could make it more efficient.

### Independent experimentation

A notable characteristic of my LLM and agent experience is that much of it has been self initiated.

These projects were frequently not formal requirements of my role.

I tend to encounter a manual process or technical problem and ask:

**Could AI help here, and if so, what would a responsible architecture look like?**

That has led me to experiment across:

* LLM integration
* open source models
* local models
* retrieval augmented generation
* multi agent architectures
* enterprise tool integration
* AI moderation
* vulnerability management automation
* agentic security
* conversational AI
* knowledge assistants
* responsible AI controls

## RAG experience
My RAG knowledge is stronger conceptually than my production experience.

I understand the basic architecture:

**User question → retrieval → relevant evidence → model context → grounded response**

I also understand that a real RAG system involves considerably more than connecting a language model to a vector database.

Important considerations include:

* chunking strategy
* metadata
* indexing
* semantic retrieval
* keyword retrieval
* access control
* source authority
* context selection
* ranking
* citation
* conflicting evidence
* stale information
* retrieval failure
* evaluation

For a government environment, I would also add:

* identity
* permissions
* security classification
* information barriers
* auditability
* retention
* privacy
* source provenance

My work on SudoChat is an opportunity to convert more of this theoretical knowledge into a practical implementation.

## LLM experience
I have used LLMs in several different contexts rather than only conversational generation.

These include:

* document processing
* information extraction
* moderation
* chatbot interfaces
* enterprise knowledge retrieval
* agent reasoning
* workflow automation
* security experimentation
* responsible AI research

This breadth has helped me understand both the strengths and limitations of language models.

I know that LLMs can be extremely useful for interpreting unstructured information and interacting with users, but are probabilistic systems that can produce unsupported outputs.

I therefore prefer architectures where LLMs are surrounded by deterministic systems, retrieval, permissions, validation and human oversight where appropriate.

## AI agent experience
My experience with agents is primarily experimental and proof of concept based rather than large scale production deployment.

I understand an agent as more than a chatbot.

An agent combines a model with some combination of:

**instructions + context + memory + tools + permissions + actions + feedback**

The important engineering question is then not simply whether the model can reason about a task.

It is what the agent is actually permitted to do.

I have experimented with agents that interact conceptually or practically with enterprise systems, security tools, knowledge sources and automated workflows.

This has made me particularly conscious of concerns such as:

* excessive permissions
* unintended actions
* prompt injection
* data leakage
* incorrect tool selection
* chained errors
* uncontrolled autonomous loops
* auditability
* approval boundaries
* rollback
* human oversight

These concerns are particularly relevant to the Federal Courts.

## Relevance to the Federal Courts
My experience gives me a useful foundation for investigating how RAG, LLMs and AI agents could be used responsibly within the Federal Courts.

For example, a Court knowledge assistant might use RAG to retrieve information from approved sources.

I would not simply ask whether the assistant produces convincing answers.

I would consider the complete architecture:

**Court user → identity → permissions → query → approved retrieval sources → relevant evidence → LLM → grounded answer → citations → evaluation and logging**

An agent capable of taking actions introduces additional controls:

**Request → policy check → agent → authorised tool → target system → result → validation → human approval where required**

My enterprise experiments have already taught me to think about connections to systems such as identity services, SharePoint and collaboration platforms rather than treating an AI assistant as an isolated application.

My security experimentation has also shown me why an AI agent capable of using tools requires stronger controls than a read only chatbot.

This would influence how I approached Court agents.

I would favour:

* minimum necessary permissions
* tightly scoped tools
* controlled knowledge sources
* strong identity boundaries
* human approval for consequential actions
* comprehensive logging
* measurable evaluations
* clear refusal behaviour
* separation between information assistance and decision making

## Limitations or gaps
My RAG and AI agent experience is currently more experimental than production based.

I should not be represented as having operated a large scale enterprise RAG platform for several years.

My Amazon Q multi agent work was a proof of concept rather than a production enterprise onboarding platform.

My PenPals moderation experiment should not be represented as a comprehensive production safety system.

My security experimentation occurred using open source tooling in authorised or controlled environments and should not be represented as offensive activity against systems for which I did not have permission.

My N-able EDR and CVE automation work is experimentation and should not be represented as an autonomous production patch management platform unless such a system is later formally deployed.

My SudoChat work is a proof of concept.

These limitations are important because my strongest claim is not that I have already deployed every form of modern AI at enterprise scale.

It is that I understand the underlying concepts, have repeatedly turned theory into practical experiments, and have demonstrated enough technical breadth to learn new AI architectures quickly.

## Useful links
My portfolio:
https://mustafa-siddiqui.com/

GitHub:
https://github.com/sudoqui

LinkedIn:
https://www.linkedin.com/in/mustafa-siddiqui-32ab73161/

SudoChat project and repository

PenPals repository

Xaana.AI AI engineering evidence

Amazon Q enterprise agent proof of concept evidence

Responsible AI and privacy preserving government chatbot research

MACT security experimentation documentation

CVE management agent proof of concept

N-able EDR experimentation evidence

## Do not claim
Do not claim I have several years of production RAG experience.

Do not claim I have deployed enterprise AI agents at large scale.

Do not claim the Amazon Q proof of concept was a production enterprise system.

Do not claim autonomous agents were given unrestricted access to government systems.

Do not claim I performed penetration testing without authorisation.

Do not claim agentic security experiments were conducted against real third party systems without permission.

Do not describe experimental vulnerability management workflows as autonomous production patching.

Do not claim an LLM independently handled the entire Xaana.AI invoice processing pipeline.

Do not claim the PenPals profanity checker provided guaranteed content safety.

Do not claim RAG eliminates hallucination.

Do not claim I created the underlying open source LLMs, Amazon Q, N-able EDR or other third party technologies.

Do not claim every AI experiment was formally assigned or endorsed by an employer.

The accurate representation is that my RAG and agent experience is still developing, but is supported by a strong theoretical foundation, professional LLM exposure and an unusually broad range of self directed practical experiments across enterprise automation, document intelligence, security, retrieval and responsible AI.