prompteval-pipeline
Provider-agnostic prompt evaluation for structured-output tasks: automatic model routing, syntax and LLM-based grading, and prompt variant comparison.
MIT
About
The project#
vektor-guard is an open-source prompt-injection classifier, built in public and released under Apache-2.0. The model, code, and training runs are all public, so every result on this site can be traced to its source.
The Inference Loop#
The Inference Loop is a newsletter on AI security and agentic AI. The Lab Log series documents how vektor-guard was built, one engineering decision at a time.
Lab Log
- #1Data Pipeline: Building the vektor-guard Training SetUpcoming
- #2Why ModernBERT over DeBERTa, and the ResultsUpcoming
- #3Building the Attack Taxonomy and Synthetic Data PipelineUpcoming
- #4Multi-Class Attack Classification ResultsUpcoming
- #5The Guard Service: SDK and FastAPIUpcoming
- #6Publishing to HuggingFace HubUpcoming
- #7The Validator Flywheel: v2 as Its Own Data GateUpcoming
Questions and contributions#
Found a misclassification or have a question? Open an issue on GitHub.
More from theinferenceloop#
Acknowledgements#
- Answer.AI, for ModernBERT, the base model.
- deepset, for the prompt-injections dataset.
- The PromptShield authors (Jacob et al., 2025), for PromptShield.
- Microsoft Research, for the BIPIA benchmark.