waterpipe is an evaluation pipeline for LLM watermark research. It handles the full experimental workflow: generating watermarked and non-watermarked text via an OpenAI-compatible API, running detection, computing quality metrics, and testing robustness against a range of attacks.
pip install llm-waterpipe
Usage instructions and configuration examples are in the repository README .
An experiment is defined by a JSON configuration file specifying the model endpoint, dataset, watermark parameters, and which stages to run. The pipeline then executes them in order and writes structured outputs to the experiment directory: generations, detection results, metrics, attack results, and aggregated statistics. Stages can be run independently, which makes iterating on specific parts of an experiment fast.
Supported attacks include truncation, word deletion and reordering, character-level perturbations, synonym substitution, paraphrasing via LLM, and masked language model substitution. Quality metrics cover perplexity, BERTScore, and lexical diversity.
waterpipe was built for the Flip, Don’t Shuffle: Watermarking LLMs at the Speed of Inference experiments and is actively used in ongoing research beyond that paper.
The code is open source. Check it out on GitHub.