Just as buid history, and XUnit trend graphs as well, plot plugin should (optionally) keep the data corresponding to buid to keep forever.
For sure, if the build is no more kept forever, it should disappear from the graph at the next run if it is outside the number of builds to show in the plot.

This implies a new parameter (boolean : keepForever) in the plugin configuration.