bmathew: Meanwhile you might have found the root cause but here is what we found from time to time: This is usually caused by a failure in the backup process. Check the Jenkins logs and search for "thin" or "backup". When the backup process runs into any exception it seems to keep processing the backup but not triggering the rotation of old backups.
We worked around by checking the number of backups in the folder daily and act immediately if there are more than the expected number of backups. As I said it usually is caused by an exception in the backup process and you need to handle it manually (e.g. by deleting some outdated job folders), see e.g. https://github.com/jenkinsci/thin-backup-plugin/pull/24.