Interesting problem, this - if we're talking hundreds of folders, it's a bit of a pain to do manually in individual operations (as I'm sure you've already concluded before asking here), so it would probably be best to A) create a simple .bat script or use Powershell if you're on Windows or B) use some software for it. I don't think this is possible to do in any easy way in the native Windows file manager.
A)
The script itself should be quite simple; i.e. "replace file x.jpg with file x(new).jpg" - but how cumbersome it might be to type up depends a bit on your actual folder/file structure, as this script would need the paths of both new and old images; as an example, that'd be something like:
"C:\Music\Library\KPM\KPM 1188 - Contempo\MDMtMTM5MTE3NDIw/MC0xMzQ4LmpwZWc.jpeg" (old photo)
and "C:\Photos\ImprovedCovers\KPM1188.jpg" (new photo)
On the other hand, if your files and folders are all something neat and sequential like:
"C:\1\1.jpg" (old photo)
and "C:\New\1\1.jpg" (new photo)
... then that'd of course be much simpler

Do the old and new photos have the same filenames?
B)
I haven't tried any of these myself, but a cursory google search indicates third-party file management software like FreeCommander might be able to do this for you, too.