Mangadex bug

[Squidy]

I've seen that, the API requests do get rate-limited when you have too many downloads going on at once. I'll implement some new per-domain rate-limiting features for (hopefully) the next update.

I updated the module again to use a different referer when accessing the images, but it's possible that getting the "wrong" images is a form of rate-limiting too. Could you guys check if the problem occurs if you only download one manga at a time?

There are also some rate-limiting settings already under Tools > General settings/preferences > Timing & delays, where you can set global or per-website rate limits. It might be worth checking if adding a delay between images mitigates the issue. If it does, maybe I'll add it by default in the next update.

If push comes to shove and this doesn't help, my last approach will just be to have it retry the request if the MIME type doesn't match, since that seems to be the indicator.

[fusedragon]

I mostly download one at a time with 0.5s delay between pages and 5s between downloads. Anyway updated the module and tried changing it to 5s/10s/15s between pages, still getting the issue.

[JohnKo23]

I only download 2 manga at a time but don't put a page delay on it. I can try changing those and see if anything changes. I also just now updated to the newest module as well.

[penina9291]

Tried timing & delays thing, didn't work. Also, I only download 1 manga at a time.

[fusedragon]

Also, not sure if it's relevant but I've also seen pages from what looks like official publishers (Jump ads, end pages from Irodori comics) and even an FBI takedown notice and a ksk.moe page screenshot which I assume aren't from MD.

[Squidy]

Also, not sure if it's relevant but I've also seen pages from what looks like official publishers (Jump ads, end pages from Irodori comics) and even an FBI takedown notice and a ksk.moe page screenshot which I assume aren't from MD.

That's so weird. As far as I know, their CDN is volunteer-based through MangaDex@Home, so my guess would be that maybe it's returning other images on their systems that it shouldn't be. I don't know if that's my fault or theirs, but I did notice that the "data-saver" image URLs the module was generating were incorrect. I'm not sure if that's the cause, but I've uploaded a fix for that, and made some other little changes too.

I also added rate limiting when accessing their API and downloading images. The rate limits will be customizable through the UI later, but for now they're hard-coded just as an attempt to fix this problem for now.

Lastly, I added a new setting under Tools > General settings > Advanced settings called "Modules.MangaDex.RetryImagesWithMismatchedContentType", which does what you would expect. Since the "content-type" header seems to be wrong when the incorrect images are downloaded, this will make the program retry downloading the images when that happens. You can enable this if the problem persists and see if it helps.

All of this is part of the latest pre-release, which you can download here.

Unfortunately this is an issue I'm still unable to reproduce myself, so once again, let me know how it goes.

[fusedragon]

Thanks for the update, still having the issue unfortunately. In this case I had 5 swapped images altogether but the extra prompts only appeared for page 1 (which was swapped anyway).

Code: Select all

12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/1-a75c731fbd422ab398aa3417922ae45115a94b2fc70a13f475b8bb3500b23be2.png
12:33:10 mangadexret [!] Content-Type 'image/jpeg' did not match expected type 'image/png'. Retrying the request.
12:33:10 mangadexret [!] Content-Type 'image/jpeg' did not match expected type 'image/png'. Retrying the request.
12:33:10 mangadexret [!] Content-Type 'image/jpeg' did not match expected type 'image/png'. Retrying the request.
12:33:10 hdoujin dow [i] Changing extension from .png to .jpg to match content-type (image/jpeg)
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/2-d0b3c8843a3e50f508c840a49ce198baa041e691e5d396815f8dec831ce96376.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/3-0f621d22de7a9b89aac14f9f3c0ac3129e9b4bbb1e43a9c750862d6d9f047957.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/4-f430bd1e5592588910c662ac26e387201447b3a3226093e3ee6432d619346be5.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/5-dd9e188edb0b8ef4c620673c6f0aaae348f3601570ee3836537928f96042fb27.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/6-0a3f10437a682f38834326e7ef9ea5d47d0bed5911d44ce852d325b33f9d38c2.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/7-ab3d1e13943bcb467f669185d7b90a90efe47eb9a34a85b0020a5fe6f8b6cced.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/8-cce3ba106c88e5127a18b40c5dc387707daf8ae760c5ad991d9aed0b321a5b27.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/9-67edbbbaf5480f92a58c118c458bc8f286015dfd02d1d127ad373a20b46488e7.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/10-62218f48b2d87fce5f0bc4512a9c5007b8d5fe9795617dd6dfeb3fad3d53974b.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/11-43db294589548c490999cad494f54bd8aeb932f0ae5b1f508e26126ea3a18094.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/12-a23084b501e4370356d3ffb1ceb1da12fc6dc6c9c99877eb4a5958327f31821d.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/13-6cfcf247ebe7046f36e781739ac836c4bf0eee86fb56262a7d8222897e7d3900.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/14-3519e9d41c10636602f6cfa78591156aea9273894a1ac96115d23548568d982a.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/15-7231c5a56c16349d58009c88c761324d4215842d692e7fb0a6675dcfff343162.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/16-312189a33ba239759c736fbeb98b6cbe20f41f39902c3fa2ea4474dd40bf78a4.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/17-a4136903a830c0dbda200efcc631f5e21888e6a984ab923f8a10f3481727fa65.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/18-35cbde0c7bf6166927c9f5e6df34dbae33a52fb34daa01984cbc8c3605d9151e.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/19-33266305c4358e0e4fb5c12e17fb44d8b5e212037dc290141a7b3dc1d2a717e5.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/20-80fdbeb5136a1a1d328f1be3db0fb3af2d09ed072e86af06447f0b20f88d0d60.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/21-28f1feebbb6d317afba569720d96e0e76d472c267d57dadb6db8643d3adc45a5.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/22-d5af02ea66b8ca48c7b2ba0a9df663d8f6c44069a25fe8b69f923049071ce657.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/23-c04824a56932d5d62eacdc86d1baac02532e7d3edd2dd8073d6ae4de2596b668.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/24-7b489a7d71bcb3f6597a71d63576174b0fd812ed3b743ff1b8bc00a15566bfe4.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/25-4e9c32ddedae98d80ff1b1929fb1846369e25d3eeb2b333114a0304b51f13044.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/26-ce9ca5cd34f203804cb793a5d244ac5af305e5d652e3684d02cd6215f3eec509.png
12:33:10 download    [i] Downloading https://uploads.mangadex.org/data/cd3866d7c8167007433d3953c54e1043/27-0c7f3f3b481e95f56b16a6ca5daf08f4a7ad325f9f4c53537dca853672ac6efc.png

[Squidy]

which was swapped anyway

Yeah, it only tries a maximum of 3 times before giving up and moving on.

It's back to the drawing board I suppose. I'm really not sure what to try next. Maybe it's a way of penalizing heavy downloaders? Because the decision to respond with a different image happens server-side it's really hard to say why.

I'll keep thinking about it.

[fusedragon]

which was swapped anyway

Yeah, it only tries a maximum of 3 times before giving up and moving on.

Seems unlikely for the swap to occur 3 times in a row but instead of re-downloading by page, would it be possible to re-download the specific pages at the end? Someone mentioned they tried re-downloading until they got the right pages but when I try re-downloading the whole chapter sometimes I get previously correct pages swapped out. If it's possible to just automatically re-download the swapped pages that might work.

It's back to the drawing board I suppose. I'm really not sure what to try next. Maybe it's a way of penalizing heavy downloaders? Because the decision to respond with a different image happens server-side it's really hard to say why.

Logically that would be the case but using different IPs doesn't seem to help... Though I guess it's possible there are other users or the IPs were blacklisted.

[JohnKo23]

When I get replaced files I delete them from the folder they were downloaded to and then do a "force download" in HDD which then only tries to grab the deleted files. It does sometimes take 4 or 5 force downloads to get the remaining correct pages but usually its only 1 or 2 tries. The process works but obviously it takes more time than if we got the correct pages from the start.

I've looked at a bunch of the replace images and there seems to be a lot of explicit and BL ones included lately. In the beginning I saw more credit pages for various scanlation groups. I had one manga earlier today that was 13 pages long and 10 of them were swapped. That was the most for one so far, usually it only gets about 1/4 to 1/3 of them.