This update contains some significant changes, and I recommend creating a backup of your HDoujin Downloader folder before updating, just in case. If you have any problems, let me know, and you can always downgrade to an earlier release here.
New external module system
What makes it significant is that I have (finally) laid the groundwork for an external module system. While the old DIY scripting system was an attempt at this, it was, frankly, a user-unfriendly mess. The new system uses lua, and offers what I believe is a far more comfortable API. You can see an example module here. It shows most of what’s available right now (in addition to a “Log” function for writing things to debug log during testing). In the spirit of getting this update out sooner rather than later, it’s the only example available right now, but I plan to do the majority of work through this system rather than program updates. I am wide-open to suggestions for this feature to help it meet the needs of those wishing to create their own modules.
New repository system
Accompanying this change is a new repository system. Repositories are comprised of a local folder where modules are stored and optionally an online source used for downloading new updates. The “official” repository is here, and this update comes with it already configured for use. Any updates made here will be transparently synced to HDoujin Downloader, allowing fixes and new modules to be provided in the background without having to update manually. Full program updates are still required to be done through the updater, but having modules separate will make this situation less frequent.
You can access the repositories list under “Tools > Modules”. From here you can disable specific modules and change the priorities of your repositories to prefer certain ones over others (in the case that they both provide modules for a given site). You can also add new ones. Right now, only GitHub folders can by synced, but I’m open to adding more options if anyone has interest in hosting their own repositories elsewhere.
The “Scripts” tab is now obsolete and has been removed. If you’re one of the few people that was actually using that feature, it will be automatically replaced by a repository named “Scripts” that you can access under “Tools > Modules”. Old scripts will continue to work, but it’s not the recommended choice for new development.
Lastly, basic sideways-compatibility for lua modules from other programs has been implemented. Many will not work yet, but some do. This will be improved in future releases. If you use another downloading program that provides modules you’d like to import into HDoujin Downloader, let me know, and I’ll see what I can do.
Site updates
Site updates included in this release are summarized in the changelog. Most notably, sites protected by Cloudflare show now be working again. The fix is provided by “cf-tool” in the “bin” folder, which wraps a Python script for bypassing the verification page. It’s not the ideal solution, but it works, especially in the absence of suitable alternatives for .NET. Let me know if you have any problems with Cloudflare going forward.
Antivirus false-positives
This has been an ongoing issue for the past couple of releases, especially with Windows Defender. Whitelisting sometimes takes a while, making things really inconvenient for early adopters. I’ve done my best to try to avoid this problem with this release as much as possible. If you run into problems in this area, let me know so I can submit the program for whitelisting to the respective AV vendor.
What’s next?
Integrating the repository and external module system was no small feat, and I have done extensive testing to make sure that everything works. I did not expect it to take as long as it did, but now that that hurdle is gone, I can hopefully resume the regular weekly update schedule. There will likely be more updates to address limitations with the current module system’s API as I start moving more sites into it and the requirements become clearer. That’s the plan for now.
Always start slowly.
It’s not bad, but it hasn’t started recently.
why ?
Do you have a lot of items in your download queue? By “it hasn’t started recently”, do you mean you’ve been unable to get it to start up at all?
If you have a really large download queue, it can take some time to load up. Optimizing the startup process is definitely something I’d like to work on soon.
Hi, I was downloading a manga and then I got the prompt for an update, so I updated and now I can’t download. I paste the link into the field and nothing happens, the manga doesn’t appear in the download/queue list below.
I’m downloading from Tsumino as it’s easier to get all the manga of a series in one go.
Is there something I could try to get it working or?
Thanks for letting me know about this. I did a little investigating, because it stopped accepting the URLs on my end too. It turns out some changes I made to how the program decides what module to use for a given URL has a bug that made some sites, such as Tsumino, stop getting properly mapped to the respective module. I’ve got it fixed now on my end, so you can expect it to be working normally again with the next update.
For now, you can downgrade to version 1.17.9.32 while you wait, which should still be working fine.
This is awesome, thanks so much!
Thanks! I’m pretty happy with it so far. Some things are still not possible through the Lua API, but I’m going to flesh it out more as I implement more site modules through it.
The downloader became heavier than before, especially when exporting .manga files (manga download list) and starting downloads.
It might be somewhat heavier. How significant is the “heaviness” you’re experiencing? How many items do you have in your download queue?
Yeah, Luscious still isn’t working.
Using the latest update but can’t seem to download the extra modules. I have a license.
“Updater not found” is the error I have.
I think it’s because from the previous release and this one, some files were moved around into a new bin folder.
You’re right, thanks for reporting that– I’ll get that fixed.
In the meantime, you can send me an email and I can send you a direct download link to the extra modules instead. Alternatively, temporarily moving or copying the updater next to “HDoujin Downloader.exe” should do the trick.
Was having a quick look over the default MangaDex lua module and had a couple of questions/ideas.
Is it currently possible to specify the url for the folder/cover image when creating a new manga entry?
In cases where we will need to scrape the page to get the images directly instead of an API like MangaDex provides, i assume the `output.Add(PageInfo.New(pageUrl))` in the parsePages function takes a url to an image directly as well?
Going to have a crack at creating one over the coming weekend hopefully. In addition i see you have a Github repository for your official modules. Wondering whether it might be worth creating a separate Git repo for community contributions rather then having them spread all over the place (But still keeping the ability to override like you stated) or at the very least a Markdown file in the existing repo that community contributes can submit a PR to to add links to their modules so they arent annoying to fine.
Another good example/idea would be implementing a site that can make use of the logins site so custom login functions can be written as i am assuming that sort of information is not exposed from the app itself. Though in terms of security this might be a little tricky to implement in a secure way, though could warn the user by providing a private lua function that modules can use to get the information, then when adding a new repo, scan through the modules and if this module is used through a warning for the user.
Do you have an example of the Log function and what it expects? Or will just dump out everything that is passed into it?
That’s definitely on the agenda. You’re able to define a “DoLogin” function (the presence of which should make the site available in the “add new login” list), but at the moment it’s never called. The security aspect is definitely tricky. If possible, I’d like to avoid giving scripts access to login details altogether, but it might be inescapable (even if I use placeholder values to keep the script from seeing them directly, a malicious person could still send them to any endpoint they want). I need to give it more thought.
The “Log” function takes a single argument that can be converted to a string (i.e. “tostring” can be called on it), which includes Json and StringList objects (the latter of which is returned by things like “betweenMany” and “regexMany”).
So at the moment, if a site already has login functionality built in and i create a new module for that site that overrides one built into the App itself (Say Toomics for example), will i be logged in automatically and the module act as if i am logged in? Or will the module act as if i am not logged in?
Thanks a lot for checking it out!
It’s not possible to specify a cover URL at the moment, though HDoujin Downloader does have fields internally for this. I have not made it accessible at the moment (partially because it’s not used for anything at the moment), but I could do that so that it’s available when needed.
Yes, you can cut out the middleman and use “output.Add(pageUrl)” instead; I used a PageInfo object here for demonstration purposes. It currently does not have any accessible fields aside from “Url”, but I plan to make more of the internal ones available as needed (such as alternate URLs, file extension (for when it needs to be specified manually), subdirectory, etc.).
PRs directly to the official repository are welcome (I’ll add more information about that later). The ability for users to create their own is just there to offer a bit more flexibility.
Thanks a lot for the update 😀
No problem! ♥