Modal login option for download url?

I have a number of downloads that I want the user to be logged in for. I am finding very inconsistent behaviour with how they can login.

I’ve set a login page and turned on modal login, but in these are not used unless I specifically create a button for them. I want them to be used for download links. Where a download says “login to download this file” it gives a link to my backend login.

In fact it gave the wrong link, as my site has the backend in a folder, so I had to edit the functions file and the class package to fix this line in multiple places:
$package[‘download_url’] = home_url(‘/admin/wp-login.php?redirect_to=’ . urlencode($_SERVER[‘REQUEST_URI’]));

Clearly I could now edit this code specificy my set login page, but actually – what I want here is to use the modal login. How can I do that?

I don’t actually want to be editing any of the core files, as an update will break everything.