Skip to main content

How to wget large files from one server to another
#fediadmins @Friendica Support @Friendica Developers
utopiArte
utopiArte  
•

update:
found the following hints pointing out that php-fpm .ini could be involved:
https://askubuntu.com/questions/332022/changes-for-upload-max-filesize-memory-limit-post-max-size-in-the-php-ini-are

Changes in that .ini and restarting apache2 and the respective -fpm didn't work out either.

Of course we are talking about management with the console over SSH.

utopiArte
utopiArte  
•

To make sure that the settings that rule are the ones I changed, I searched for a comand line command to print the respective values within the console:
php -r "echo ini_get('my_value');"
(the print is a littel bit strange tho, just a number in line before the user handle, like:
8000MrootName@VPShosting:~# )

php -r "echo ini_get('upload_max_filesize');";"
8000M
php -r "echo ini_get('post_max_size');"e');"
8000M
php -r "echo ini_get('max_execution_time');"e');"
0
php -r "echo ini_get('memory_limit');"t');"
-1
php -r "echo ini_get('max_input_time');";"
-1

utopiArte
utopiArte  
•
Just managed to pull two files of 1.9gig and 1.6gig with the existing setting and the command as displayed, so the limitiation is above 1.9gig and most likely a wget problem itself (?).
Daniel
Daniel  
•
If you have SSH access to both systems you can just use SFTP. Wget sounds like the wrong tool for this.
utopiArte
utopiArte  
•

@Daniel

If you have SSH access to both systems you can just use SFTP

Not really.
I do have ftp access to the shared hoster using fileZilla not SSH, so I can use those credentials and SSH with putty over console on the VPS. I was just starting to look into such an option but not sure for the correct command to do so. I'd probably also need to install some ftp package I guess. It's a ubuntu 2022.04 standard setup VPS.

@Daniel
utopiArte
utopiArte  
•

@Rainer "friendica" Sokoll

Read that a lot, but I guess that my access to the shared hosting only with a tool like filezilla isn't enough to use resync.

Also, I'm in the middle of nowhere with internet access thru a cellphone located at a specific 20x10cm spot in the roof where I catch a littel bit of the provider antena that's at a distance of like 12km. So downloading 3.7gig to than upload it with scp is kinda impossible.

@Daniel

@Daniel @Rainer "friendica" Sokoll
Rainer "friendica" Sokoll
Rainer "friendica" Sokoll  
•
You said you have ssh access. That should be enough for god's own rsync 😀
And: rsync can resume (-P option), so you would not have to start over after an interrupted transfer.
utopiArte
utopiArte  
•

Not sure if I understand what you are saying @Rainer "friendica" Sokoll.

As for what I saw, on a brief look in conversations on the matter on the web, was that rsync has to be installed on both sides, on both servers. I ony can install it on the VPS, not on the shared hosting where the files is.

@Daniel

@Daniel @Rainer "friendica" Sokoll
Rainer "friendica" Sokoll
Rainer "friendica" Sokoll  
•
There is a good chance that rsync is already installed on the server side. You may try it.
utopiArte
utopiArte  
•

@Rainer "friendica" Sokoll @Daniel The web say's this is not possible:

It is not possible to use rsync on http URLs.
To use the rsync algorithm the client-side rsync needs to interact with a server-side rsync process. This is done either directly, through ssh or (less common) rsh. HTTP is not an option.
https://stackoverflow.com/questions/33756108/how-can-i-use-rsync-to-retrieve-files-from-an-http-url

Unless I can use sFTP credentials to contact the shared hosting with rsync this doesn't look like an option.

@Daniel @Rainer "friendica" Sokoll
Rainer "friendica" Sokoll
Rainer "friendica" Sokoll  
•

You wrote:

I do have ftp access to the shared hoster using fileZilla not SSH, so I can use those credentials and SSH with putty over console on the VPS.


So I assumed you have ssh access. If you have and rsync is available an both sides, then you really should give rsync a try.

utopiArte
utopiArte  
•

@Rainer "friendica" Sokoll @Daniel

I assumed you have ssh access

Nope.
(actually I detailed that in another answer, I guess you didn't get that displayed)

Right now I most likely will try to get that instance running and up to date on the shared hosting, than try to split up the file storage and than move it in parts.
Another option might be looking for a day with a good cellphone reception and do all of it with scp from over here.
Also, investigating how to do this with sFTP over SSH from the VPS is most likely an appropriate option that add's experience and tools to the general setup, as the wget investigation is kinda "ausgereizt" already.

Last but not least there could be the option to ask anyone else who does have (good fiber optics) connections to get the file onto his VPS and than create a resync setup.
👍

@Daniel @Rainer "friendica" Sokoll
Rainer "friendica" Sokoll
Rainer "friendica" Sokoll  
•
I assumed you have ssh access

Nope.


[…]

investigating how to do this with sFTP over SSH


I am confused. If you do not have ssh access, sftp will fail.

utopiArte
utopiArte  
•

@Rainer "friendica" Sokoll

I am confused. If you do not have ssh access, sftp will fail.

two server

  • [li] server 1
    shared hosting
    hosts the file
    access over sftp
    no access over SSH
    access over https, locating the file in a public folder, access with wget, works at least up to 1.9gig[li] server2
    VPS hosting
    access over SSH
    never intended or tested to access over sFTP till now
    needs/requests the 3.7gig file


Probable sFTP option:
rootName@VPShosting:# someSFTPtool sftpUSERNAME@sftpsSharedhostingID/sharedHostingstoragefolder/file.tar.gz /thisVPSserver/localstorage/file.tar.gz

@Rainer "friendica" Sokoll
⇧