File downloading [ SOLVED ]

Fri, 2012-12-07 13:07

#2

zzd10h

Offline

Last seen: 7 years 2 weeks ago

Joined: 2012-08-24 20:56

And you don't want to use

And you don't want to use wget ? (ugly method)

Fri, 2012-12-07 14:45

#3

trixie

Offline

Last seen: 6 months 3 weeks ago

Joined: 2011-02-03 13:58

Re: File downloading [ SOLVED ]

@massi

Use libcurl, it's on OS4depot:

http://www.os4depot.net/index.php?function=showfile&file=development/library/misc/libcurl.lha

AmigaOne X5000-020 / 2GB RAM / Sapphire Pulse Radeon RX 560 / AmigaOS 4.1 Final Edition Update 2

Fri, 2012-12-07 17:05

(Reply to #3) #4

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

@zzd10h I prefer a clean

@zzd10h

I prefer a clean solution.

@trixie

Can it be done in a native way, without additional dependencies?

Fri, 2012-12-07 17:31

(Reply to #4) #5

trixie

Offline

Last seen: 6 months 3 weeks ago

Joined: 2011-02-03 13:58

Re: File downloading [ SOLVED ]

@massi

It's compiled for OS4 - how more native would you like it? :-) If you meant whether the OS provides a function for downloading files then the answer is no, AFAIK.

And it's a link library, not a shared library or object, so there are no dependencies.

AmigaOne X5000-020 / 2GB RAM / Sapphire Pulse Radeon RX 560 / AmigaOS 4.1 Final Edition Update 2

Fri, 2012-12-07 17:59

(Reply to #5) #6

kas1e

Offline

Last seen: 3 months 4 days ago

Joined: 2010-11-30 15:30

@massi Or you can go

@massi

Or you can go very-native-oldshool-way, and write everything from scratch on plain C: handle ip/tcp packets/structures/connects for youself, and download the file as binary data from whatever you need via recv() kind functions. I.e. netinet/in.h, ip.h, maybe tcp.h and co, depends of how deep you want to jump into the roots.

Buld tcp/ip connection like this:

        sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
 
        cli.sin_family = AF_INET;
        cli.sin_addr.s_addr = ip;
        cli.sin_port = port;
 
        if (connect(sock, (struct sockaddr *) &cli,
            sizeof(cli)) < 0) {
            close(sock);
	    printf("No connect\n");
            exit(0);
        }

after you connect, recieve the data to buffer via recvfrom() or recv(), and write to local file , then close socket and co. Of course if server to which you will connect are not your one (which after connection just will send data), but plain http, then you will need to deal with http packets as well manually (so RFC on http protocol, to see the bits you need and co). In general all what you will need to do, its just do some http protocol related stuff, and recieve answer (your data).

http protocol are pretty simply (https support will be another deal, because there as far as i remember openssl will be involved), and it can be enough to do something like this:

 if(getsockopt(sockets[i].s, SOL_SOCKET, SO_ERROR, &error, &err_len) < 0 || error != 0)
     {
      shutdown(sockets[i].s,2);
      close(sockets[i].s);
      sockets[i].state=0;
     }
 else
     write(sockets[i].s,"GET /patch/to_file_for_download/file.zip HTTP/1.0\x0d\x0a\x0d\x0a",54);

And answer will be your data.

I.e. you make a ip / tcp connection yourself from scratch via ip structs, and deals with http.

Fri, 2012-12-07 17:49

(Reply to #6) #7

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

@trixie OK thanks, it can be

@trixie

OK thanks, it can be a way.

@kas1e

I always like the from "scratch" approach :) thanks.

Sat, 2012-12-08 11:54

#8

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

Perhaps it can be achieved

Perhaps it can be achieved through a dedicated AmigaDOS handler / device (HTTP:, ...)?
Anyone aware of this?
Any already existing solutions?

Sun, 2012-12-09 00:03

(Reply to #8) #9

salass00

Offline

Last seen: 6 months 1 week ago

Joined: 2011-02-03 11:27

@Massi Perhaps it can be

@Massi

Perhaps it can be achieved through a dedicated AmigaDOS handler / device (HTTP:, ...)?
Anyone aware of this?
Any already existing solutions?

There is a http-handler from Chris Young that works pretty well:
http://aminet.net/package/comm/www/httphandler

Sun, 2012-12-09 10:28

(Reply to #9) #10

thomas

Offline

Last seen: 3 months 1 week ago

Joined: 2011-05-16 14:23

Why is this less ugly than

Why is this less ugly than the wget approach?

IMHO it is even more ugly because wget is a simple program you can include in your program's drawer, but the http handler needs installation into the user's system directories.

Why don't you just search Google for a simple httpget example and use this in your code?

Sun, 2012-12-09 15:02

(Reply to #10) #11

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

@salass00,

@salass00, @thomas

Thanks.
At this stage I am evaluating possible solutions. Overall I prefer Amiga styled techniques.

Sun, 2012-12-09 18:54

(Reply to #11) #12

PJS

Offline

Last seen: 4 years 10 months ago

Joined: 2012-12-09 18:38

@massi FYI, Roadshow does

@massi

FYI, Roadshow does provide a simple handler that you can use, the TCP: device.

You can access online content just like reading/writing a file with a "TCP:[url]/[port]/..."
syntax. I've used it a fair bit in ARexx both for sending/getting data.

For more info, see:

SYS:Documentation/Roadshow/tcp-handler.doc

Then again, I've been told this isn't the best way to do the fastest or most reliable
transfers. For that, you can use kas1e's low level, longhand approach.

Good luck,

PJS

Sun, 2012-12-09 20:27

(Reply to #12) #13

kas1e

Offline

Last seen: 3 months 4 days ago

Joined: 2010-11-30 15:30

@massi With tcp/ip, "native

@massi
With tcp/ip, "native amiga way" will be the same as any bsd/linux one, just you can use bsdsocket.library (which make no big sense in compare with newlib sockets).

All that downloading of files on lowlevel are discussed a lot by all the hackers for last 20 years everythere , so Thomas's suggestion about getting any gethttp.c from google will almost do all what you need, just some adoptation and understanding will need it.

You can even go deeper if you want real hardcore: once you write a C code for httpget, you can translate that all to the ppc assembler with removing un-necessary parts and optimise it (as most of those hackers do when do shellcodes). For example check x86/linux realisation here. You can see how small is it (and C source is no much bigger), for downloading a file from http and executing it. And as our RoadShow its just almost the same BSD tcp/ip stack, its all will be kind of the same by logic, just different assembler. I.e. will be the same small and lowlevel.

I even found one more example in google which i compile and test on os4 without changes (except that i change ip on os4coding.net's one, and compile it with -D__USE_INLINE__):

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
 
int main ( void ) {
int socket_handle ;
struct sockaddr_in socket_detials ;
char * input_buffer;
char * httpget = "GET HTTP 1.1 / \x0D\x0A\n\x0D\x0A\n" ;
 
input_buffer = malloc(20000);
 
socket_handle = socket ( AF_INET, SOCK_STREAM, 0) ;
socket_detials.sin_family = AF_INET ;
socket_detials.sin_addr.s_addr=inet_addr("68.90.68.66");
socket_detials.sin_port = htons(80);
bzero ( &(socket_detials.sin_zero), 8 ) ;
 
if ( connect (socket_handle,(struct sockaddr*)&socket_detials, sizeof ( struct sockaddr)) == -1 ){
printf ( "Couldnt connect to server\n" ) ;
}
printf ( "Sending %d bytes\n", send ( socket_handle , httpget, strlen(httpget), 0 ) ) ;
printf ( "Received %d bytes\n", recv ( socket_handle , input_buffer , 20000, 0 ) ) ;
printf ( "%s\n", input_buffer ) ;
 
return 0 ;
}

And there is result:

6/0.Work:code_examples> gcc -D__USE_INLINE__ httpget2.c -o httpget

6/0.Work:code_examples> httpget

Sending 21 bytes
Received 1360 bytes
HTTP/1.1 400 Bad Request
Date: Sun, 09 Dec 2012 17:45:40 GMT
Server: Apache/2.2.16 (Debian)
Last-Modified: Tue, 23 Oct 2012 05:33:02 GMT
ETag: "2ca015-700-4ccb34db89f80"
Accept-Ranges: bytes
Content-Length: 1792
Vary: Accept-Encoding
Connection: close
Content-Type: text/html
 
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
    <title>ERROR 400 - Bad Request!</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta name="robots" content="noindex" />
    <style type="text/css"><!--
    body {
        color: #444444;
        background-color: #EEEEEE;
        font-family: 'Trebuchet MS', sans-serif;
        font-size: 80%;
    }
    h1 {}
    h2 { font-size: 1.2em; }
    #page{
        background-color: #FFFFFF;
        width: 60%;
        margin: 24px auto;
        padding: 12px;
    }
    #header {
        padding: 6px ;
        text-align: center;
    }
    .status3xx { background-color: #475076; color: #FFFFFF; }
    .status4xx { background-color: #C55042; color: #FFFFFF; }
    .status5xx { background-color: #F2E81A; color: #000000; }
    #content {
        padding: 4px 0 24px 0;
    }
    #footer {
        color: #666666;
        background: 
 
6/0.Work:code_examples>

I.e. its already works , and get from os4coding http answer. Now you just need to add code for filling correct ip/domen and path to file from whatevewer you need, and save answer to file. Then you can compile it with -S and -O2, and make a real hardcore on asm :)

ps. Blah, that source even from amiga.org archives, there is full thread with details, and Piru's example is much better of course

Mon, 2012-12-10 11:25

(Reply to #13) #14

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

@PJS Thanks, AmigaDOS

@PJS

Thanks, AmigaDOS handlers (HTTP:, TCP:) are definitely what I was looking for.

@kas1e

Thanks, very valuable contribution and really hacker style which I always appreciate ;)

Mon, 2012-12-10 18:12

(Reply to #14) #15

thomas

Offline

Last seen: 3 months 1 week ago

Joined: 2011-05-16 14:23

Here is a Rexx program which

Here is a Rexx program which downloads a file using the TCP handler:

/* rexx */
 
parse arg url
 
if url = "" then url = "http://www.os4coding.net/sites/default/files/interbase_light_logo.png"
 
parse var url prot "://" server "/" path
 
if prot ~= "http" then do
	say "Only http URLs are supported,"
	exit 20
end
 
parse var server server ":" port
 
if server = "" then do
	say "invalid URL."
	exit 20
end
 
if port = "" then port = 80
 
path = "/"path
 
x = lastpos("/",path)
file = translate(substr(path,x+1),"________",":#?()[] ")
 
if file = "" then file = "index.html"
 
file = "RAM:"file
 
say "server = <"server">"
say "port   = <"port">"
say "path   = <"path">"
say "file   = <"file">"
say ""
 
cr = '0d'x
ok = 0
 
if Open(tcp,"tcp:"server"/"port,write) then do
	call WriteLn tcp,"GET" path "HTTP/1.1"cr
	call WriteLn tcp,"Host:" server||cr
	call WriteLn tcp,cr
	maxsize = 0
	l = strip(ReadLn(tcp),'T',cr)
	do while (l ~= "")
		say l
		if word(l,1) = "Content-Length:" then maxsize = word(l,2)
		l = strip(ReadLn(tcp),'T',cr)
	end
	if Open(out,file,write) then do
		size = 0
		do while (~eof(tcp))
			buffer = ReadCh(tcp,1024)
			len = length(buffer)
			size = size + len
			call WriteCh out,buffer
			if maxsize > 0 & size >= maxsize then leave
		end
		call Close(out)
		say ""
		say "wrote" size "bytes to" file
		ok = 1
	end
	else
		say "cannot open output file" file
	call Close(tcp)
end
else
	say "cannot connect to" server "port" port
 
if ok then
	address command 'multiview "'file'"'

Tue, 2012-12-11 17:17

(Reply to #15) #16

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

@thomas Thanks for this good

@thomas

Thanks for this good start, it should be straight to adapt it to C language.

Wed, 2012-12-12 15:05

(Reply to #16) #17

afxgroup

Offline

Last seen: 5 months 1 week ago

Joined: 2011-02-03 15:26

All those solutions doesn't

All those solutions doesn't handle web server response like "moved" or other kind of response.. the best was to use libcurl that can handle everything.. or wget

Sat, 2012-12-15 19:21

(Reply to #17) #18

TSK

Offline

Last seen: 1 year 1 month ago

Joined: 2011-06-28 02:06

@thomas Thanks big times ! I

@thomas

Thanks big times ! I got it finally how to use this TCP: device.

Sat, 2012-12-15 22:41

#19

Hans

Offline

Last seen: 3 months 3 weeks ago

Joined: 2010-12-09 22:04

Re: File downloading [ SOLVED ]

@Massi

I'll add my voice to those suggesting using LibCURL. If you were to write something from scratch, you'll effectively end up writing your own version of LibCURL's HTTP handling code. AmigaOS' networking API is the same BSD sockets that all major OSes uses, and LibCurl calls the BSD socket functions directly. So, LibCURL's code is about as "native" as you can get on all of the platforms that it supports.

There is no real reason to write your own from scratch, unless you're doing it for the learning experience.

Hans

Join the Kea Campus - upgrade your skills; support my work; enjoy the Amiga corner.
https://keasigmadelta.com/ - see more of my work

Mon, 2012-12-17 12:01

(Reply to #19) #20

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

@Hans I agree with you,

@Hans

I agree with you, there is no need to "reinvent the wheel" each time ...
I personally prefer AmigaDOS handlers over other solutions because they are the Amiga way of doing certain things.
Good to list and talk about different techniques anyway.

Thu, 2013-01-03 14:46

(Reply to #20) #21

hypex

Offline

Last seen: 3 months 2 weeks ago

Joined: 2011-09-09 16:20

@thomas Why is this less

@thomas

Why is this less ugly than the wget approach?

Isn't it obvious? ;-)

Tue, 2013-01-08 11:11

(Reply to #21) #22

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

@thomas Does the Rexx script

@thomas

Does the Rexx script work with HTTP 1.1 chunked transfer mode (response arriving in multiple parts)?

Perhaps chunked messages are managed within the TCP: handler or?

Tue, 2013-01-08 12:42

(Reply to #22) #23

thomas

Offline

Last seen: 3 months 1 week ago

Joined: 2011-05-16 14:23

The TCP handler only opens a

The TCP handler only opens a simple TCP (telnet) connection. It does not know HTTP or any other protocol. If you want to support chunked transfers, you need to program that. The example only shows a very simple "get that small file from my own server", for example to check for a new version of your software. It's obviously not a complete implementation of all features of the HTTP protocol.

Wed, 2013-02-13 20:17

(Reply to #23) #24

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

Thanks to all, I now have a

Thanks to all, I now have a working example, it is written in C language and it makes use of the TCP: handler. A bit of refining is needed anyway.
Going forward in my project.

Thu, 2013-04-11 17:23

#25

Massi

Offline

Last seen: 4 years 6 months ago

Joined: 2012-03-28 17:16

For those interested, I have

For those interested, I have finalized this topic to a complete application.
The project name is BigSister, available from OS4 Depot, source code included in the package.

Regards

Search form

User login