Automatska provjera promjene datoteke na serveru

[svebee]

Tražim postoji li neki program/automatizirani način provjere promjene nekoliko desetaka datoteka na XX serverima?

Npr. imam http://www.server1.com/datoteka1.pdf, Npr. imam http://www.server2.com/datoteka151.pdf, Npr. imam http://www.server4.com/datoteke/datoteka1.pdf...

I želio bih da mi program provjeri jesu li se one mijenjale (ne treba biti ograničeno na ime, već na i veličinu npr. tako da ime može ostati isto, ali se datoteka mijenjala) od zadnjeg puta kada sam im pristupio/provjeravao za "update"?

Hvala puno

[kian]

Mozda Fsum Frontend.

[svebee]

hmmm...nisam ga pronašao korisnim za to što ja tražim, ili nisam dobro gledao?

tražim nešto gdje ne bi uopće trebao "skidati" datoteku (ajmo reći ručno) već bi ja upisao

http://www.website.com/datoteka1.pdf
http://www.website.com/datoteka2.pdf
http://www.website.com/datoteka3.pdf

kliknuo "Check" i on bi napisao "File datoteka2.pdf on http://www.website.com has changed since the last check" - tako nekako

[Rulac]

svebee, izprogramirao sam ti program koji radi baš to što trebaš.

U check.txt upises URL-ove koje zeliš testirati (svaki URL u novi red).
Zatim pokreneš last-modified.exe koji će kreirati report.txt

u koji se zapisuje ovakvo nešto:
Tue, 30 Oct 2007 19:06:43 GMT http://sunearthday.nasa.gov/2008/mat..._1920x1200.jpg
Tue, 30 Oct 2007 19:07:39 GMT http://sunearthday.nasa.gov/2008/mat..._1920x1200.jpg
Fri, 09 Feb 2007 19:08:47 GMT http://sunearthday.nasa.gov/2007/mat...007_folder.pdf

Program je skroz spartanski bez opcija.
Radi tako da requesta header, te ako server vrača header sa podatkom "Last-Modified" zapisuje u report.txt
Ubacio sam ti par primjera u check.txt, gdje je Opera Setup exe dosta velika no svejedno brzo provjeri kada je uplodana.

Neki serveri ne vračaju header Last-Modified tako da ti neki URL-ovi neće raditi.

Program možeš skinuti tu: last-modified.zip

[svebee]

hmmm...meni ne radi =/ Windows XP SP2 / 32 bit.

niti jedan link ne radi...

[Rulac]

Autor svebee

hmmm...meni ne radi =/ Windows XP SP2 / 32 bit.

niti jedan link ne radi...

Uplodao sam novu verziju. Sada bi trebalo raditi.

[Bubba]

Autor Rulac

Uplodao sam novu verziju. Sada bi trebalo raditi.

Svaka cast na trudu, ali FFS, citav megabajt za dobiti HTTP header i to jos CLI aplikacija?

@svebee

 /*Project 1: Simple HTTP Client and Server

  For CS586(Computer Network Architectures), Fall 2002
  2002-10-05
  
  Bao Jie 
  Dept of Computer Science
  Iowa State University
  baojie@cs.iastate.edu
  
Command line usage: client [-h] [-d <time-interval>] <URL>

The client takes two options "-h" and "-d" and a required argument <URL>. <URL> specifies 
the URL of the object that the client is requesting from server. The format is 
http://hostname[:port]/filepath. Option "-h" specifies that the client only wants the 
response headers to be sent back from server. You should use HEAD method in your HTTP 
request when "-h" is specified in the command line. Option "-d" along with its argument 
<time-interval> specify that the client wants the object only if it was modified in the 
past <time-interval>. <time-interval> is in the format "day:hour:minute" For example, 
if "-d 1:2:15" is included in the command line, then the client wants to get the object
only if it was modified in the past 1 day, 2 hours and 15 minutes. When -d option is 
specified, the client should include an "If-Modified-Since:" header in the HTTP request 
and the value of this header is computed using the current time and the <time-interval> 
argument.

In client.c, you need to parse the <URL> given in the command line, connect to the server, 
construct an HTTP request based on the options specified in the command line, send the 
HTTP request to server, receive an HTTP response and display the response on the screen. 
You are provided with the code segments for parsing the URL. Please click here to get the 
code. 
*/

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <stdio.h>
//#include <curses.h>
   


#define LOCAL_SERVER_PORT 9798
#define MAX_MSG 100
#define END_LINE 0x0A

#define SUCCESS 0
#define ERROR   1

#define MAX_STR_LEN 512 
#define BUFFER_SIZE 10000       

/*---------------------------------------------------------------------------*
 * 
 * Parses an url into hostname, port number and resource identifier.
 *
 *---------------------------------------------------------------------------*/
parse_URL(char *url, char *hostname, int *port, char *identifier)
{
    char protocol[MAX_STR_LEN], scratch[MAX_STR_LEN], *ptr=0, *nptr=0;
    
    strcpy(scratch, url);
    ptr = (char *)strchr(scratch, ':');
    if (!ptr)
    {
    fprintf(stderr, "Wrong url: no protocol specified\n");
    exit(ERROR);
    }
    strcpy(ptr, "\0");
    strcpy(protocol, scratch);
    if (strcmp(protocol, "http"))
    {
    fprintf(stderr, "Wrong protocol: %s\n", protocol);
    exit(ERROR);
    }

    strcpy(scratch, url);
    ptr = (char *)strstr(scratch, "//");
    if (!ptr)
    {
    fprintf(stderr, "Wrong url: no server specified\n");
    exit(ERROR);
    }
    ptr += 2;

    strcpy(hostname, ptr);
    nptr = (char *)strchr(ptr, ':');
    if (!nptr)
    {
    *port = 80; /* use the default HTTP port number */
    nptr = (char *)strchr(hostname, '/');
    }
    else
    {    
    sscanf(nptr, ":%d", port);
    nptr = (char *)strchr(hostname, ':');
    }

    if (nptr)
      *nptr = '\0';

    nptr = (char *)strchr(ptr, '/');
    
    if (!nptr)
    {
    fprintf(stderr, "Wrong url: no file specified\n");
    exit(ERROR);
    }
    
    strcpy(identifier, nptr);

}

parse_TIME(char *time, int *day, int *hour, int *min)
{
    char scratch[MAX_STR_LEN], *ptr=0, *oldptr = 0;
    char TEMP[MAX_STR_LEN];
    
    strcpy(scratch, time);
    ptr = (char *)strchr(scratch, ':');
    if (!ptr)
    {
        printf("Wrong time format, code 1\n");
        exit(ERROR);
    }
    strcpy(ptr, "\0");
    strcpy(TEMP, scratch);
    *day = atoi(TEMP);
    
    oldptr = ptr;
    ptr = (char *)strchr(ptr+1, ':');
    if (!ptr)
    {
        printf("Wrong time format, code 2\n");
        exit(ERROR);
    }
    strcpy(ptr, "\0");
    strcpy(TEMP, oldptr+1);
    *hour = atoi(TEMP);
    
    strcpy(TEMP, ptr+1);
    *min = atoi(TEMP);        
}


main( int argc, char *argv[ ])
{
    char url[MAX_STR_LEN]; //="http://popeye.cs.iastate.edu:9798/index.html/";
    char hostname[MAX_STR_LEN];
    int port;
    char identifier[MAX_STR_LEN];
    
    int sd, rc, i;
    struct sockaddr_in localAddr, servAddr;
    struct hostent *h;

    char *request=0;//[MAX_STR_LEN];
    char buf[MAX_STR_LEN];

    char line[MAX_MSG];
    char buffer[BUFFER_SIZE];
    int len, size; 
    
    int head_flag=0, date_flag =0, default_flag = 0;
    char s_time[30];  //value to be used in "If-Modified-Since" header

//    printf("\nargc = %d\n", argc);
    request = buf;

    if ( 1 == argc)
    {
        printf("\n Usage: client [-h] [-d <time-interval>] <URL> \n");
        exit(ERROR);
    }


    for ( i = 1; i < argc ; i ++)
    {
        if( (strcmp(argv[i],"-l") == 0 ) ||  (strcmp(argv[i],"-L") == 0 ) )
        {     default_flag = 1;  }

        if( (strcmp(argv[i],"-h") == 0 ) ||  (strcmp(argv[i],"-H") == 0 ) )
            head_flag = 1;

        else if( (strcmp(argv[i],"-d") == 0 ) ||  (strcmp(argv[i],"-D") == 0 ) )
        {
            int day=0, hour=0, min=0; // get from <time-interval> argument 
            time_t n_time;

            if ( i == argc-1)
            {
                printf("\n Usage: client [-h] [-d <time-interval>] <URL> \n");
                exit(ERROR);
            }

            date_flag = 1;
//            printf("\n"); printf(argv[i+1]); printf("\n");
            parse_TIME(    argv[i+1] ,// <time-interval>
                 &day, &hour, &min);

            n_time=time(0);
            n_time=n_time-(day*24*3600+hour*3600+min*60);
            strcpy(s_time, (char *)ctime(&n_time)); 
//            printf("\n%s\n",s_time);
            i++;

        }
        else 
        {
            strcpy(url, argv[i]);        
        }
    }
    
    if (default_flag)
        strcpy(url ,"http://www.cs.iastate.edu:80/~baojie/");

    // In client.c, you need to parse the <URL> given in the command line, connect to 
    // the server, construct an HTTP request based on the options specified in the 
    // command line, send the HTTP request to server, receive an HTTP response and 
    // display the response on the screen. 
    parse_URL(url, hostname, &port, identifier);

    printf("\n-- Hostname = %s , Port = %d , Identifier = %s\n", hostname, port, identifier);


    if(head_flag)
        strcpy (request ,"HEAD "); 
    else
        strcpy (request ,"GET "); 
    
    request = (char *)strcat( request, identifier);
    request = (char *)strcat( request, " HTTP/1.1\r\nHOST: ");
    request = (char *)strcat( request, hostname);
    request = (char *)strcat( request, "\r\n");
    
    if(date_flag)
    {
        request = (char *)strcat( request, "If-Modified-Since: ");
        request = (char *)strcat( request, s_time);
        request = (char *)strcat( request, "\r\n");
    }

    request = (char *)strcat( request, "\r\n");

//    printf("HTTP request =\n%s\nLEN = %d", request, strlen(request));
//    exit(1);
//    getch();

    h = gethostbyname(hostname);
    if(h==NULL) 
    {
        printf("unknown host: %s \n ", hostname);
        exit(ERROR);
    }

    servAddr.sin_family = h->h_addrtype;
    memcpy((char *) &servAddr.sin_addr.s_addr, h->h_addr_list[0], h->h_length);
    servAddr.sin_port = htons(port);//(LOCAL_SERVER_PORT);

    // create socket 
    printf("-- Create socket...    ");
    sd = socket(AF_INET, SOCK_STREAM, 0);
    if(sd<0) 
    {
        perror("cannot open socket ");
        exit(ERROR);
    }

    //  bind port number 
    printf("Bind port number...  ");
    
    localAddr.sin_family = AF_INET;
    localAddr.sin_addr.s_addr = htonl(INADDR_ANY);
    localAddr.sin_port = htons(0);
  
    rc = bind(sd, (struct sockaddr *) &localAddr, sizeof(localAddr));
    if(rc<0) 
    {
        printf("%s: cannot bind port TCP %u\n",argv[0],port);//(LOCAL_SERVER_PORT);
        perror("error ");
        exit(ERROR);
    }
                
    // connect to server 
    printf("Connect to server...\n");
    rc = connect(sd, (struct sockaddr *) &servAddr, sizeof(servAddr));
    if(rc<0) 
    {
        perror("cannot connect ");
        exit(ERROR);
    }

    // send request
    printf("-- Send HTTP request:\n\n%s", request);
    rc = write(sd, request, strlen(request));
    if(rc<0) 
    {   
        perror("cannot send data ");
        close(sd);
        exit(ERROR);  
        }
    
    // display response
/*       memset(buffer,0x0,BUFFER_SIZE);    //  init line 
    if ((rc=read(sd, buffer, BUFFER_SIZE)) < 0) {
            perror("read");
            exit(1);
        }
    printf("-- Recieved response:\n\tfrom server: %s, IP = %s,\n\tlength = %d,%d\n\n%s\n\n",
        url, inet_ntoa(servAddr.sin_addr), rc, strlen(buffer), buffer);         
*/    
    printf("-- Recieved response:\n\tfrom server: %s, IP = %s,\n\n",
        url, inet_ntoa(servAddr.sin_addr));
    rc = 0, size = 0;
    do
       {
        memset(buffer,0x0,BUFFER_SIZE);    //  init line 
        rc = read(sd, buffer, BUFFER_SIZE);
        if( rc > 0)
        { 
            printf("%s",buffer);
            size +=rc;
        }
//        printf("  %d  ",size);
    }while(rc>0);

    printf("\n   Total recieved response bytes: %d\n",size);

     // return
    close(sd);
    return SUCCESS;
}

Ovo je kud i kamo vise nego sto tebi treba, pa mozes urediti kod (koji je poprilicno neucinkovit i "C prljav") i smanjtit ga na minimum (a i ovako kompajliran ima citavih 15 kb).

[Rulac]

Autor Bubba

Svaka cast na trudu, ali FFS, citav megabajt za dobiti HTTP header i to jos CLI aplikacija?

Bubba, moj kod ima oko 83 retka zajedno sa komentarima. Problem je u tome što ja znam raditi jedino Web aplikacije pa sam u PHP-u isprogramirao taj programčić (tj pola toga skopirao sa nekog prethodnog projekta), i nakraju prebacio u CMD pomoću Bambalam PHP EXE Compiler/Embeddera. Veličina je skočila jer samo morao includati neke DLL-love za par funkcija.

Šta moš

[Bubba]

Autor Rulac

Bubba, moj kod ima oko 83 retka zajedno sa komentarima. Problem je u tome što ja znam raditi jedino Web aplikacije pa sam u PHP-u isprogramirao taj programčić (tj pola toga skopirao sa nekog prethodnog projekta), i nakraju prebacio u CMD pomoću Bambalam PHP EXE Compiler/Embeddera. Veličina je skočila jer samo morao includati neke DLL-love za par funkcija.

Zanimljivo ovo cudo, hmm...

Je's da je overhead monstruozan, ali...

Šta moš

Objaviti svoj kod?

[Rulac]

Autor Bubba

Zanimljivo ovo cudo, hmm...

Je's da je overhead monstruozan, ali...

Objaviti svoj kod?

Ovo je last-modified.php

<?php
echo "\r\nRead URL-s from check.txt file.\r\n\r\n";

if (file_exists("check.txt")){
	// reads entire file into a string
	$file		= file_get_contents("check.txt");

	$lines		= split_file_by_line($file);

	// blank var
	$txt_array	= array();

	foreach ($lines as $line_num => $line_string){

			if(preg_match("/^[a-zA-Z]+[:\/\/]+[A-Za-z0-9\-_]+\\.+[A-Za-z0-9\.\/%&=\?\-_]+$/i", $line_string)){



				$link		= $line_string;


				$ch		= curl_init();
				$timeout	= 25; // set to zero for no timeout
				curl_setopt($ch, CURLOPT_URL, $link);
				curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
				curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
				curl_setopt($ch, CURLOPT_HEADER, true);
				curl_setopt($ch, CURLOPT_NOBODY, true);
				$header		= curl_exec($ch);
				curl_close($ch);

				$lines		= split_file_by_line($header);



				foreach ($lines as $key => $line){
					// find Last-Modified
					if (strpos($line, "Last-Modified: ") !== false){
						$line_array		= explode("Last-Modified: ", $line);
						$txt_array[$line_num]	= $line_array[1]."\t".$line_string;
						echo "Checking: $line_string\r\n\r\n";
					}
					unset($line_array);
				}
		}
	}
} else {
	echo "\r\nError: file check.txt not found.\r\n\r\n";
}
if (count($txt_array) > 0){
	echo "\r\nCreate report.txt file.\r\n\r\n";
	
	// create filename
	$fp = fopen("report.txt", "w");
	fwrite($fp, $txt_line."autor: Matej Posavec\r\n");
	fwrite($fp, $txt_line."web: www.rulac.net\r\n\r\n");
	
	foreach ($txt_array as $txt_line){
		fwrite($fp, $txt_line."\r\n");
	}
	
	fclose($fp);
}

// debug
/*
echo "<pre>";
print_r($txt_array);
echo "</pre>";
*/



function split_file_by_line($file){
	// convert windows to unix
	$file		= str_replace("\r\n", "\n", $file);
	// convert mac to unix
	$file		= str_replace("\r",  "\n", $file);
	// split a string by string
	$lines		= explode("\n", $file);
	return $lines;
}
?>

I na kraju pretvorim sa

bamcompile -c -e:php_curl.dll -d -i:icon.ico last-modified.php

php_curl.dll sam nabavio sa php-4.4.4-Win32.zip, a on da bi radio treba libeay32.dll and ssleay32.dll.

Vjerojatno bi se CURL mogao zamjeniti sa nekom starijom PHP funkcijom, ali velim bio je to copy/past nabrzaka pa je ispalo kako je ispalo. Također i ove kozmetička ikona je poždrala prostora.