mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > PrimeNet > GPU to 72

Reply
 
Thread Tools
Old 2011-12-13, 18:51   #1
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

36·13 Posts
Default Automatic Submission Spider for Workers...

Based on a question by Dubslow about how to do HTTP requests using Bash (the context being automating the submission of results until Christenson completes his work.

I don't know how to do script HTTP in Bash, but I do know how to in Perl... So I've put together a quick script which will hopefully be useful to some of you.

Please see: Submission Spider.

Note that this is currently targeted to Unix users, but it should be possible to adapt it to Windows without too much effort. Advise on how to do that would be appreciated (I don't do Windows).

As always, feedback welcome.

Last fiddled with by chalsall on 2014-03-29 at 13:36
chalsall is offline   Reply With Quote
Old 2011-12-13, 21:00   #2
nucleon
 
nucleon's Avatar
 
Mar 2003
Melbourne

10038 Posts
Default

For those wanting to try on windows - cygwin tools is your best bet.

I've been using a bash script and lynx (command line browser) to auto submit answers and get more work.

-- Craig
nucleon is offline   Reply With Quote
Old 2011-12-13, 22:12   #3
Dubslow
Basketry That Evening!
 
Dubslow's Avatar
 
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88

1C3516 Posts
Default

Can I see the bash script you have nucleon? Thank you chalsall, but it's as much about bash/http as being lazy. (Though I will look at it to learn http... hopefully...)
Dubslow is offline   Reply With Quote
Old 2011-12-13, 22:23   #4
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

36·13 Posts
Default

Quote:
Originally Posted by Dubslow View Post
Can I see the bash script you have nucleon? Thank you chalsall, but it's as much about bash/http as being lazy. (Though I will look at it to learn http... hopefully...)
OK, but I would strongly recommend you learn Perl if you want to get into spidering -- it's one of Perl's sweet spots.

You can spider using Bash, gluing together other programs like lynx, wget, sed, awk, et al. But I would argue you can do everything you need just with Perl (and a few modules), and have much more control and better error handling with less overall work. And half of the job of spidering is dealing with errors...

But as always with software, there's more than one way to do things....
chalsall is offline   Reply With Quote
Old 2011-12-13, 22:38   #5
Dubslow
Basketry That Evening!
 
Dubslow's Avatar
 
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88

722110 Posts
Default

Learning one thing at a time. Programming will come after programming class. Programming in general is not something that interests me, just something that is useful, and I'm not a very good self motivator, and a CS course is mandatory, which should help jumpstart things.
Dubslow is offline   Reply With Quote
Old 2011-12-14, 18:03   #6
Dubslow
Basketry That Evening!
 
Dubslow's Avatar
 
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88

160658 Posts
Default

I was also going to try and use the PrimeNet API.
Dubslow is offline   Reply With Quote
Old 2011-12-15, 09:00   #7
frmky
 
frmky's Avatar
 
Jul 2003
So Cal

22×11×47 Posts
Default

A minor bug correction to the spider...
unlink("submitted_${Now}.txt");
should be
unlink("submitting_${Now}.txt");
frmky is offline   Reply With Quote
Old 2011-12-31, 23:06   #8
oswald
 
oswald's Avatar
 
Apr 2011
in vivo

3·52 Posts
Default

Thanks for recommending cygwin. I was trying to use the unix subsystem that microsoft gives away and it is very limited. cygwin is better for me.

So now I use a bash script to hit the directories and call the submission spider. I use the windows scheduler to call the bash script. Maybe a strange combo, but it only took a 10 minutes to set up.

Anyway, submit_spider is great and I just feed the wordtodo.txt and relax.

Life is good.
oswald is offline   Reply With Quote
Old 2012-01-01, 00:18   #9
flashjh
 
flashjh's Avatar
 
"Jerry"
Nov 2011
Vancouver, WA

1,123 Posts
Default

I installed ActivePerl 5.14.2 Build 1402 (64-bit) for Windows, wrote a couple of batch files to combine all my results.txt files and use Windows scheduler to run everthing. I combine results before submission so I have one 'submitted' file that I archive each time -- makes everything neater and easier to double check once in a while.

The submission spider is awesome! Thanks.

If anyone needs Windows advice, PM me.

Jerry
flashjh is offline   Reply With Quote
Old 2012-01-08, 04:25   #10
oswald
 
oswald's Avatar
 
Apr 2011
in vivo

3×52 Posts
Default

I just ran into a hiccup.
So far it has worked on all submissions I think, but failed on
Code:
no factor for M45833773 from 2^74 to 2^75 [mfaktc 0.18 barrett79_mul32]
Submitting to http://www.mersenne.org/manual_result/ worked ok.
I've tried again expecting a not needed result, but received the following. It is the same error as before.

I'll reboot and try again just to be sure.

Code:
20120108_032034 INFO:  Submission spider starting...
20120108_032034 INFO:  Attempting to log into PrimeNet.  This can take a little while...
20120108_032058 INFO:  Successfully logged into PrimeNet.
Use of uninitialized value $GHzDays in concatenation (.) or string at /usr/local/bin/submit_spider line 151, <IN> line 1.
20120108_032121 INFO:  M49831559 submitted;  GHz Days credit.
20120108_032122 INFO:  Submission complete.  Moving results file to "submitted_20120108_032058.txt"
20120108_032122 INFO:  Spider has finished.  Exiting.
oswald is offline   Reply With Quote
Old 2012-01-08, 04:40   #11
oswald
 
oswald's Avatar
 
Apr 2011
in vivo

3·52 Posts
Default

I'm getting timeouts now. Hold off looking into this until I can verify this.
Thanks,
Roy
oswald is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
GPU to 72 spider poaching exponents? ixfd64 GPU to 72 2 2014-07-27 20:38
It Seems The Great Spider ... Dubslow Forum Feedback 13 2012-06-01 13:39
Automatic temperature control TObject Software 13 2012-04-27 19:33
Went from 8 workers to 4 workers on v26.6 upgrade dmoran Software 13 2011-05-23 12:36
Manual submission of automatic assignment result tichy PrimeNet 4 2010-12-17 09:57

All times are UTC. The time now is 12:37.

Mon Mar 1 12:37:14 UTC 2021 up 88 days, 8:48, 0 users, load averages: 1.54, 1.47, 1.48

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.