mersenneforum.org Automatic Submission Spider for Workers...
 Register FAQ Search Today's Posts Mark Forums Read

 2011-12-13, 18:51 #1 chalsall If I May     "Chris Halsall" Sep 2002 Barbados 36·13 Posts Automatic Submission Spider for Workers... Based on a question by Dubslow about how to do HTTP requests using Bash (the context being automating the submission of results until Christenson completes his work. I don't know how to do script HTTP in Bash, but I do know how to in Perl... So I've put together a quick script which will hopefully be useful to some of you. Please see: Submission Spider. Note that this is currently targeted to Unix users, but it should be possible to adapt it to Windows without too much effort. Advise on how to do that would be appreciated (I don't do Windows). As always, feedback welcome. Last fiddled with by chalsall on 2014-03-29 at 13:36
 2011-12-13, 21:00 #2 nucleon     Mar 2003 Melbourne 10038 Posts For those wanting to try on windows - cygwin tools is your best bet. I've been using a bash script and lynx (command line browser) to auto submit answers and get more work. -- Craig
 2011-12-13, 22:12 #3 Dubslow Basketry That Evening!     "Bunslow the Bold" Jun 2011 40
2011-12-13, 22:23   #4
chalsall
If I May

"Chris Halsall"
Sep 2002

36·13 Posts

Quote:
 Originally Posted by Dubslow Can I see the bash script you have nucleon? Thank you chalsall, but it's as much about bash/http as being lazy. (Though I will look at it to learn http... hopefully...)
OK, but I would strongly recommend you learn Perl if you want to get into spidering -- it's one of Perl's sweet spots.

You can spider using Bash, gluing together other programs like lynx, wget, sed, awk, et al. But I would argue you can do everything you need just with Perl (and a few modules), and have much more control and better error handling with less overall work. And half of the job of spidering is dealing with errors...

But as always with software, there's more than one way to do things....

 2011-12-13, 22:38 #5 Dubslow Basketry That Evening!     "Bunslow the Bold" Jun 2011 40
 2011-12-14, 18:03 #6 Dubslow Basketry That Evening!     "Bunslow the Bold" Jun 2011 40
 2011-12-15, 09:00 #7 frmky     Jul 2003 So Cal 22×11×47 Posts A minor bug correction to the spider... unlink("submitted_${Now}.txt"); should be unlink("submitting_${Now}.txt");
 2011-12-31, 23:06 #8 oswald     Apr 2011 in vivo 3·52 Posts Thanks for recommending cygwin. I was trying to use the unix subsystem that microsoft gives away and it is very limited. cygwin is better for me. So now I use a bash script to hit the directories and call the submission spider. I use the windows scheduler to call the bash script. Maybe a strange combo, but it only took a 10 minutes to set up. Anyway, submit_spider is great and I just feed the wordtodo.txt and relax. Life is good.
 2012-01-01, 00:18 #9 flashjh     "Jerry" Nov 2011 Vancouver, WA 1,123 Posts I installed ActivePerl 5.14.2 Build 1402 (64-bit) for Windows, wrote a couple of batch files to combine all my results.txt files and use Windows scheduler to run everthing. I combine results before submission so I have one 'submitted' file that I archive each time -- makes everything neater and easier to double check once in a while. The submission spider is awesome! Thanks. If anyone needs Windows advice, PM me. Jerry
 2012-01-08, 04:25 #10 oswald     Apr 2011 in vivo 3×52 Posts I just ran into a hiccup. So far it has worked on all submissions I think, but failed on Code: no factor for M45833773 from 2^74 to 2^75 [mfaktc 0.18 barrett79_mul32] Submitting to http://www.mersenne.org/manual_result/ worked ok. I've tried again expecting a not needed result, but received the following. It is the same error as before. I'll reboot and try again just to be sure. Code: 20120108_032034 INFO: Submission spider starting... 20120108_032034 INFO: Attempting to log into PrimeNet. This can take a little while... 20120108_032058 INFO: Successfully logged into PrimeNet. Use of uninitialized value \$GHzDays in concatenation (.) or string at /usr/local/bin/submit_spider line 151, line 1. 20120108_032121 INFO: M49831559 submitted; GHz Days credit. 20120108_032122 INFO: Submission complete. Moving results file to "submitted_20120108_032058.txt" 20120108_032122 INFO: Spider has finished. Exiting.
 2012-01-08, 04:40 #11 oswald     Apr 2011 in vivo 3·52 Posts I'm getting timeouts now. Hold off looking into this until I can verify this. Thanks, Roy

 Similar Threads Thread Thread Starter Forum Replies Last Post ixfd64 GPU to 72 2 2014-07-27 20:38 Dubslow Forum Feedback 13 2012-06-01 13:39 TObject Software 13 2012-04-27 19:33 dmoran Software 13 2011-05-23 12:36 tichy PrimeNet 4 2010-12-17 09:57

All times are UTC. The time now is 12:37.

Mon Mar 1 12:37:14 UTC 2021 up 88 days, 8:48, 0 users, load averages: 1.54, 1.47, 1.48