bash and HTTP?

Dubslow · 2011-12-13, 02:47

Hey guys,
I know Christenson's working on automating mfaktc, but I was wondering, is it possible to incorporate sending HTTP stuff via bash? I'd like to crap together something just for me to report results.

Random other question to get maximum thread efficiency: Is there a way to pause just on worker in MPrime? All I can do is stop them all or start them all, without changing the prime.txt/local.txt settings every time I play Minecraft. I cannot use PauseWhileRunning because for whatever reason it always leaves a bunch of zombie processes that I can't seem to kill for the life of me, but that as far as I can tell take no system resources except lines in the process list. (That is to say, clicking end process or issuing killall Minecraft does nothing to affect the zombie processes, but MPrime still reads them.)

Christenson · 2011-12-13, 04:52

Dubslow:
There are certainly 9 ways to skin the mfaktc automation cat...and a bash (or perl, or Ruby, or...oh, pick your language!) script can certainly be used.

I seem to remember zombie processes as still having slots in the process table, but not doing execution or having remaining memory allocation. I don't think it would be hard under mprime to let you stop just one worker....you can certainly do it in the Windows GUI...

Dubslow · 2011-12-13, 06:42

Well yes, but MPrime doesn't have a GUI...

How would you do HTTP? This is about me learning something about it and being lazy than me automating mfaktc properly. That's beyond my ability and you already claimed it.

Xyzzy · 2011-12-13, 16:18

Quote:

Random other question to get maximum thread efficiency: Is there a way to pause just on worker in MPrime?

In the past we have run four instances of Mprime to be able to control each one individually. It is a bit of a hassle but it works.

Xyzzy · 2011-12-13, 16:20

Quote:

How would you do HTTP?

Start with learning to communicate with the server via Telnet.

chalsall · 2011-12-13, 18:54

Quote:

Originally Posted by Dubslow

I know Christenson's working on automating mfaktc, but I was wondering, is it possible to incorporate sending HTTP stuff via bash? I'd like to crap together something just for me to report results.

I don't know how to do this with just Bash, but I do know how to with Perl. Please see the Automatic Submission Spider for Workers thread.

It should give you a good idea about just how easy it is to build spiders using Perl. Also, hopefully, it will be useful.

Christenson · 2011-12-13, 23:48

I think he would need wget or something to do it with just bash...

Dubslow · 2011-12-14, 17:58

So I could construct my own HTTP requests with wget? That's what a quick google leads me to believe...

chalsall · 2011-12-14, 18:26

Quote:

Originally Posted by Dubslow

So I could construct my own HTTP requests with wget? That's what a quick google leads me to believe...

Sure. That's what wget does. But just to be pedantic, this isn't Bash doing the work -- it's wget being called from Bash (or whatever).

At your console (or a web browser), "man wget".

But I would still argue Perl with the LWP::UserAgent and HTTP::Cookies (optionally et al) modules is a better solution space.

Dubslow · 2011-12-14, 18:37

I'm sure it is, but again, this isn't about being practical. I just want a tool to mess with HTTP, and interacting with PrimeNet seems like a great way to get started (I promise I won't spam/overload it)
Also with bash I can do if [ $(wc -l results.txt) -ge 10 ]; then wget ....
and I'm sure Perl would be great too, but...

chalsall · 2011-12-14, 19:20

Quote:

Originally Posted by Dubslow

I'm sure it is, but again, this isn't about being practical. I just want a tool to mess with HTTP, and interacting with PrimeNet seems like a great way to get started (I promise I won't spam/overload it)
Also with bash I can do if [ $(wc -l results.txt) -ge 10 ]; then wget ....
and I'm sure Perl would be great too, but...

In Perl it's only a little more complicated:

Code:

@LC = split(" ", `wc -l results.txt`);
if ($LC[0] > 10) {
   ...
}

But I must warn you again, half of the work of spidering is dealing with unexpected results and errors.

Nearly the entire other half has to do with dealing with parcing the responses. How do you plan on handling the regular expressions? Sed? Or Awk?

2011-12-13, 02:47	#1
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 16065₈ Posts	bash and HTTP? Hey guys, I know Christenson's working on automating mfaktc, but I was wondering, is it possible to incorporate sending HTTP stuff via bash? I'd like to crap together something just for me to report results. Random other question to get maximum thread efficiency: Is there a way to pause just on worker in MPrime? All I can do is stop them all or start them all, without changing the prime.txt/local.txt settings every time I play Minecraft. I cannot use PauseWhileRunning because for whatever reason it always leaves a bunch of zombie processes that I can't seem to kill for the life of me, but that as far as I can tell take no system resources except lines in the process list. (That is to say, clicking end process or issuing killall Minecraft does nothing to affect the zombie processes, but MPrime still reads them.) Last fiddled with by Dubslow on 2011-12-13 at 02:51

2011-12-14, 18:37	#10
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 7221₁₀ Posts	I'm sure it is, but again, this isn't about being practical. I just want a tool to mess with HTTP, and interacting with PrimeNet seems like a great way to get started (I promise I won't spam/overload it) Also with bash I can do if [ $(wc -l results.txt) -ge 10 ]; then wget .... and I'm sure Perl would be great too, but... Last fiddled with by Dubslow on 2011-12-14 at 18:39

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Connecting to the bash shell in Windows 10	wombatman	Software	10	2018-04-09 02:10
Bash on Ubuntu on Windows	henryzz	Software	11	2017-07-28 21:24
Fedora gedit for bash has become useless	EdH	Linux	11	2016-05-13 15:36
Escape sequences in bash scripts?	CRGreathouse	Software	16	2009-03-26 08:42
Could you help me change this into a bash script?	jasong	Programming	1	2007-11-29 05:59

2011-12-13, 04:52	#2
Christenson Dec 2010 Monticello 11100000011₂ Posts	Dubslow: There are certainly 9 ways to skin the mfaktc automation cat...and a bash (or perl, or Ruby, or...oh, pick your language!) script can certainly be used. I seem to remember zombie processes as still having slots in the process table, but not doing execution or having remaining memory allocation. I don't think it would be hard under mprime to let you stop just one worker....you can certainly do it in the Windows GUI...

2011-12-13, 06:42	#3
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 1110000110101₂ Posts	Well yes, but MPrime doesn't have a GUI... How would you do HTTP? This is about me learning something about it and being lazy than me automating mfaktc properly. That's beyond my ability and you already claimed it.

2011-12-13, 23:48	#7
Christenson Dec 2010 Monticello 703₁₆ Posts	I think he would need wget or something to do it with just bash...

2011-12-14, 17:58	#8
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 1110000110101₂ Posts	So I could construct my own HTTP requests with wget? That's what a quick google leads me to believe...