mersenneforum.org  

Go Back   mersenneforum.org > Other Stuff > Archived Projects > NFSNET Discussion

 
 
Thread Tools
Old 2005-07-07, 18:04   #1
Wacky
 
Wacky's Avatar
 
Jun 2003
The Texas Hill Country

2·541 Posts
Default Call for help

Although we are spread extremely thin, some progress is being made in getting new sievers operational.
However, I could certainly use some help.
First, in the code area, I have two fairly small tasks that someone could perform without either a significant knowledge of NFS or a large or ongoing commitment of time.
a) A regression testing program, written in C or C++ that can verify that the output file from a test run "matches" a reference copy. (I can presently do this on Unix-style systems using awk/sed/diff or perl. However, I need something readily portable to many test systems)
b) Bob Silverman has some MP Math routines for x86 machines. These need to be reformatted to compile under gcc. This is a translation task, not a programming task, per se.

The second area where I could use some help is in the area of test machines.
As we are able to get the various sievers running, we need to be able to compile and test for a wide range of system setups. Although we are not yet to that point, I would like to be able to test a wide range of CPU/OS combinations. The setup is fairly well automated. You maintain a copy to the sources on your hard disk. A scheduled task runs a program each night (or other time of the day) to update the sources from our CVS repository and recompile the changed routines. It then runs some tests and reports the results back to my server which posts a summary. By examining the errors and/or timing reports, we can track the effect of code changes on machine configurations that we otherwise would not have access.
To participate, you would need a C compiler development environment and the willingness to participate on a continuing basis.

If you would like to help in either area, please e-mail me.

<join at nfsnet dot org>

Last fiddled with by Wacky on 2005-07-07 at 18:18
Wacky is offline  
Old 2005-07-08, 17:21   #2
Jeff Gilchrist
 
Jeff Gilchrist's Avatar
 
Jun 2003
Ottawa, Canada

7·167 Posts
Default

Quote:
Originally Posted by Wacky
a) A regression testing program, written in C or C++ that can verify that the output file from a test run "matches" a reference copy. (I can presently do this on Unix-style systems using awk/sed/diff or perl. However, I need something readily portable to many test systems)
How closely do they have to match? Identical output files or more refined than that?

If identical, why not just take the MD5/SHA-1 hash of the files to see if they match?

Jeff.
Jeff Gilchrist is offline  
Old 2005-07-08, 17:39   #3
Wacky
 
Wacky's Avatar
 
Jun 2003
The Texas Hill Country

2·541 Posts
Default

How close.

Identical EXCEPT ....
We need to be able to supply the two files to compare and a definition of the acceptably mismatched lines.
For example, a typical ASCII Factor Base has a line or two that contain information about where and when the file was generated. I'm happy to simply skip the matching of those 2 or 3 lines. If it is necessary to go to a full programming language, I'd probably suggest Perl. But I prefer to avoid that dependency.
Wacky is offline  
Old 2005-07-08, 19:07   #4
xilman
Bamboozled!
 
xilman's Avatar
 
"π’‰Ίπ’ŒŒπ’‡·π’†·π’€­"
May 2003
Down not across

3·43·79 Posts
Default

Quote:
Originally Posted by Jeff Gilchrist
How closely do they have to match? Identical output files or more refined than that?

If identical, why not just take the MD5/SHA-1 hash of the files to see if they match?

Jeff.
I observe that Perl5 is freely available for almost all systems, and is much less system-dependent than awk. Cygwin awk, for instance, rarely gave the same results as BSD awk when I tried to port earlier NFSNET code.

sed and its friends seem not to be as widespread as Perl.


Paul
xilman is offline  
Old 2005-07-08, 19:30   #5
Wacky
 
Wacky's Avatar
 
Jun 2003
The Texas Hill Country

2·541 Posts
Default

Quote:
Originally Posted by xilman
I observe that Perl5 is freely available for almost all systems, and is much less system-dependent than awk. Cygwin awk, for instance, rarely gave the same results as BSD awk when I tried to port earlier NFSNET code.

sed and its friends seem not to be as widespread as Perl.


Paul
Paul,
I do not disagree with your statement. That is why I said that if it got to a point where we really needed that level of generality, I would probably choose Perl over other alternatives.

However, there are two problems with Perl:

It is TOO general. That has implications upon data integrity and security. (For example, you might write to the wrong file) A less general tool that does limited things based on its input is less likely to do the wrong thing because of some input data error.

Although it is "freely available", that does not mean that it IS present. Requiring it is just one more setup hurdle that I think we can avoid.

Richard

Last fiddled with by Wacky on 2005-07-08 at 19:33
Wacky is offline  
Old 2005-07-08, 19:31   #6
akruppa
 
akruppa's Avatar
 
"Nancy"
Aug 2002
Alexandria

25×7×11 Posts
Default

What rights on part of the user does the test suite need? I.e. is root access required to install it (for cron jobs or such)?

The only PC I personally own is a P3-500 running Linux which is probably neither a targeted, nor a rare platform. At the University, the only machine I can access freely are x86 (Athlon XP mostly) Linux boxes, a few Opteron 150 running Linux, and some UltraSparc 3 and 4 under SunOS 5.9. The Opteron and Sparc are machines for general use by students so I better get permission before I start scheduled jobs or they won't stay for long.

If any of these would be any use to you, let me know.

Alex
akruppa is offline  
Old 2005-07-08, 19:46   #7
Wacky
 
Wacky's Avatar
 
Jun 2003
The Texas Hill Country

2×541 Posts
Default

Quote:
Originally Posted by akruppa
What rights on part of the user does the test suite need? I.e. is root access required to install it (for cron jobs or such)?

The only PC I personally own is a P3-500 running Linux which is probably neither a targeted, nor a rare platform. At the University, the only machine I can access freely are x86 (Athlon XP mostly) Linux boxes, a few Opteron 150 running Linux, and some UltraSparc 3 and 4 under SunOS 5.9. The Opteron and Sparc are machines for general use by students so I better get permission before I start scheduled jobs or they won't stay for long.

If any of these would be any use to you, let me know.

Alex
It would need no special user permissions, assuming that the general user can compile and execute an arbitrary program. Some systems do require higher access to set up the running of scheduled jobs. Others permit a normal user to run scheduled jobs under their own id. YMMV.

As for the UltraSparc, access to them for the purpose might be of interest if there are more than one or two out there that would want to sieve on a regular basis. However, Dan reports that they have phased out all of theirs and I don't know of anyone in particular that has a number of them which would participate if we had a binary available. However, I will keep you in mind should the need arise.

Richard

Last fiddled with by Wacky on 2005-07-08 at 20:19
Wacky is offline  
Old 2005-07-11, 09:25   #8
Mystwalker
 
Mystwalker's Avatar
 
Jul 2004
Potsdam, Germany

3×277 Posts
Default

I could write an application for a) using Java. Portability shouldn't be an issue, but (at least outside of the Windows world), I guess it's not more often pre-installed than Perl.
In addition, when it is the only Java application and it needs to be run many times, loading issues might impact performance. OTOH, if it is used frequently, maybe a good option would be to let it stay resident and hand over work instead or starting it each time...
Mystwalker is offline  
Old 2005-07-11, 13:13   #9
R.D. Silverman
 
R.D. Silverman's Avatar
 
Nov 2003

723210 Posts
Thumbs up

Quote:
Originally Posted by Wacky
Although we are spread extremely thin, some progress is being made in getting new sievers operational.
However, I could certainly use some help.
First, in the code area, I have two fairly small tasks that someone could perform without either a significant knowledge of NFS or a large or ongoing commitment of time.
a) A regression testing program, written in C or C++ that can verify that the output file from a test run "matches" a reference copy. (I can presently do this on Unix-style systems using awk/sed/diff or perl. However, I need something readily portable to many test systems)
b) Bob Silverman has some MP Math routines for x86 machines. These need to be reformatted to compile under gcc. This is a translation task, not a programming task, per se.

The second area where I could use some help is in the area of test machines.
As we are able to get the various sievers running, we need to be able to compile and test for a wide range of system setups. Although we are not yet to that point, I would like to be able to test a wide range of CPU/OS combinations. The setup is fairly well automated. You maintain a copy to the sources on your hard disk. A scheduled task runs a program each night (or other time of the day) to update the sources from our CVS repository and recompile the changed routines. It then runs some tests and reports the results back to my server which posts a summary. By examining the errors and/or timing reports, we can track the effect of code changes on machine configurations that we otherwise would not have access.
To participate, you would need a C compiler development environment and the willingness to participate on a continuing basis.

If you would like to help in either area, please e-mail me.

<join at nfsnet dot org>

Once upon a time doing this kind of work was my job.
Now I have a real-world job.

I will be happy to provide guidance/advice/discussion to anyone working
on a port of my lattice siever and underlying X86 64-bit arithmetic code, but
I no longer have time to do the port myself. C'est la vie.
R.D. Silverman is offline  
Old 2005-07-11, 13:30   #10
cjohnsto
 
Jun 2005

3·5 Posts
Default

How about the following ANSI C code for comparisons. Sorry about the lack of comments, it should be fairly obvious what I have done. I have tested it a bit but programs can always use more. Especialy those written at midnight as quickly as possible.
usage is program file1 file2 [skipfile] where skipfile contains a sorted list of line numbers (on seperate lines) to skip in both file1 and file2.
If this isn't what you wanted tell me why and what you do want.

Code:
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[])
{
	FILE *skipfile = NULL, *in1 = NULL, *in2 = NULL;
	int skip = -1;
	char line1[1024], line2[1024];
	int lineno = 0;
	int skipped = 1;

	if ((argc < 3) || (argc > 4))
	{
		fprintf(stderr, "usage: %s file1 file2 [skipfile]\nskipfile contains line numbers on seperate lines that will not be compared between file1 and file2 this file must be sorted\n", argv[0]);
		return 1;
	}
	if (argc == 4)
	{
		skipfile = fopen(argv[3], "r");
		if (!skipfile)
		{
			fprintf(stderr, "Could not open %s\n", argv[3]);
			return 2;
		}
	}

	in1 = fopen(argv[1], "r");
	in2 = fopen(argv[2], "r");
	if (!in1)
	{
		fprintf(stderr, "could not open %s\n", argv[1]);
		return 3;
	}
	if (!in2)
	{
		fprintf(stderr, "could not open %s\n", argv[2]);
		return 4;
	}

	while(1)
	{
		int eol = 0;

		lineno++;
		if (skipfile && skipped)
		{
			if (fgets(line1, 1024, skipfile) != NULL)
			{
				skip = atoi(line1);
				skipped = 0;
			}
		}
		while (!eol)
		{
			int i;
			char* res1 = fgets(line1, 1024, in1);
			char* res2 = fgets(line2, 1024, in2);
			if (!res1 || !res2)
			{
				if (feof(in1) || feof(in2))
				{
					if (feof(in1) && feof(in2))
					{
						fprintf(stderr, "Files are Identical\n");
						return 0;
					}
					else
					{
						fprintf(stderr, "Files differ in length\n");
						if (feof(in1))
							printf("%s is shorter\n", argv[1]);
						if (feof(in2))
							printf("%s is shorter\n", argv[2]);
						return -1;
					}
				}
				else
				{
					fprintf(stderr, "Error occured reading one of the files\n");
					return 5;
				}
			}
			
			
			for (i = 0; ; ++i)
			{
				if ((line1[i] == 0) || (line2[i] == 0))
				{
					if (skip == lineno)
					{
						break;
					}
					else if ((line1[i] == 0) && (line2[i] == 0))
						break;
					else
					{
						fprintf(stderr, "Difference on line %d\n", lineno);
						printf("%s\n%s\n", line1, line2);
						return -1;
					}
				}
				else if ((line1[i] == '\n') || (line2[i] == '\n'))
				{
					if (skip == lineno)
					{
						skipped = 1;
						eol = 1;
						break;
					}
					else if ((line1[i] == '\n') && (line2[i] == '\n'))
					{
						eol = 1;
						break;
					}
					else
					{
						fprintf(stderr, "Difference on line %d\n", lineno);
						printf("%s\n%s\n", line1, line2);
						return -1;
					}
				}
				else if (skip == lineno)
				{
					continue;
				}
				else if (line1[i] == line2[i])
				{
					continue;
				}
				else
				{
					fprintf(stderr, "Difference on line %d\n", lineno);
					printf("%s\n%s\n", line1, line2);
					return -1;
				}
			}
		}
		
	}
}
cjohnsto is offline  
Old 2005-07-12, 03:35   #11
Wacky
 
Wacky's Avatar
 
Jun 2003
The Texas Hill Country

108210 Posts
Default

Quote:
Originally Posted by R.D. Silverman
I will be happy to provide guidance/advice/discussion to anyone working on a port of my lattice siever and underlying X86 64-bit arithmetic code, but I no longer have time to do the port myself. C'est la vie.
Bob,
Thanks for the offer. I did not ever consider asking you to do this because, IMHO, if you have any time available for NFS sieving, your time can be better spent on improving the code rather than a task as mundane as translating a page of assembler code from one "dialect" to another. You have already provided the appropriate lines of code. The only problem is that I need to also be able to compile those lines on a different compiler. Hopefully we can get this done soon.
Wacky is offline  
 

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
OPN Project Roll Call ThomRuley Factoring 14 2019-05-25 00:12
Team_Italia call for participants ET_ Teams 6 2018-12-18 12:08
What do you suppose they'll want to call it? schickel Science & Technology 6 2010-04-18 10:36
Call me lazy but... davieddy Puzzles 19 2009-10-06 05:41
Call to arms em99010pepe No Prime Left Behind 2 2008-11-26 13:21

All times are UTC. The time now is 04:24.

Wed Nov 25 04:24:14 UTC 2020 up 76 days, 1:35, 4 users, load averages: 1.33, 1.53, 1.47

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.