Parameter explorations for CADO 165-170 digits - Page 4

VBCurtis · 2021-05-20, 15:23

As you go up in relations, you will go up in TD too.

I think if you do this for 190 vs 195 vs 200M rels you'll find out the interesting parts about how extra sieving -> higher TD -> smaller-er matrix -> possible saved time overall. If TD 110 fails to build a matrix, try 105? If you work from the top down, you won't need to try all the TDs for all the rels sizes- the idea is to locate the largest TD that works for each relation set, as that "should" save the most time.

I don't think you can run multiple copies of filtering at the same time unless you're really careful about specifying all the output file names to be different for each copy; else they'll trample each other.

VBCurtis · 2021-05-20, 16:19

Quote:

Originally Posted by bur

BTW, I am just factoring a C147 with your experimental params.c145, should I keep the relations to do similar tests?

No, the files at C150 and below are pretty well tested and refined. It's C155+ that have few jobs run.

bur · 2021-05-22, 06:50

Unfortunately you were right about the simultaneous runs, I hardlinked the rels.dat file to different directories, as I thought it was only read from. But apparently the processes want write access to it, because they all quit at the first singleton removal with an error about writing access. I should build an HDD into that computer...

But if working down from TD=110 on 190,195,200 is sufficient, then it's not too many tests anyway.

charybdis · 2021-05-22, 12:25

Quote:

Originally Posted by bur

Unfortunately you were right about the simultaneous runs, I hardlinked the rels.dat file to different directories, as I thought it was only read from. But apparently the processes want write access to it, because they all quit at the first singleton removal with an error about writing access. I should build an HDD into that computer...

Yes - msieve needs write access to add free relations.

bur · 2021-05-23, 06:56

Here are the results, no major gains, but if using more than 10 cores it will be worth to use 195/105 instead of 190/95. On my 10 core i9 the sieving would take 3:30 h longer with 2:40 h saved at LA, so already at 20 cores it would help a bit.

200M rels / TD=110

Code:

commencing 2-way merge
reduce to 15092846 relation sets and 14808858 unique ideals
commencing full merge
memory use: 682.9 MB
found 21197 cycles, need 3146938
 too few cycles, matrix probably cannot build

200M rels / TD=105 (took 11:54 h for TD=90)

Code:

matrix is 7010462 x 7010687 (2900.8 MB) with weight 774175808 (110.43/col)
sparse part has weight 690320583 (98.47/col)
[...]
 linear algebra completed 70075 of 7010687 dimensions (1.0%, ETA 11h36m)

195M rels / TD=105 (took 13:25 h for TD=90)

Code:

matrix is 7353588 x 7353812 (3046.7 MB) with weight 812669648 (110.51/col)
sparse part has weight 725127644 (98.61/col)
[...]
 linear algebra completed 73595 of 7353812 dimensions (1.0%, ETA 12h49m]

190M rels / TD=100

Code:

commencing 2-way merge
reduce to 17035108 relation sets and 16791673 unique ideals
commencing full merge
memory use: 776.3 MB
found 80952 cycles, need 3526279
too few cycles, matrix probably cannot build

190M rels / TD=95 (took 15:43 h for TD=90)

Code:

matrix is 8132721 x 8132946 (3102.7 MB) with weight 822289451 (101.11/col)
sparse part has weight 732019314 (90.01/col)
[...]
linear algebra completed 81232 of 8132946 dimensions (1.0%, ETA 15h30m)

bur · 2021-05-27, 07:00

The c159 is in progress, found 65 % of the relations. One thing I notice compared to the c163 is that the number of relations per q decreases strongly. Already at q ~ 29e6 it's only 35,000 rels per 10,000 range, whereas with the c163 it still was 35,000 rels at q ~ 66e6. The c159 will end up at q around 43e6.

So apparently that was a lucky polynomial with the c163. I am not sure if that changes anything regarding the parameters.

bur · 2021-05-29, 06:48

The matrix was build at first try with 160,000,000 relations.

Final q sieved: 45300000

matrix is 5597069 x 5597293 (2017.8 MB) with weight 529305051 (94.56/col)
sparse part has weight 472974971 (84.50/col)

Is this as expected or does it make sense to see if fewer relations would work? TD was again the msieve default of 90, while your params.c160 aims for 145. To be honest, I forgot to adjust the msieve parameters...

VBCurtis · 2021-05-29, 16:30

5.5M matrix is still a bit on the small side. I bet 155M relations would work fine; if you don't mind spending the time to check, I'd appreciate it.

Note that CADO and msieve measure density in different ways; I subtract 50 from CADO density to get a rough msieve density, so your msieve default 90 isn't far from the density I guessed is best.

Thanks for the stats report!

Charybdis sent me a params trial and stats report on PM this week for a C165. These two reports provide ideas for a bit of tinkering with the settings. If you're going to do another 158-167 sized job, let me know and I'll send you a new file to try out.

If we're going to try to refine these settings, we will need one more data point: The poly score as reported by *msieve* (or by the optimal skew calculator at http://myfactors.mooo.com/). That's because poly scores can vary by 5% for a given size of input, and we should take that into account when comparing runs.

charybdis · 2021-05-29, 18:37

Quote:

Originally Posted by VBCurtis

If we're going to try to refine these settings, we will need one more data point: The poly score as reported by *msieve* (or by the optimal skew calculator at http://myfactors.mooo.com/). That's because poly scores can vary by 5% for a given size of input, and we should take that into account when comparing runs.

Another interesting data point is the number of unique relations, as reported by msieve filtering, which is a more consistent indicator of when a matrix will build than the number of raw relations. Some polynomials seem to produce more duplicate relations than others, for reasons that I'm not really sure about.

bur · 2021-06-10, 08:06

I missed your reply, I'll run tests on the minimum number of relations after that strange C165 SNFS factorization finishes.

I entered the poly parameters into myfactor, but it came up with a different skew (3.3e6 instead of 2.6e6) than the one I used for the factorization and the scores were on the order of 10^-12 instead of 10^7. I found this surprising, is the score calculated that differently or were the myfactor skews so bad? How can I calculate scores by using msieve?

Regarding unique relations, I'll record those for the c159 tests. I don't have that data for the c164, I could re-run the matrix tests on it, should I just do it for the lowest number of relations that worked?

swellman · 2021-06-10, 12:17

Quote:

Originally Posted by bur

I entered the poly parameters into myfactor, but it came up with a different skew (3.3e6 instead of 2.6e6) than the one I used for the factorization and the scores were on the order of 10^-12 instead of 10^7. I found this surprising, is the score calculated that differently or were the myfactor skews so bad? How can I calculate scores by using msieve?

See this post. The method works quite well.

2021-05-29, 06:48	#40
bur Aug 2020 796581e-4;32539e-3 1320₈ Posts	The matrix was build at first try with 160,000,000 relations. Final q sieved: 45300000 matrix is 5597069 x 5597293 (2017.8 MB) with weight 529305051 (94.56/col) sparse part has weight 472974971 (84.50/col) Is this as expected or does it make sense to see if fewer relations would work? TD was again the msieve default of 90, while your params.c160 aims for 145. To be honest, I forgot to adjust the msieve parameters... Last fiddled with by bur on 2021-05-29 at 06:50

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Some CADO-NFS Work At Around 175-180 Decimal Digits	EdH	CADO-NFS	127	2020-10-07 01:47
Sigma parameter in ecm	storm5510	Information & Answers	4	2019-11-30 21:32
PrimeNet error 7: Invalid parameter	ksteczk	PrimeNet	6	2018-03-26 15:11
Parameter Underestimation	R.D. Silverman	Cunningham Tables	14	2010-09-29 19:56
ECM Work and Parameter Choices	R.D. Silverman	Cunningham Tables	11	2006-03-06 18:46

2021-05-20, 15:23	#34
VBCurtis "Curtis" Feb 2005 Riverside, CA 13134₈ Posts	As you go up in relations, you will go up in TD too. I think if you do this for 190 vs 195 vs 200M rels you'll find out the interesting parts about how extra sieving -> higher TD -> smaller-er matrix -> possible saved time overall. If TD 110 fails to build a matrix, try 105? If you work from the top down, you won't need to try all the TDs for all the rels sizes- the idea is to locate the largest TD that works for each relation set, as that "should" save the most time. I don't think you can run multiple copies of filtering at the same time unless you're really careful about specifying all the output file names to be different for each copy; else they'll trample each other.

2021-05-22, 06:50	#36
bur Aug 2020 796581e-4;32539e-3 1320₈ Posts	Unfortunately you were right about the simultaneous runs, I hardlinked the rels.dat file to different directories, as I thought it was only read from. But apparently the processes want write access to it, because they all quit at the first singleton removal with an error about writing access. I should build an HDD into that computer... But if working down from TD=110 on 190,195,200 is sufficient, then it's not too many tests anyway.

2021-05-27, 07:00	#39
bur Aug 2020 796581e-4;32539e-3 1320₈ Posts	The c159 is in progress, found 65 % of the relations. One thing I notice compared to the c163 is that the number of relations per q decreases strongly. Already at q ~ 29e6 it's only 35,000 rels per 10,000 range, whereas with the c163 it still was 35,000 rels at q ~ 66e6. The c159 will end up at q around 43e6. So apparently that was a lucky polynomial with the c163. I am not sure if that changes anything regarding the parameters.

2021-05-29, 16:30	#41
VBCurtis "Curtis" Feb 2005 Riverside, CA 1011001011100₂ Posts	5.5M matrix is still a bit on the small side. I bet 155M relations would work fine; if you don't mind spending the time to check, I'd appreciate it. Note that CADO and msieve measure density in different ways; I subtract 50 from CADO density to get a rough msieve density, so your msieve default 90 isn't far from the density I guessed is best. Thanks for the stats report! Charybdis sent me a params trial and stats report on PM this week for a C165. These two reports provide ideas for a bit of tinkering with the settings. If you're going to do another 158-167 sized job, let me know and I'll send you a new file to try out. If we're going to try to refine these settings, we will need one more data point: The poly score as reported by msieve (or by the optimal skew calculator at http://myfactors.mooo.com/). That's because poly scores can vary by 5% for a given size of input, and we should take that into account when comparing runs.

2021-06-10, 08:06	#43
bur Aug 2020 796581e-4;32539e-3 2⁴·3²·5 Posts	I missed your reply, I'll run tests on the minimum number of relations after that strange C165 SNFS factorization finishes. I entered the poly parameters into myfactor, but it came up with a different skew (3.3e6 instead of 2.6e6) than the one I used for the factorization and the scores were on the order of 10^-12 instead of 10^7. I found this surprising, is the score calculated that differently or were the myfactor skews so bad? How can I calculate scores by using msieve? Regarding unique relations, I'll record those for the c159 tests. I don't have that data for the c164, I could re-run the matrix tests on it, should I just do it for the lowest number of relations that worked?