Parameter explorations for CADO 165-170 digits - Page 6

VBCurtis · 2021-09-02, 15:33

Each polyselect process runs two-threaded, with each thread working independently. Using adrange of 8 * incr means each thread gets four leading coefficients to search. It is unavoidable that one set of four will finish before the other, so every polyselect process will run single-threaded for some time at the end of its workunit.

This is part of the reason why using hyperthreads gains speed with CADO.

I haven't changed anything else on params.c165; in fact, I haven't solved a job that big since you did so.

bur · 2021-09-02, 17:30

Quote:

This is part of the reason why using hyperthreads gains speed with CADO.

I didn't know that. For a 10 core should I use 5 clients à 4 threads or 10 à 2? And does sieving also benefit from hyperthreading? I didn't notice that problem during sieving.

VBCurtis · 2021-09-02, 21:28

I leave all CADO processes the default two-threaded unless I run low on memory. But number of threads per process is separate from number of total threads used- CADO defaults to the total number of hyperthreads on a machine, and that is clearly faster than number of cores. In fact, someone 'round these parts said they run 40-threaded on an 18 core HT machine because the overhead threads for DB recording etc are quite slow. Can't say I've tried that extreme myself...

Anyway, if you run 20 threads on a 10-core the specifics don't seem to matter.

bur · 2021-09-07, 12:06

Ok, good to know. Does it apply to sieving as well? I never observed non-utilized cores during sieving.

I started sieving a c167 from 1992:1644 on Sunday using a polynomial I searched previously with the parameters from your file but instead using admax=5e6 instead of 1e6. Cownoise is 5.7e-13 so nothing too extreme, but will this interfere with adjusting the other parameters?

Specifically in my two runs on ~165 digit numbers I had a low number of duplicates (+70% uniques) and could go with 190M rels. Though I think that the quality of the poly does not influence the number of duplicates? Unfortunately I never checked the polys from those two previous composites.

VBCurtis · 2021-09-07, 14:22

More poly select doesn't interfere with anything else- perhaps 2x more poly select saves you 5% of sieve time; that's around half a digit of difficulty. I have poly select calibrated to take about 5% of the total job time; if you double poly select and find a poly that sieves 5% faster, the net gain in job time is roughly zero.

The number of uniques varies from poly to poly, and not in a way correlated to score. A much-better poly might sieve over a substantially smaller Q-region, which ought to lead to fewer duplicates, but otherwise it's pretty random.

bur · 2021-09-08, 06:04

In an older post I saw why that initial c163 sieved so fast:

Code:

skew 8385289.96, size 8.965e-16, alpha -7.745, combined = 1.077e-12 rroots = 5

The record is 1.096e-12.

bur · 2021-09-19, 16:08

I'm currently at mksol for the c167 (took 197M relations). I started sieving at q = 17M. Since you mentioned earlier that a value of 17M was likely too large and the unique percentage is high (76%) I wanted to see what happens when I resieve the lower q-ranges and try and build a matrix with those included.

Could you post the appropriate command line for las? I checked the help but there are a lot of parameters that apparently aren't used normally in the command line, at least from what I remember seeing in top.

swellman · 2021-09-20, 01:23

Quote:

Originally Posted by bur

I'm currently at mksol for the c167 (took 197M relations). I started sieving at q = 17M. Since you mentioned earlier that a value of 17M was likely too large and the unique percentage is high (76%) I wanted to see what happens when I resieve the lower q-ranges and try and build a matrix with those included.

Could you post the appropriate command line for las? I checked the help but there are a lot of parameters that apparently aren't used normally in the command line, at least from what I remember seeing in top.

See https://www.mersenneforum.org/showthread.php?t=26319 for a worked example using las. Might not have all the answers you seek but it could help.

bur · 2021-09-20, 09:04

Thanks, that's very helpful. I also found that the commandline is saved to the sieving file.

bur · 2021-09-23, 16:00

I did some tests on a C167 to see the differences between various q-min values.

I sieved in the region q0=10M to q1=108M with the parameters from curtis' file. The I concatenated the sieve files of various regions and made msieve build a matrix.

These are the results:

Code:

--
q=15M-104M (89M)

202836566 relations
50758954 duplicates and 152199568 unique relations

matrix is 10451195 x 10451420 (3780.8 MB) with weight 983955554 (94.15/col)
sparse part has weight 886591531 (84.83/col)
--

--
q=15M-105M (90M)

204611251 relations
51190337 duplicates and 153542870 unique relations

matrix is 10205132 x 10205356 (3689.9 MB) with weight 960718330 (94.14/col)
sparse part has weight 865223080 (84.78/col)
--

--
q=15M-106M (91M)

206510998 relations
51623746 duplicates and 154887252 unique relations

matrix is 9999089 x 9999314 (3614.4 MB) with weight 941369223 (94.14/col)
sparse part has weight 847506249 (84.76/col)
--

--
q=16M-105M (89M)

201149205 relations
found 49063275 duplicates and 152207886 unique relations

matrix is 10458080 x 10458305 (3784.1 MB) with weight 984860664 (94.17/col)
sparse part has weight 887403885 (84.85/col)
--

--
q=16M-106M (90M)

202926996 relations
found 49492280 duplicates and 153556672 unique relations

matrix is 10212921 x 10213146 (3693.3 MB) with weight 961603325 (94.15/col)
sparse part has weight 866042288 (84.80/col)
--

--
q=16M-107M (91M)

204688978 relations
49920242 duplicates and 154890692 unique relations

matrix is 10010922 x 10011147 (3618.0 MB) with weight 942278762 (94.12/col)
sparse part has weight 848335435 (84.74/col)
--

--
q=17M-106M (89M)

199442289 relations
found 47446144 duplicates and 152118101 unique relations

matrix is 10476553 x 10476778 (3789.4 MB) with weight 986164792 (94.13/col)
sparse part has weight 888588171 (84.82/col)
--

--
q=17M-107M (90M)

201204271 relations
found 47869683 duplicates and 153456544 unique relations

matrix is 10231547 x 10231772 (3699.3 MB) with weight 963128811 (94.13/col)
sparse part has weight 867441644 (84.78/col)
--

--
q=17M-108M (91M)

202962678 relations
48294306 duplicates and 154790328 unique relations

matrix is 10028735 x 10028960 (3624.6 MB) with weight 943951416 (94.12/col)
sparse part has weight 849864687 (84.74/col)
 --

I compared runs with the same length of the q-range, i.e. 89M, 90M and 91M. Strangely a q-min of 16M is the best choice in terms of uniques yield, while the smallest matrix was achieved with q-min of 15M.

I don't know exactly how many unique relations are required to build a matrix, but 150.7M failed, 152M succeeded.

My conclusion would be to start at q0=15M with rels_wanted of 203M-205M. Are there any more ranges or other tests that would be of interest?

VBCurtis · 2021-09-23, 16:40

I'd like to see 10-100M and 10-105M. The CADO group reports that duplicate ratio rises meaningfully when q-max is beyond 8 * q-min. Your data is all within that ratio, but you started sieving at 10M so you have a chance to measure a range outside that ratio to see how duplicates and matrix building behave.

A faster test would be to try remdups on 15-108, 12-108. 10-108; we can do some subtraction to see how the duplicate ratio is on specifically 12-15 and 10-12 within the data set of "sieved to 108M". Those Q-ranges look much faster than higher ranges, but if there are over 50% duplicates down there (when filtered with the entire dataset) then the faster sieving is an illusion. We don't need full filtering / matrix generation runs there, just the part until "xxx raw relations, yyy unique".

2021-09-02, 21:28	#58
VBCurtis "Curtis" Feb 2005 Riverside, CA 2²·3³·53 Posts	I leave all CADO processes the default two-threaded unless I run low on memory. But number of threads per process is separate from number of total threads used- CADO defaults to the total number of hyperthreads on a machine, and that is clearly faster than number of cores. In fact, someone 'round these parts said they run 40-threaded on an 18 core HT machine because the overhead threads for DB recording etc are quite slow. Can't say I've tried that extreme myself... Anyway, if you run 20 threads on a 10-core the specifics don't seem to matter. Last fiddled with by VBCurtis on 2021-09-02 at 21:28

2021-09-07, 12:06	#59
bur Aug 2020 796581e-4;32539e-3 2⁴×3²×5 Posts	Ok, good to know. Does it apply to sieving as well? I never observed non-utilized cores during sieving. I started sieving a c167 from 1992:1644 on Sunday using a polynomial I searched previously with the parameters from your file but instead using admax=5e6 instead of 1e6. Cownoise is 5.7e-13 so nothing too extreme, but will this interfere with adjusting the other parameters? Specifically in my two runs on ~165 digit numbers I had a low number of duplicates (+70% uniques) and could go with 190M rels. Though I think that the quality of the poly does not influence the number of duplicates? Unfortunately I never checked the polys from those two previous composites. Last fiddled with by bur on 2021-09-07 at 12:07

2021-09-08, 06:04	#61
bur Aug 2020 796581e-4;32539e-3 2⁴×3²×5 Posts	In an older post I saw why that initial c163 sieved so fast: Code: skew 8385289.96, size 8.965e-16, alpha -7.745, combined = 1.077e-12 rroots = 5 The record is 1.096e-12. Last fiddled with by bur on 2021-09-08 at 06:05

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Some CADO-NFS Work At Around 175-180 Decimal Digits	EdH	CADO-NFS	127	2020-10-07 01:47
Sigma parameter in ecm	storm5510	Information & Answers	4	2019-11-30 21:32
PrimeNet error 7: Invalid parameter	ksteczk	PrimeNet	6	2018-03-26 15:11
Parameter Underestimation	R.D. Silverman	Cunningham Tables	14	2010-09-29 19:56
ECM Work and Parameter Choices	R.D. Silverman	Cunningham Tables	11	2006-03-06 18:46

2021-09-02, 15:33	#56
VBCurtis "Curtis" Feb 2005 Riverside, CA 2²×3³×53 Posts	Each polyselect process runs two-threaded, with each thread working independently. Using adrange of 8 * incr means each thread gets four leading coefficients to search. It is unavoidable that one set of four will finish before the other, so every polyselect process will run single-threaded for some time at the end of its workunit. This is part of the reason why using hyperthreads gains speed with CADO. I haven't changed anything else on params.c165; in fact, I haven't solved a job that big since you did so.

2021-09-07, 14:22	#60
VBCurtis "Curtis" Feb 2005 Riverside, CA 2²×3³×53 Posts	More poly select doesn't interfere with anything else- perhaps 2x more poly select saves you 5% of sieve time; that's around half a digit of difficulty. I have poly select calibrated to take about 5% of the total job time; if you double poly select and find a poly that sieves 5% faster, the net gain in job time is roughly zero. The number of uniques varies from poly to poly, and not in a way correlated to score. A much-better poly might sieve over a substantially smaller Q-region, which ought to lead to fewer duplicates, but otherwise it's pretty random.

2021-09-19, 16:08	#62
bur Aug 2020 796581e-4;32539e-3 2⁴×3²×5 Posts	I'm currently at mksol for the c167 (took 197M relations). I started sieving at q = 17M. Since you mentioned earlier that a value of 17M was likely too large and the unique percentage is high (76%) I wanted to see what happens when I resieve the lower q-ranges and try and build a matrix with those included. Could you post the appropriate command line for las? I checked the help but there are a lot of parameters that apparently aren't used normally in the command line, at least from what I remember seeing in top.

2021-09-20, 09:04	#64
bur Aug 2020 796581e-4;32539e-3 2⁴·3²·5 Posts	Thanks, that's very helpful. I also found that the commandline is saved to the sieving file.

2021-09-23, 16:40	#66
VBCurtis "Curtis" Feb 2005 Riverside, CA 2²×3³×53 Posts	I'd like to see 10-100M and 10-105M. The CADO group reports that duplicate ratio rises meaningfully when q-max is beyond 8 * q-min. Your data is all within that ratio, but you started sieving at 10M so you have a chance to measure a range outside that ratio to see how duplicates and matrix building behave. A faster test would be to try remdups on 15-108, 12-108. 10-108; we can do some subtraction to see how the duplicate ratio is on specifically 12-15 and 10-12 within the data set of "sieved to 108M". Those Q-ranges look much faster than higher ranges, but if there are over 50% duplicates down there (when filtered with the entire dataset) then the faster sieving is an illusion. We don't need full filtering / matrix generation runs there, just the part until "xxx raw relations, yyy unique".