mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Forum Feedback (https://www.mersenneforum.org/forumdisplay.php?f=61)
-   -   Could you allow attachments of documents and spreadsheets? (https://www.mersenneforum.org/showthread.php?t=27027)

drkirkby 2021-07-26 18:52

Could you allow attachments of documents and spreadsheets?
 
Would you consider allowing people to attach documents, such as .doc, .odt and spreadsheets such as .xls and .ods? I realise that one could put them in a zip file, but they are not as convenient to see.

paulunderwood 2021-07-26 18:55

What about viruses and macros? :unsure:

kriesel 2021-07-27 00:34

[CODE]Valid file extensions: 7z bz2 c cpp csv exe gif gz ini jpg log lz md5 npg pdf png svg tgz txt wav xz zip[/CODE][CODE][B]Filetype[/B][B] Max Filesize [/B][B]Max Width[/B][B] Max Height[/B]
[B]7z[/B] 4.00 MB - -
[B]bz2[/B] 4.00 MB - -
[B]c[/B] 1.00 MB - -
[B]cpp[/B] 1.00 MB - -
[B]csv[/B] 1.00 MB - -
[B]exe[/B] 1.00 MB - -
[B]gif[/B] 1.00 MB 4096 4096
[B]gz[/B] 4.00 MB - -
[B]ini[/B] 1.00 MB - -
[B]jpg[/B] 1.00 MB 4096 4096
[B]log[/B] 1.00 MB - -
[B]lz[/B] 4.00 MB - -
[B]md5[/B] 1.00 MB - -
[B]npg[/B] 1.00 MB - -
[B]pdf[/B] 4.00 MB - -
[B]png[/B] 1.00 MB 4096 4096
[B]svg[/B] 1.00 MB 4096 4096
[B]tgz [/B] 4.00 MB - -
[B]txt[/B] 1.00 MB - -
[B]wav[/B] 1.00 MB - -
[B]xz[/B] 4.00 MB - -
[B]zip [/B]4.00 MB - -[/CODE]Multiple compressed file formats, image formats, csv is a spreadsheet import/export format, and complex documents with layout including spreadsheets with graphs and charts and embedded images can be posted as pdf.

Viliam Furik 2021-07-27 00:37

One could also make a link to a shared folder in some cloud storage... Dropbox, Google Drive, OneDrive, or any other.

Uncwilly 2021-07-27 05:27

There is also the option to use the fine pastebin tags. [PASTEBIN][/PASTEBIN] There are reasons that the file types that are allowed and the ones that aren't aren't.

Xyzzy 2021-07-27 12:41

We just added support for .doc files.

If we can find suitable (16×16) icons for the others we will add those.

:mike:

axn 2021-07-27 14:08

[url]https://github.com/LibreOffice/core/tree/master/sysui/desktop/icons/hicolor/16x16/apps[/url]
[url]https://github.com/LibreOffice/core/tree/master/sysui/desktop/icons/hicolor/128x128/apps[/url]
etc..

drkirkby 2021-07-27 14:56

[QUOTE=Xyzzy;584056]We just added support for .doc files.

If we can find suitable (16×16) icons for the others we will add those.

:mike:[/QUOTE]Thank you. The first link provided by axn, for LibreOffice, has some pretty decent 16x16 icons. For example, calc.png.

[URL]https://github.com/LibreOffice/core/blob/master/sysui/desktop/icons/hicolor/16x16/apps/calc.png[/URL]
would do fine for .ods, .xlx and xlsx spreadsheets.
[URL]https://github.com/LibreOffice/core/blob/master/sysui/desktop/icons/hicolor/16x16/apps/writer.png[/URL]
would do well for the .odt text document.

For those suggesting linking to other sites, the problem I see with that is that files get deleted on the other accounts, or people lose access to the accounts.
[LIST][*]If someone is running short of space, they will delete things of minor importance to them. A spreadsheet they created long ago is not going to be high on their priority list.[*]People lose access to accounts. They don't log into Google or whatever for a year, and the account gets closed.[/LIST]As for macros and viruses, I think most programs like spreadsheets will not execute macros without the user specifically enabling it. Viruses are a bit different, but people need to use their common sense. There must be millions of Word documents that can be downloaded off the internet.

@kriesel I don't think PDF works well to display large spreadsheets, and even poorer if someone might want to take that spreadsheet and edit it themselves.

drkirkby 2021-07-27 15:05

[QUOTE=Uncwilly;584035]There is also the option to use the fine pastebin tags. There are reasons that the file types that are allowed and the ones that aren't aren't.[/QUOTE]There are good reasons for not allowing .exe files, but the reasons for some other files formats can't be uploaded is probably just a case of nobody asking for them, rather than them being "not allowed".

The .doc file has been added by Xyzzy today just because I asked, and he has said others can be added if icons were found. User axn has given a link to some nice 16x16 icons.

Xyzzy 2021-07-27 15:45

We have added .odp, .ods, .odt, .ppt and .xls to the list.

We think we have the [URL="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types"]mime types[/URL] input correctly.

Please check to see if they work.

:mike:

kriesel 2021-07-27 15:53

Could the zip file formats' max size be doubled?

a1call 2021-07-27 16:13

[QUOTE=paulunderwood;584019]What about viruses and macros? :unsure:[/QUOTE]


[QUOTE=Xyzzy;584056]We just added support for .doc files.

If we can find suitable (16×16) icons for the others we will add those.

:mike:[/QUOTE]

[QUOTE]
Some macro viruses are annoying but not harmful; others can be very destructive. Also, Word macro viruses can spread across platforms; for example, the macro virus can infect files on the Windows platform as well as on the Macintosh platform.

[/QUOTE]

[url]https://support.microsoft.com/en-us/topic/frequently-asked-questions-about-word-macro-viruses-b935417e-9064-8121-6e63-26f115d7290a[/url]


Sound like an unnecessary liability to me.
A disclaimer at the minimum seems like a must.

ETA FTR, I am not likely to knowingly download any documents posted by random users anytime soon even utilising sandboxes and antivirus. If they can't protect banks and credit agencies, my defenses don't stand much of a chance either.

Xyzzy 2021-07-27 16:46

[QUOTE=kriesel;584075]Could the zip file formats' max size be doubled?[/QUOTE]The forum currently uses half of its allocated storage. Moving beyond that requires a total system change. We haven't done the change because we don't know what will break and also because we are lazy.

So we are letting things ride as they are while hoping that the day to change over never arrives. Keeping file attachments "small" helps.

:mike:

masser 2021-07-27 18:07

[QUOTE=Xyzzy;584083]The forum currently uses half of its allocated storage. Moving beyond that requires a total system change. We haven't done the change because we don't know what will break and also because we are lazy.

So we are letting things ride as they are while hoping that the day to change over never arrives. Keeping file attachments "small" helps.

:mike:[/QUOTE]

Would it be possible to reduce the forum's current usage by removing old attachments of little-to-no worth?

Can you see a list of the largest files in storage?

M344587487 2021-07-27 18:36

[QUOTE=Xyzzy;584083]The forum currently uses half of its allocated storage. Moving beyond that requires a total system change. We haven't done the change because we don't know what will break and also because we are lazy.

So we are letting things ride as they are while hoping that the day to change over never arrives. Keeping file attachments "small" helps.

:mike:[/QUOTE]
How hard would it be to re-compress all zip into .tar.xz (or LZMA/2 .7z if you must)? Zip is archaic, it doesn't matter that it can support more modern algorithms if DEFLATE is still mostly what's used for compatibility. I'm all for annoying windows users that can't be bothered to install a proper archiver (if I was the compression czar I'd rule with a well-compressed iron fist).

While we're talking formats, JpegXL (.jxl) should become the new standard image format for lossless and lossy images (by merit, standardisation and common sense, so I give it a 50% chance). It can losslessly recompress lossy jpegs with a significant size saving, and is far better than png at compressing lossless images. I encourage compression nerds to try it out even though it's going to be many years before it's commonplace in the wild.

Xyzzy 2021-07-27 20:43

1 Attachment(s)
The forum has 20GB allocated to it and it is using ~10GB.

Every year disk space costs less.

We figure by the time we get close to using it up, they might roll the system into a new instance with a bigger drive.

Or we'll face the system jump, which could be entirely painless.

All of the easy disk space optimizations have been done.

The forum mersenne.org download mirror uses 3.7GB so as we close in on the limit we can buy some breathing room by deleting that.

Deleting old attachments is possible but we prefer not to do it. (Just like old posts!)

We can see the biggest attachments. (We are guilty of having most of the large ones!)

In short: The forum is doing okay, but we see no need to introduce larger attachments. However, we will consider it and if you have something big to post just send it to us and we'll attach it manually. Maybe it will turn out that 8MB zip files are needed.

:mike:

drkirkby 2021-07-27 21:29

3 Attachment(s)
[QUOTE=Xyzzy;584074]We have added .odp, .ods, .odt, .ppt and .xls to the list.

We think we have the [URL="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types"]mime types[/URL] input correctly.

Please check to see if they work.

:mike:[/QUOTE]In order to check if they work, here are 3 LibreOffice attachments. The checksums are in the filenames. Looking at LibreOffice, it has a "Math" tool, which saves files with the extension .odf, which is probably worth supporting, given the nature of the forum. An icon is here
[URL]https://github.com/LibreOffice/core/blob/master/sysui/desktop/icons/hicolor/16x16/apps/math.svg[/URL]

There's also a drawing tool, which saves in .odg format too.

[B]EDIT. I downloaded the 3 files, and they opened fine in LibreOffice - there was no need to select the application, as the application was suggested. They opened in "read-only" mode. I also saved the files and compared their md5 checksums, to those of the files I uploaded. The checksums agree. [/B]

xilman 2021-07-27 21:34

[QUOTE=Xyzzy;584074]We have added .odp, .ods, .odt, .ppt and .xls to the list.

We think we have the [URL="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types"]mime types[/URL] input correctly.

Please check to see if they work.

:mike:[/QUOTE]With all due respect, I think this is a bad idea.

Let someone else carry the can for hosting malware-infected files.

Spreadsheets can be exported as TSV or CSV files. PDF works well for most everything else.

kriesel 2021-07-27 23:52

[QUOTE=Xyzzy;584118]
We can see the biggest attachments. (We are guilty of having most of the large ones!)

In short: The forum is doing okay, but we see no need to introduce larger attachments. However, we will consider it and if you have something big to post just send it to us and we'll attach it manually. Maybe it will turn out that 8MB zip files are needed.

:mike:[/QUOTE]Initial reason I asked about larger attachments is #8 of [URL]https://www.mersenneforum.org/showpost.php?p=553120&postcount=26[/URL]. I have exhausted most but not quite all possibilities, but have not yet achieved compilation small enough that will fit in a 4MB compressed archive. Then got distracted by other things.
See also [URL]https://mersenneforum.org/showpost.php?p=556432&postcount=24[/URL], in which the trailing "soon" should now be read as "some month or year". [url]https://mersenneforum.org/showpost.php?p=556447&postcount=30[/url] says the compiled and compressed were running around 8MB.

My custom program for log analysis, error alerting, results gathering, and other features yet to be added for multiple instances of multiple common GPU applications is >3.4MB compressed, and growing.

Re: largest total accumulated size of attachments: Whoo-hoo, made the top 5 without even trying!
If/when total space gets tight, I'd gladly advise on which Gpuowl Windows builds could be dropped first.
[QUOTE=xilman;584125]Let someone else carry the can for hosting malware-infected files.
Spreadsheets can be exported as TSV or CSV files. PDF works well for most everything else.[/QUOTE]I'm not fond of making malware spread easier either. But note that spreadsheets with graphs or font formatting don't export fully in TSV or CSV. I use PDF for them. Storage efficiency is sometimes poor in PDF. As an example, a 55 KB .odt produces a 267 KB pdf, nearly 5x as large.

chalsall 2021-07-28 00:09

[QUOTE=xilman;584125]With all due respect, I think this is a bad idea.[/QUOTE]

Security vs. Convenience is an ongoing balance. No one is ever happy. And most are not nearly paranoid "enough".

[QUOTE=xilman;584125]Spreadsheets can be exported as TSV or CSV files.[/QUOTE]

Doesn't easily retain equations for cells. This can both be a good thing (no attack vectors) and bad (no "rich content").

[QUOTE=xilman;584125]PDF works well for most everything else.[/QUOTE]

PDFs can contain code that can in some cases be considered an attack vector.

Are we there yet?

Uncwilly 2021-07-28 00:23

[QUOTE=chalsall;584137]Doesn't easily retain equations for cells. This can both be a good thing (no attack vectors) and bad (no "rich content").[/QUOTE]
"This is cell A1","25","100"
"The total is","=SUM(B1:C1)"

The user can choose to go to cell B2 and hit enter. Boom, we have a formula. Complex formulas can mess with that (but then again placing your literal values in cells and referencing them can fix many of those problems). Fixed width or using an alternate separator can help.

xilman 2021-07-28 00:50

1 Attachment(s)
[QUOTE=chalsall;584137]Doesn't easily retain equations for cells. This can both be a good thing (no attack vectors) and bad (no "rich content").[/QUOTE]Works for me.

I created three columns of ten rows. Col A has numbers 1..10 and Col B 4..40 by 4. Col C is the sum of Col A and Col B.

Saves just fine in CSV format and reads back in with precisely the expected behaviour.

Here is the spreadsheet in question:

[QUOTE]1,4,"=SUM(A1:B1)"
2,8,"=SUM(A2:B2)"
3,12,"=SUM(A3:B3)"
4,16,"=SUM(A4:B4)"
5,20,"=SUM(A5:B5)"
6,24,"=SUM(A6:B6)"
7,28,"=SUM(A7:B7)"
8,32,"=SUM(A8:B8)"
9,36,"=SUM(A9:B9)"
10,40,"=SUM(A10:B10)"[/QUOTE]

Please give it a try by cutting and pasting everything within the QUOTE tags as test.csv and loading that file into the spreadsheet program of your choice. Alternatively, open the attachment. Each should give the same result.

kriesel 2021-07-28 05:09

5 Attachment(s)
.ods ->.csv -> .ods
Font formatting: gone (bold, italic, underlined, color)
Border formats: gone
Color backgrounds: gone
Conditional formatting: gone
Column widths: gone
Cell merges: gone
Charts: gone
Multiple sheets: no, choose one at a time.

If people are already complaining about a few clicks to zoom pdfs, and they are, there's no way they'll tolerate the extra effort of importing and deciphering what it takes to re-create effective formatting.

a1call 2021-07-28 05:23

[QUOTE=chalsall;584137]

PDFs can contain code that can in some cases be considered an attack vector.

Are we there yet?[/QUOTE]

Yes, and they can be launched without enabling-JavaScript/"Trust"-ing-"this"-document, which is probably why Adobe has discontinued Flash. But as you said it is a balancing act. Still, it seems unwise to enable extensions which are disabled by default in forum software.

LaurV 2021-07-28 06:39

I still don't get it what the complain is, as long as the zip files are allowed. Make a zip of what you (general you) want to attach, and that's it. Then the safety part, as well as responsibility of scanning, etc., will fall on the downloader, not on the forum.

paulunderwood 2021-07-28 06:54

[QUOTE=LaurV;584174]I still don't get it what the complain is, as long as the zip files are allowed. Make a zip of what you (general you) want to attach, and that's it. Then the safety part, as well as responsibility of scanning, etc., will fall on the downloader, not on the forum.[/QUOTE]

I disagree. Restricting attack vectors should be enforced. Sure, one can upload virus.exe.zip but the mods can keep an eye on such zips. Macros hidden in spreadsheets, word processor documents and presentations taxes policing.

LaurV 2021-07-28 08:50

We mainly say the same thing, nothing to disagree about (unless I understand your post wrong, or you mine). As somebody here would say, we are in violent agreement. People requested more extensions to be allowed, and my question was "why?". What can't you attach, as long as zip was already permitted? Why should we lower the security by allowing docs and excel files, etc?

drkirkby 2021-07-28 09:18

[QUOTE=kriesel;584133]Initial reason I asked about larger attachments is ...[/QUOTE]I just unzipped your uploader.zip, which is 2020344 bytes in length, converted to a tar file, then used lmza compression on that. The length of the compressed file was 168867 bytes, so 83.6% of the zipped version. Attempts to reduce the size further with more aggressive optimisation on lzma did not improve upon the defaults.[CODE]drkirkby@canary:~/tmp$ ls -lrt
total 7996
-rw-rw-r-- 1 drkirkby drkirkby 530552 Apr 21 2020 libssl-1_1-x64.dll
-rw-rw-r-- 1 drkirkby drkirkby 2785400 Apr 21 2020 libcrypto-1_1-x64.dll
-rw-rw-r-- 1 drkirkby drkirkby 1102968 Apr 29 2020 libcurl-x64.dll
-rw-rw-r-- 1 drkirkby drkirkby 43520 Jul 25 2020 uploader.exe
-rw-rw-r-- 1 drkirkby drkirkby 1688675 Jul 28 09:31 uploader.tar.lzma
-rw-rw-r-- 1 drkirkby drkirkby 2020344 Jul 28 09:42 uploader.zip
drkirkby@canary:~/tmp$ echo 1688675/2020344 | bc -l
.83583538248931865068[/CODE]

xilman 2021-07-28 10:12

[QUOTE=kriesel;584163].ods ->.csv -> .ods
Font formatting: gone (bold, italic, underlined, color)
Border formats: gone
Color backgrounds: gone
Conditional formatting: gone
Column widths: gone
Cell merges: gone
Charts: gone
Multiple sheets: no, choose one at a time.

If people are already complaining about a few clicks to zoom pdfs, and they are, there's no way they'll tolerate the extra effort of importing and deciphering what it takes to re-create effective formatting.[/QUOTE]So you are a form over function man. Doesn't matter what it does as long as it looks pretty.

Just give me the data. If I want it to look pretty I will beautify it myself.

As for the complainants I have absolutely no sympathy for them. If they find that intolerable they weren't very interested in the information anyway.

chalsall 2021-07-28 16:08

[QUOTE=xilman;584142]Please give it a try...[/QUOTE]

Hmmm... Cute hack.

But, at least for LibraOffice Calc under Fedora 34, I had to go to each cell and delete the equals sign and then insert it again before it became a calculated cell. Simply clicking on a cell, then clicking in the editing field and hitting enter didn't work.

kriesel 2021-07-28 16:12

[QUOTE=xilman;584188]So you are a form over function man. Doesn't matter what it does as long as it looks pretty.
[/QUOTE]No. Formatting promotes readability. It is the responsibility of a poster to communicate effectively. That is function.

Oh, and
header fields: gone (not just the formatting, the content too)
footer fields: gone (not just the formatting, the content too)
In some of my spreadsheets, background or font color encode/represent status or class of the exponent in the cell; formatting IS content there too.
(example: one color text means unassigned, another means in TF, another in P-1, another in primality test)


All times are UTC. The time now is 08:28.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.