The MyNews-Project: Step-8: Download Restrictions


This step will discuss how to prevent MyNews from excessive downloading. If we set no limits and tell MyNews "download everything automatically", then you will have to stay 36 hours online per day :-)

The default installation already restricts automatic downloads to 100 lines per message. So nearly no binaries are downloaded. And the standard-setup for downloads from the Usenet is "headers only". So no messages are downloaded at all.

MyNews has very powerful and sophisticated settings - but also very simple ones:  

The very simple restrictions

If you never want to download messages automatically, then you use this strategy for downloads from MyNews-Servers:

Dont use the START-Button. Instead you use:
MainWindow - MyNews - Login        (this does not download)
MainWindow - MyNews - Get headers  (gets the header-lines for survey)
Select all messages you want to see and mark them for download
MainWindow - MyNews - Get marked bodies
MainWindow - MyNews - Logout

The download from Usenet-Servers is already restricted to headers.
You already have to select all messages manually.
 

The "normal" restrictions

If you subscribe to a MyNews-Newsgroup, then you normally want to know what happens there. But you dont want to spent hours with downloading everything. The default setup downloads all messages with a size up to 100 lines (maximum about 8000 bytes). This is enough to get all the discussions and announcements. All larger messages (binaries) must be marked manually.  (Just a few messages in longer threads are longer than 100 lines).
 

Less severe restrictions

If you see more and more messages with a size up to 300 lines, and you are tired of downloading them all manually, then you can change the default-restriction-values. We will now show this in detail with the "maximum line restriction" as an example:

First you open the download-setup-settings:
MainWindow - Config - ParameterSetup - Fiter.Tab

Download everything

You can set the restriction for "maximum lines" to 15000. Or 99999. Then MyNews will download _everything_ from other MyNews-Servers. I do not recommend this - except for longterm-servers which are online for a long time.

IMPORTANT:  Programmer's efficiency (laziness) requires a method to "switch off" a condition. While it would be possible to have a checkbox for every single condition, MyNews uses the more easy approach (easy for the programmer):  If the value is ZERO, then the condition is not checked - it is OFF.  

Restrictions for downloading headers

Normally downloading headers is fast. But if you join a busy newsgroup for the very first time, then 10.000 messages can be there. You dont know this before. You see it just when the download of headers starts. It makes sense to have also restrictions for the header-download:

xmax=1000

This limits the download of headers to 1000.
If there are less new headers in a group, then the restriction is not used. If there are MORE headers, then just the FIRST 1000 headers are downloaded. You dont loose anything. The next time you download headers you will get the next 1000 lines... until everything is in your Xover-Window.
I recommend to change this value just temporarily. If you join "alt.mag.playboy" and download headers from your providers server, then 12.000 message can be there. You have to use the "download headers" 12 times. If you change the value to 50000, then download header and then change it back, then it is easier for you.

xfirst=0

xfirst limits the header-download to the first ### items.
This is a special setup for automatic running servers.

xlast=0

xlast limits the header-download to the last ### items.
Again this is a special setup for automatic servers.

xpercent=0

This is a sophisticated setting. If you set this value to "xpercent=50", then MyNews will download just the first half of the messages from a server. This makes sense if you want to get a "complete" feed and download "missing items" from a fixed server.
I use this setting for my "fill-holes-servers". Normally I try to get everything from my three main-feeds. But my server # 4 is more complete - but slower. So I try to get everything from server #1-3 first. The next day MyNews checks the "elder" messages from server #4. Then just a few messages are downloaded slowly.
Again this is a setting for 'automatic' servers.

max_groups

If the group-list is generated automatically, then you can limit the number of groups for downloading headers. (Only for sophisticated combinations with other news-servers as DNews).
 

Content based Restrictions

OnlyGood

This is a pretty nice option. Perhaps you remember, that you can mark single senders as "Good" in the Xover-Window.
If you activate the "onlygood"-option, then the downloaded headers are checked for "good senders" (identical match). Messages from these good senders are downloaded automatically.

If a 'sender' turns bad (or boring), then I remove him manually from the "good-list".

Keywords

You can create a set of keywords which enable the downloading.
(For example the signals like "attn"  or "ABPEAP")
But the keywords must be entered manually in the MYNEWS.INI-File.
This here is just the number of keywords you use (the setup is not complete).

Replyflag

If someone replies to a message (a follow-up), then not just the letters "Re:" are added to the Subject-line. But there is a "reference" header added to the message - which points to the original message.
If you want to download all this replies, then set this value to the maximum number of lines a reply may have to be downloaded automatically.
Example:    replyflag = 100
All replies up to 100 lines will be downloaded automatically (this will NOT include longer binaries in replies).
A nice side-effect: Spammers are using "Re:" in their spam - but they dont really reply - the "References"-Header is missing. So their spam is not downloaded. And those neticens who use "antique" news-readers which dont use the "references"-header are ignored, too.

max_lines

Messages which are longer than this value (in lines) are NOT downloaded automatically.
 

Restrictions for downloading messages

max_kBytes

This is a restriction during downloads of complete messages, NOT for the headers:
If the amount of max_kBytes is reached, then the download of messages STOPS. The rest is downloaded "the next time".

max_messages

This is also a restriction during download of messages.
If the previous filters would have allowed 10.000 messages for download, but this value is 100, then just 100 messages are downloaded. Then the download stops. All other messages STAY in the download-queue - until the next download is started.

check_filenames

This is a feature for the full-version of MyNews. If it is activated, then MyNews tries to find filenames in the subject lines. If a filename (arxfx001.jpg) is found, then MyNews checks your database whether you have this file already. If you have the file already, then it is NOT downloaded automatically.

I use this feature in combination with the "OnlyGood"-Flag:
If someone (who is good) posts 1000 pictures, then I download just those which I dont have yet. :-)  

Not difficult enough ?  :->

You can define also another "!default.ini" for all Usenet-Downloads in the jobs-directory. The "!body.ini", the "!head.ini" affects the downloads and every multi#.ini-file can have its own settings. You can define the same server three times with different newsgroups. The number of combinations is incredible.
And you can also create scheduled jobs for downloading - each one with a different setup.
So MyNews can serve also the most complicated and complex setups!  

Too difficult ?

Forget this chapter!
Download the standard - that's enough for 90% of you.
If your wishes grow, then re-read this chapter or ask in the group  mynews.server.questions  for help.
 

More restrictions and features ?

You want restrictions on a "by group" base? (see the option menu in the groups window)
Or you want to lockout specific servers? (see the kill-list in the security menue)
The Kill-List for Users & Servers should be respected? (see the text filter pages)
A Kill-List on keywords would be nice? (also text filters)
Formulas for "Download" or "Reject" ? (not really.... this is even too difficult for me)

Oh, I could implement TONS of further things (and some will be added in the next time). But actually this is really enough.


The next step will be about auto-posting of binaries.
Perhaps more interesting for you :-)


Juergen, <InfStar>
All rights reserved. Document contains mistakes in spelling, grammar and terms.