 |
Index for
Section 5 |
|
 |
Alphabetical
listing for I |
|
 |
Bottom of
page |
|
innfeed.conf(5)
NAME
innfeed.conf - configuration file for innfeed
DESCRIPTION
This man page describes the configuration file for version 1.0 of innfeed.
This format has changed dramatically since version 0.9.3.
The file innfeed.conf is used to control the innfeed(1) program. It is a
fairly free-format file that consists of three types of entries: key/value,
peer and group. Comments are from the hash character ``#'' to the end of
the line.
Key/value entries are a keyword and a value separated by a colon (which can
itself be surrounded by whitespace). For example:
max-connections: 10
A legal key starts with a letter and contains only letters, digits, and
``_'', ``-''.
There are 5 different type of values: integers, floating-point numbers,
characters, booleans, and strings. Integer and floating point numbers are
as to be expected except that exponents in floating point numbers are not
supported. A boolean value is either ``true'' or ``false'' (case is not
significant). A character value is a single-quoted character as defined by
the C-language. A string value is any other sequence of characters. If the
string needs to contain whitespace, then it must be quoted with double
quotes, and uses the same format for embedding non-printing characters as
normal C-language string.
Peer entries look like:
peer <name> {
# body ...
}
The word ``peer'' is required. The ``<name>'' is the same as the site name
in INN's newsfeeds file. The body of a peer entry contains some number
(possibly zero) of key/value entries.
Group entries look like:
group <name> {
# body
}
The word ``group'' is required. The ``<name>'' is any string valid as a
key. The body of a group entry contains any number of the three types of
entries. So key/value pairs can be defined inside a group, and peers can be
nested inside a group, and other groups can be nested inside a group.
Key/value entries that are defined outside of all peer and group entries
are said to be at ``global scope''. There are global key/value entries that
apply to the process as a whole (for example the location of the backlog
file directory), and there are global key/value entries that act as
defaults for peers. When innfeed looks for a specific value in a peer entry
(for example, the maximum number of connections to set up), if the value is
not defined in the peer entry, then the enclosing groups are examined for
the entry (starting at the closest enclosing group). If there are no
enclosing groups, or the enclosing groups don't define the key/value, then
the value at global scope is used.
A small example could be:
# Global value applied to all peers that have
# no value of their own.
max-connections: 5
# A peer definition. ``uunet'' is the name used by innd in
# the newsfeeds file.
peer uunet {
ip-name: usenet1.uu.net
}
peer vixie {
ip-name: gw.home.vix.com
max-connections: 10 # override global value.
}
# A group of two peers who can handle more connections
# than normal
group fast-sites {
max-connections: 15
# Another peer. The ``max-connections'' value from the
# ``fast-sites'' group scope is used. The ``ip-name'' value
# defaults to the peer's name.
peer data.ramona.vix.com {
}
peer bb.home.vix.com {
max-connections: 20 # he can really cook.
}
}
Given the above configuration file, the defined peers would have the
following values for the ``max-connections'' key.
uunet 5
vixie 10
data.ramona.vix.com 15
bb.home.vix.com 20
Innfeed ignores key/value pairs it is not interested in. Some config file
values can be set via a command line option, in which case that setting
overrides the settings in the file.
Config files can be included in other config files via the syntax:
$INCLUDE filename
There is a maximum nesting depth of 10.
For a fuller example config file, see the supplied innfeed.conf.
GLOBAL VALUES
The following listing show all the keys that apply to the process as whole.
These are not required (compiled-in defaults are used where needed).
news-spool
This key requires a pathname value. It specifies where the top of the
article spool is. This corresponds to the ``-a'' command-line option.
input-file
This key requires a pathname value. It specifies the pathname
(relative to the backlog-directory) that should be read in funnel-file
mode. This corresponds to giving a filename as an argument on the
command-line (i.e. its presence also implies that funnel-file mode
should be used).
pid-file
This key requires a pathname value. It specifies the pathname
(relative to the backlog-directory) where the pid of the innfeed
process should be stored. This corresponds to the ``-p'' command-line
option.
debug-level
This key defines the debug level for the process. A non-zero number
generates a lot of messages to stderr, or to the config-defined
``log-file''. This corresponds to the ``-d'' command-line option.
use-mmap
This key requires a boolean value. It specifies whether mmaping should
be used if innfeed has been built with mmap support. If article data
on disk is not in NNTP-ready format (CR/LF at the end of each line),
then after mmaping the article is read into memory and fixed up, so
mmaping has no positive effect (and possibly some negative effect
depending on your system), and so in such a case this value should be
false. This corresponds to the ``-M'' command-line option.
log-file
This key requires a pathname value. It specifies where any logging
messages that couldn't be sent via syslog(3) should go (such as those
generated when a positive value for ``debug-value'', is used). This
corresponds to the ``-l'' command-line option. A relative pathname is
relative to the ``backlog-directory'' value.
backlog-directory
This key requires a pathname value. It specifies where the current
innfeed process should store backlog files. This corresponds to the
``-b'' command-line option.
backlog-highwater
This key requires a positive integer value. It specifies how many
articles should be kept on the backlog file queue before starting to
write new entries to disk.
backlog-ckpt-period
This key requires a positive integer value. It specifies how many
seconds between checkpoints of the input backlog file. Too small a
number will mean frequent disk accesses, too large a number will mean
after a crash innfeed will re-offer more already-processed articles
than necessary.
backlog-newfile-period
This key requires a positive integer value. It specifies how many
seconds before each checks for externally generated backlog files that
are to be picked up and processed.
dns-retry
This key requires a positive integer value. It defines the number of
seconds between attempts to re-lookup host information that previous
failed to be resolved.
dns-expire
This key requires a positive integer value. It defines the number of
seconds between refreshes of name to address DNS translation. This is
so long-running processes don't get stuck with stale data, should peer
ip addresses change.
close-period
This key requires a positive integer value. It is the maximum number
of seconds a connection should be kept open. Some NNTP servers don't
deal well with connections being held open for long periods.
gen-html
This key requires a boolean value. It specifies whether the status-
file should be HTML-ified.
status-file
This key requires a pathname value. It specifies the pathname
(relative to the backlog-directory) where the periodic status of the
innfeed process should be stored. This corresponds to the ``-S''
command-line option.
connection-stats
This key requires a boolean value. If the value is true, then whenever
the transmission statistics for a peer are logged, then each active
connection logs its own statistics. This corresponds to the ``-z''
command-line option.
host-queue-highwater
This key requires a positive integer value. It defines how many
articles will be held internally for a peer before new arrivals cause
article information to be spooled to the backlog file.
stats-period
This key requires a positive integer value. It defines how many
seconds innfeed waits between generating statistics on transfer rates.
stats-reset
This key requires a positive integer value. It defines how many
seconds innfeed waits before resetting all internal transfer counters
back to zero (after logging one final time). This is so a innfeed-
process running more than a day will generate ``final'' stats that
will be picked up by logfile processing scripts.
initial-reconnect-time
This key requires a positive integer value. It defines how many
seconds to first wait before retrying to reconnect after a connection
failure. If the next attempt fails too, then the reconnect time is
approximately doubled until the connection succeeds, or max-
reconnection-time is reached.
max-reconnect-time
This key requires an integer value. It defines the maximum number of
seconds to wait between attempt to reconnect to a peer. The initial
value for reconnection attempts is defined by initial-reconnect-time,
and it is doubled after each failure, up to this value.
stdio-fdmax
This key requires a non-negative integer value. If the value is
greater than zero, then whenever a network socket file descriptor is
created and it has a value less than this, the file descriptor will be
dup'ed to bring the value up greater than this. This is to leave lower
numbered file descriptors free for stdio. Certain systems, Sun's in
particular, require this. SunOS 4.1.x usually requires a value of 128
and Solaris requires a value of 256. The default if this is not
specified, is 0.
bindaddress
This key requires a string value. It specifies which outgoing IPv4
address innfeed should bind the local end of its connections to. Must
be in dotted-quad format (nnn.nnn.nnn.nnn). If not set, innfeed
defaults to letting the kernel choose this address. The default value
is unset.
bindaddress6
This key requires a string value. It behave like bindaddress except
for outgoing IPv6 connections.
GLOBAL PEER DEFAULTS
All the key/value pairs mentioned in this section must be specified at
global scope. They may also be specified inside a group or peer definition.
Note that when peers are added dynamically (i.e. when innfeed receives an
article for an unspecified peer), it will add the peer site using the
parameters specified at global scope.
article-timeout
This key requires a non-negative integer value. If no articles need to
be sent to the peer for this many seconds, then the peer is considered
idle and all its active connections are torn down.
response-timeout
This key requires a non-negative integer value. It defines the maximum
amount of time to wait for a response from the peer after issuing a
command.
initial-connections
This key requires a non-negative integer value. It defines the number
of connections to be opened immediately when setting up a peer
binding. A value of 0 means no connections will be created until an
article needs to be sent.
max-connections
This key requires positive integer value. It defines the maximum
number of connections to run in parallel to the peer. A value of zero
specifies an unlimited number of maximum connections. In general use
of an unlimited number of maximum connections is not recommended. Do
not ever set max-connections to zero with dynamic-method 0 set, as
this will saturate peer hosts with connections. [ Note that in
previous versions of innfeed, a value of 1 had a special meaning. This
is no longer the case, 1 means a maximum of 1 connection ].
dynamic-method
This key requires an integer value between 0 and 3. It controls how
connections (up to max-connections) are opened, up to the maximum
specified by max-connections. In general (and specifically, with
dynamic-method 0), a new connection is opened when the current number
of connections is below max-connections, and an article is to be sent
while no current connections are idle. Without further restraint (i.e.
using dynamic-method 0), in practice this means that max-connections
connections are established while articles are being sent. Use of
other dynamic-method settings imposes a further limit on the amount of
connections opened below that specified by max-connections. This limit
is calculated in different ways, depending of the value of dynamic-
method. Users should note that adding additional connections is not
always productive - just because opening twice as many connections
results in a small percentage increase of articles accepted by the
remote peer, this may be at considerable resource cost both locally
and at the remote site, whereas the remote site might well have
received the extra articles sent from another peer a fraction of a
second later. Opening large numbers of connections is considered
antisocial. The meanings of the various settings are:
0 no method
Increase of connections up to max-connections is unrestrained.
1 maximize articles per second
Connections are increased (up to max-connections) and decreased so
as to maximize the number of articles per second sent, while using
the fewest connections to do this.
2 set target queue length
Connections are increased (up to max-connections) and decreased so
as to keep the queue of articles to be sent within the bounds set by
dynamic-backlog-low and dynamic-backlog-high, while using the
minimum resources possible. As the queue will tend to fill if the
site is not keeping up, this method ensures that the maximum number
of articles are offered to the peer while using the minimum number
of connections to achieve this.
3 combination
This method uses a combination of methods 1 and 2 above. For sites
accepting a large percentage of articles, method 2 will be used to
ensure these sites are offered as complete a feed as possible. For
sites accepting a small percentage of articles, method 1 is used, to
minimize remote resource usage. For intermediate sites, an
appropriate combination is used.
dynamic-backlog-low
This key requires an integer value between 0 and 100. It represents
(as a percentage) the low water mark for the host queue. If the host
queue falls below this level while using dynamic-method 2 or 3, and if
2 or more connections are open, innfeed will attempt to drop
connections to the host. An IIR filter is applied to the value to
prevent connection flap (see dynamic-filter). A value of 25.0 is
recommended. This value must be smaller than dynamic-backlog-high.
dynamic-backlog-high
This key requries an integer value between 0 and 100. It represents
(as a percentage) the high water mark for the host queue. If the host
queue rises above this level while using dynamic-method 2 or 3, and if
less than max-connections are open to the host, innfeed will attempt
to open further connections to the host. An IIR filter is applied to
the value to prevent connection flap (see dynamic-filter). A value of
50.0 is recommended. This value must be larger than dynamic-backlog-
low.
dynamic-backlog-filter
This key requires a floating-point value between 0 and 1. It
represents the filter coefficient used by the IIR filter used to
implement dynamic-method 2 and 3. The recommended value of this
filter is 0.7, giving a time constant of 1/(1-0.7) articles. Higher
values will result in slower response to queue fullness changes, lower
values in faster response.
max-queue-size
This key requires a positive integer value. It defines the maximum
number of articles to process at one time when using streaming to
transmit to a peer. Larger numbers mean more memory consumed as
articles usually get pulled into memory (see the description of use-
mmap).
streaming
This key requires a boolean value. It defines whether streaming
commands are used to transmit articles to the peers.
no-check-high
This key requires a floating-point number which must be in the range
[0.0, 100.0]. When running transmitting with the streaming commands,
innfeed attempts an optimization called ``no-CHECK'' mode. This
involves not asking the peer if it wants the article, but just sending
it. This optimization occurs when the percentage of the articles the
peer has accepted gets larger than this number. If this value is set
to 100.0, then this effectively turns off no-CHECK mode, as the
percentage can never get above 100.0. If this value is too small, then
the number of articles the peer rejects will get bigger (and your
bandwidth will be wasted). A value of 95.0 usually works pretty well.
NOTE: In innfeed 0.9.3 and earlier this value was in the range [0.0,
9.0].
no-check-low:
This key requires a floating-point number which must be in the range
[0.0, 100.0), and it must be smaller that the value for no-check-high.
When running in no-CHECK mode, as described above, if the percentage
of articles the remote accepts drops below this number, then the no-
CHECK optimization is turned off until the percentage gets above the
no-check-high value again. If there is small difference between this
and the no-check-high value (less than about 5.0), then innfeed may
frequently go in and out of no-CHECK mode. If the difference is too
big, then it will make it harder to get out of no-CHECK mode when
necessary (wasting bandwidth). Keeping this to between 5.0 and 10.0
less than no-check-high usually works pretty well.
no-check-filter
This is a floating point value representing the time constant, in
articles, over which the CHECK / no-CHECK calculations are done. The
recommended value is 50.0 which will implement an IIR filter of time
constant 50. This roughly equates to making a decision about the mode
over the previous 50 articles. A higher number will result in a slower
response to changing percentages of articles accepted; a lower number
will result in a faster response.
port-number
This key requires a positive integer value. It defines the tcp/ip port
number to use when connecting to the remote.
drop-deferred
This key requires a boolean value. By default it is set to false. When
set to true, and a peer replies with code 431 or 436 (try again later)
just drop the article and don't try to re-send it. This is useful for
some peers that keep on deferring articles for a long time to prevent
innfeed from trying to offer the same article over and over again.
min-queue-connection
This key requires a boolean value. By default it is set to false. When
set to true, innfeed will attempt to use a connection with the least
queue size (or the first empty connection). If this key is set to
true, it is recommended that dynamic-method be set to 0. This allows
for article propagation with the least delay.
no-backlog
This key requires a boolean value. It specifies whether spooling
should be enabled (false, the default) or disabled (true). Note that
when no-backlog is set, articles reported as "spooled" are actually
silently discarded.
backlog-limit
This key requires a non-negative integer value. If the number is 0
then backlog files are allowed to grown without bound when the peer is
unable to keep up with the article flow. If this number is greater
than 0 then it specifies the size (in bytes) the backlog file should
get truncated to when the backlog file reaches a certain limit. The
limit depends on whether backlog-factor or backlog-limit-high is used.
backlog-factor
This key requires a floating point value, which must be larger than
1.0. It is used in conjunction with the peer key backlog-limit. If
backlog-limit has a value greater than zero, then when the backlog
file gets larger than the value backlog-limit * backlog-factor, then
the backlog file will be truncated to the size backlog-limit. For
example if backlog-limit has a value of 1000000, and backlog-factor
has a value of 2.0, then when the backlogfile gets to be larger than
2000000 bytes in size, it will be truncated to 1000000 bytes. The
front portion of the file is removed, and the trimming happens on line
boundaries, so the final size may be a bit less than this number. If
backlog-limit-highwater is defined too, then backlog-factor takes
precedence.
backlog-limit-highwater
This key requires a positive integer value that must be larger than
the value for backlog-limit. If the size of the backlog file gets
larger than this value (in bytes), then the backlog file will be
shrunk down to the size of backlog-limit. If both backlog-factor and
backlog-limit-highwater are defined, then the value of backlog-factor
is used.
backlog-feed-first
This key requires a boolean value. By default it is set to false. When
set to true, the backlog is fed before new files. This is intended to
enforce in-order delivery, so setting this to true when initial-
connections or max-connections is more than 1 is inconsistent.
username
This key requires a string value. If the value is defined, then
innfeed tries to authenticate by ``AUTHINFO USER'' and this value used
for user name. password must also be defined, if this key is defined.
password
This key requires a string value. The value is the password used for
``AUTHINFO PASS''. username must also be defined, if this key is
defined.
deliver
This key is used with imapfeed to authenticate to a remote host. It
is optional. There are several parameters that must be included with
deliver:
deliver-authname
The authname is who you want to authenticate as.
deliver-password
This is the appropriate password for authname.
deliver-username
The username is who you want to "act" as, that is, who is actually
going to be using the server.
deliver-realm
In this case, the "realm" is the realm in which the specified
authname is valid. Currently this is only needed by the DIGEST-MD5
SASL mechanism.
deliver-rcpt-to
A printf-style format string for creating the envelope recipient
address. The pattern MUST include a single string specifier which
will be replaced with the newgroup (e.g "bb+%s"). The default is
"+%s".
deliver-to-header
An optional printf-style format string for creating a To: header to
be prepended to the article. The pattern MUST include a single
string specifier which will be replaced with the newgroup (e.g
"post+%s@domain"). If not specified, the To: header will not be
prepended.
PEER VALUES
As previously explained, the peer definitions can contain redefinitions of
any of the key/value pairs described in the GLOBAL PEER DEFAULTS section
above. There is one key/value pair that is specific to a peer definition.
ip-name
This key requires a word value. The word is the host's FQDN, or the
dotted quad ip-address. If this value is not specified then the name
of the peer is taken to also be its ip-name. See the entry for
data.ramona.vix.com in the example below.
RELOADING
If innfeed gets a SIGHUP signal, then it will reread the config file. All
values at global scope except for ``backlog-directory'' can be changed. Any
new peers are added and any missing peers have their connections closed.
EXAMPLE
Below is the sample innfeed.conf file.
#
# innfeed.conf file. See the comment block at the
# end for a fuller description.
#
##
## Global values. Not specific to any peer. These
## are optional, but if used will override the
## compiled in values. Command-line options used
## will override these values.
##
pid-file: innfeed.pid
debug-level: 0
use-mmap: false
log-file: innfeed.log
stdio-fdmax: 0
backlog-directory: innfeed
backlog-rotate-period: 60
backlog-ckpt-period: 30
backlog-newfile-period: 600
dns-retry: 900
dns-expire: 86400
close-period: 3600
gen-html: false
status-file: innfeed.status
connection-stats: false
host-queue-highwater: 200
stats-period: 600
stats-reset: 43200
max-reconnect-time: 3600
initial-reconnect-time: 30
##
## Defaults for all peers. These must all exist at
## global scope. Any of them can be redefined
## inside a peer or group definition.
##
article-timeout: 600
response-timeout: 300
initial-connections: 1
max-connections: 5
max-queue-size: 25
streaming: true
no-check-high: 95.0
no-check-low: 90.0
no-check-filter: 50.0
port-number: 119
backlog-limit: 0
backlog-factor: 1.10
backlog-limit-highwater:0
dynamic-method: 3
dynamic-backlog-filter: 0.7
dynamic-backlog-low: 25.0
dynamic-backlog-high: 50.0
no-backlog: false
##
## Peers.
##
peer decwrl {
ip-name: news1.pa.dec.com
}
peer uunet {
ip-name: news.uunet.uu.net
max-connections: 10
}
peer data.ramona.vix.com {
# ip-name defaults to data.ramona.vix.com
streaming: false
}
peer bb.home.vix.com {
ip-name: 192.5.5.33
}
# Blank lines are ignored. Everything after a '#'
# is ignored too.
#
# Format is:
# key : value
#
# See innfeed.conf(5) for a description of
# necessary & useful keys. Unknown keys and their
# values are ignored.
#
# Values may be a integer, floating-point, c-style
# single-quoted characters, boolean, and strings.
#
# If a string value contains whitespace, or
# embedded quotes, or the comment character
# (``#''), then the whole string must be quoted
# with double quotes. Inside the quotes, you may
# use the standard c-escape sequence
# (\t,\n,\r,\f,\v,\",\').
#
# Examples:
# eg-string: "NewConfigfile0
# eg-long-string: "A long string that goes
# over multiple lines. The
# newline is kept in the
# string except when quoted
# with a backslash # as here."
# eg-simple-string: A-no-quote-string
# eg-integer: 10
# eg-boolean: true
# eg-char: 'a'
# eg-ctrl-g: ' 07'
HISTORY
Written by James Brister <brister@vix.com> for InterNetNews. This is
revision 1.16, dated 2003/01/13.
SEE ALSO
innfeed(1), newsfeeds(5)
|
Index for
Section 5 |
|
|
Alphabetical
listing for I |
|
 |
Top of
page |
|