March 5, 2002 Telcordia Technologies

Over the last few years, the Internet has grown from a research tool to a household

commodity. Our understanding of its behavior has not grown at the same pace. The

research community is still trying to understand, predict and control the behavior of the

Internet. Good datasets are an essential ingredient of this process. Good datasets can be

used to validate existing models, and to suggest new ones. Experience has shown that

good datasets must contain fine detail, down to the packet level if possible. Summaries

can always be produced from detail, but the converse it not true.

The IPEX project, funded by DARPA/NMS, aims to provide an open measurement

infrastructure that can gather coordinated high quality datasets from multiple

measurement sites. These datasets will be collected at the packet level, and will

ultimately be made available to the network modeling and simulation community. This

raises serious security and privacy concerns. Specifically, by the time the data become

public (even in the NMS community), they must not contain anything private or

proprietary. Furthermore, the probe machines themselves must be protected from

compromise.

The first IPEX probes will be configured by Telcordia Technologies in accordance with

the above requirements, and shipped to participating sites for deployment. In this

document, we outline configuration requirements for these sites. We discuss our security

policy, and the steps we have taken to implement it.

Each IPEX probe will need a source of traffic to be monitored. This could be from a

switch monitor port or a hub. A switch monitor port has a number of advantages. It can

be used when the link to be monitored is not compatible with the monitor interface (e.g. a

T-1 or fractional T-3), and it cannot transmit any traffic by accident. On the other hand,

there may not be a spare port available, or the switch may not be able to handle the extra

workload. In this case, care must be taken that the hub directs all traffic to the monitor

link, and does not function as a switch or bridge.

Initially, IPEX probes will be installed at five sites (not counting Telcordia

Technologies). Each probe will consist of:

• One Sparc Netra T1-100 or T1-200. This machine has two 10/100baseT network

ports, two serial ports, and one (T1-100) or two (T1-200) 9 Gb internal SCSI

disks. There is also an empty PCI slot. For historical reasons, the T1-100’s are

DC-powered. Power supplies will be provided. The T1-200’s are AC powered.

Future probes will also be AC powered.

• One 72 Gb external SCSI disk

• One Cyclades console server for secure access to the Netra console. The server

has a network connection, a console port, and four ports for serial lines.

One network port on the Netra is used to administer the machine, and to download data

from it. This port needs an externally visible IP address, to be provided by the

organization hosting the probe. The other port handles the traffic that is to be monitored.

One of the serial ports on the Netra is used for the system console. This port is connected

to one of the serial ports on the Cyclades console server. A second serial port on the

console server is used to access the Cyclades console. The Cyclades console server will

need at least three externally visible IP addresses, one for maintenance and general

Internet access, and two for the connections to the Sparc and Cyclades consoles.

To minimize the administrative burden of operating the probes, and to coordinate the

simultaneous activity of multiple probes, we will set up two central machines: an

administration machine, and a data server. The administration machine will need

network access to the probe and to the console server. The data server will need network

access to the probe only. The administration machine is already (3/1/2002) in place. The

data server is still under construction. In the interim, the administration machine will

double as the data server.

Placing the Internet connections to the probe and console server behind the host site’s

firewall is more secure, and in our opinion preferable. The disadvantage is that it

requires active co-operation from the firewall administrator to open a path through the

firewall. (Paths are needed from the administration and data server machines to the SSH

and NIMI ports on the probe machine and the SSH port on the console server.)

Configuring access through a firewall can be tricky, and may not be acceptable to the

host’s security administration.

One of the important features of IPEX is its ability to provide coordinated data collection

at multiple sites. Such coordination requires accurate timestamps and synchronized

clocks. We would like to do this with NTP, using a local clock if possible. This will

have to be negotiated on a per-site basis.

to the Internet will almost certainly be compromised in a matter of days. A compromised

system can have far-reaching consequences, including theft of intellectual property and

loss of reputation. The consequences of a compromised probe are much greater, because

the probes will have a network interface running in promiscuous mode, exposing all the

traffic going by for anyone to see. The raw data may contain sensitive information, such

as trade secrets, passwords and personal information. The interface itself may be a

gateway to more information or a platform to mount an attack inside the host’s corporate

firewall.

For all these reasons, we take security seriously. In this section, we outline the steps that

we are taking to keep an attacker from getting access to the raw data, and to the interface

itself. In the next section, we outline the steps that we will take to remove sensitive

information from our copy of this data. We believe that these steps are consistent with

current practice in the field

The probes have been designed on the assumption that administrative network

connections to the probe machine and the console server will not be protected by a

firewall. To prevent unauthorized intrusion, we have closed all unnecessary Internet

ports. Ports that must be open use authentication, backed by strong encryption. All data

sent to the probe is encrypted, to prevent discovery of clear-text passwords. Specifically:

• The only open Internet ports on the Netra are for Syslog, SSH and NIMI (and

perhaps NTP, as discussed below).

• The only open Internet port on the console server is for SSH.

• All other access (e.g. rlogin) is disabled.

SSH and NIMI use strong encryption for authentication. SSH also uses strong encryption

for transmission.

Securing NTP is a little harder. The NTP daemon binds to an Internet port even when it

is running in client mode. It is therefore subject to attack. A buffer overflow attack was

recently found for this daemon. Strong encryption can be used to provide some access

control. Our feeling at the moment is that we will run NTP on any machines that are

protected by firewalls, where the NTP port can be blocked from the outside world.

The host organization will be given SSH access to the probe and the console server. This

will enable them to help with administration, and will also short-circuit access to their

data. All other access will be via the data server.

The central administration machine and the data server both sit in Telcordia Techologies’

"DMZ". This means they are protected by a firewall, but can also be accessed by outside

parties. Access to the central administration machine is restricted on a need-to-know

basis. This machine uses host-based authentication in order to administer the probes

without human intervention. Accounts on the data server will be available to all IPEX

members. This machine will also run a Web server to make the (anonymized) data more

generally available.

System administrators sometimes require that the system logs of important machines be

copied to the syslog port on a special logging machines. The Netra and the console

server are both capable of such remote logging. We will provide it for any site that

requests it. Otherwise, the syslog stream for the console server will be directed to the

Netra. This requires that the syslog port on the Netra be open. We are not aware of any

security implications of this practice (except perhaps an increased vulnerability to denial-of-

service attacks).

Details of our security process will be shared with any sites wishing to build their own

probes. At the very least we suggest the use of security auditing tools, such as Nmap, to

check probes before they are deployed.

A vulnerability found recently in the SSH server allows anyone to gain root access on

any machine running older versions of this server. The vulnerability is described in

version number that matches vulnerable versions of SSH. Cyclades technical support has

assured us that they have extensively modified their version of SSH, and that it is not

vulnerable to this attack. Nevertheless, we have taken the following steps to ensure that

our servers cannot be compromised:

1) The vulnerability is in version 1 of the protocol. The console server has been

configured not to accept version 1 connections. According to the CERT advisory,

this removes the vulnerability. (The probe is running a more recent version of

SSH that is not vulnerable.)

2) The console server runs Linux with a V2.2 kernel. We have turned on ipchains

filtering for the SSH port so connections may only be made from a small set of

trusted addresses. This also stops public advertising of the server version number.

3) The console server is Power PC based, not Intel based. As of March 1 2002, there

were no known exploits for the Power PC architecture.

We understand that the organizations hosting the probes have an obligation to safeguard

the confidentiality of the data flowing on their networks, and the privacy of their

employees and clients. We alter the data that we collect so that these obligations are

satisfied. The alterations are made by the program collecting the data from the network

interface, so that there is never any unaltered data sitting on disk. The only other route to

disk is via the creation of a "core" file. This has been disallowed.

Two kinds of alteration are necessary in order to preserve privacy: deletion of any data

that are not relevant to traffic analysis, and anonymization of any data that are kept. We

keep only packet headers. This rules out certain kinds of analysis of the data (e.g. HTTP

URL’s) but on balance we feel that this is the best way to protect privacy. In the case of

TCP and UDP packets, we keep all data up to and including the TCP or UDP header. All

options in the header are anonymized. When a packet of unknown type is encountered

(currently this is anything not an IP packet), we save the contents of the header up to the

field with an unrecognized value, and discard the rest. This approach gives us enough

information to determine the type of the packet that caused the problem, and (if we so

desire) augment the processing code to deal with it. At the same time, it ensures that we

will not inadvertently save something we should not.

We use a modified version of the tcpdpriv program to collect data from the network

interface. Tcpdpriv is a modified version of tcpdump that can obfuscate the source and

destination IP addresses of traffic that it captures. Obfuscation is done as soon as a

packet is received, so that addresses sit around in the clear for as short a time as possible.

The level of obfuscation will be selectable by the host organization, consistent with their

security and privacy needs.

We have set up two mailing lists. The first list is for general discussion, and has an open

subscription policy. The address of the list is

Anyone wishing to subscribe to the list should send e-mail to

ipex-request@research.telcordia.com

with the word "subscribe" as the subject.

The second list is intended for discussion of probe administration and security issues.

The address of this list is

ipex-admin@research.telcordia.com

This list is closed, i.e. additions to the list are not automatic. Anyone wishing to be

placed on the list should send e-mail to

ipex-admin-request@research.tecordia.com

with the word "subscribe" as the subject.