NetDMF Overview

From NetDMF
Jump to: navigation, search

DEPRECATED

The NetDMF project is no longer under active development. Improvements that motivated this project have been implemented directly in XDMF.

This site is scheduled for take-down as of 1/1/2019. Contact kitware@kitware.com if you have a need for this site beyond then. DEPRECATED


NetDMF.jpg

An Extensible Discrete-Event Mobile Networking Data Model and Format

  • Objective

Common Data Models and Formats have proven to be extremely valuable in the integration of codes and tools into useful environment for simulation and analysis in many areas of computational science. Mobile network modeling (MNM) has recently become a major focus area in the DoD and at ARL in particular. Proposed here is the development of a data model and format particularly designed for the setup of MNM simulations and the analysis of MNM data. This is to be an XML based standard that borrows some of the lessons learned from previous efforts and will start under the working name of NetDMF.

  • The Need

Currently, a flexible software environment to setup and run simulations as well as analyze data from experiments and emulations does not exist. The foundation that will underlay the entire system is a common data model and format that should not be targeted at the particular interests of any one entity. Rather, the design and ongoing support of the standard and the software tools that go along with it, should be open and available to the community.

NetDMF is to be both a format capable of representing network topology and scenarios for network modeling simulations and a format capable of organizing the results. Necessary details to extend the format into networking experiments and emulations will aid in its adoption by a wide range of networking research projects. As we continue move towards the digitization of the battlefield, information is the key element in success of military operations. Allowing as many research projects as possible to seamlessly share results will accelerate the accomplishment of a network centric force.

  • Background

Discrete-Event Network Simulations are accomplished by various commercial and open source network simulators. Commercial simulators of interest include OPNET and Qualnet. Open source efforts (some with restrictions) include such simulators as NS-2, NS-3, GTNetS, OMNET++, YANS, and SSFNet. These simulators require the user to define the network topology and communication devices being used. Next the user defines which networking protocols are to be used and discrete events that will cause network data to flow (i.e. an FTP transfer from one node to another). The simulation is run resulting in packet information, logs and other statistics to be saved to a file or files. If the network is mobile, position information is also saved.

Experiments like those conducted at C4ISR-OTM use precision GPS, timing and packet capture to produce similar data from live exercises. Emulations mix virtual simulations and physical devices but again result in similar data being produced.

The output data is enormous; regularly saving every packet of data produced by every node. For analysis, information about the packets (protocols, length, etc.) tends to be of more interest that the actual payload data, but this information is saved in binary format intermixed with payload data itself. Further complicating the situation, other logging data and data about node movement tend to be produced in simulator specific, human readable formats which are not compatible with existing general purpose visualization and analysis tools.

Setting up a simulation can be problematic as well. Some of these simulators are actually libraries of functionality. Setting up a simulation requires the user to write a program or script that calls the necessary functions or methods to define and run a simulation. While some tools exist to create these programs via a graphical user interface, they are not portable across codes. If an input file capability is available, it tends to be in a proprietary format that is unlikely to be adopted by any other simulator. Finally, some graphical user interfaces, like BRITE, that can produce input files or scripts for a handful of simulators are available But using this approach will lead to every tool needing to know the input specifics for every simulator.

  • Approach

Approaching this problem we will rely heavily on experience gained from the design, development and maintenance of the eXtensible Data Model and Format (Xdmf). Xdmf is designed to store data from physics based simulations and as an abstracted yet efficient data exchange mechanism for software modules.

Xdmf has been widely used in the US and around the world. It is natively read by visualization and analysis tools like ParaView, Visit, and EnSight. The advantage of Xdmf is that it logically, and physically, separates the “Light Data” (meta-data and small quantities) from the “Heavy Data”. For the light data Xdmf uses XML. Many tools exist to produce and parse XML files in a variety of languages. For the heavy data, Xdmf currently supports HDF5 and MySQL, both widely used industry standards. In addition to defining a standard, Xdmf also provides an open source C++ class library that is used by programs to read and write Xdmf data. The C++ class library is then “wrapped” for scripting languages like Python, Tcl, and Java to allow for the rapid development of tools that use Xdmf.

NetDMF will use the same basic approach. We will define an XML based format for storing the light data. Heavy data will be stored in some reasonable format (perhaps PCAP and MySQL) and only accessed when necessary. We will develop a C++ class library for reading and writing NetDMF data and wrap that library for use from scripting languages.

OneSAF (a Semi-Automated Forces modeling project) is developing an XML based scenario description format called the “Military Scenario Definition Language” (MSDL). From the OneSAF website :

MSDL is being developed by OneSAF to provide simulations with a mechanism for loading Military Scenarios. As a standard, MSDL is not being developed for simulation alone. The intent is for MSDL to define Military Scenarios that are independent of the application of that scenario. To that end, MSDL is an XML based data interchange format that enables C2 planning applications to interchange the military portions of scenarios with Simulations and other applications.

Basing NetDMF on XML and having the C++ class library provide the necessary functionality will provide a pathway for potential integration with the MSDL format.

For input to network simulators, NetDMF will need to express:

  • Physical Terrain and Conditions
  • Physical Communications Devices
  • Networking Protocol Layers
  • Network Topologies
  • Discrete Network Events (i.e. begin File Transfer)
  • Node Movement Events (i.e. Node 0 moves to location 0.1, 0.2)

For output NetDMF will, at a minimum, need to express:

  • Node Location, Speed, etc.
  • Movement and Communication TimeStamps
  • Communication Events (i.e. Packet Communication)
  • Some Common Calculated Communication Statistics

Understanding the complexities of designing such a system will require that some rudimentary tools be developed that use NetDMF, in parallel, while NetDMF itself is being developed. For example, a converter that reads an NetDMF file and produces a simulator input file or script will help determine if all necessary information is being represented, is easily accessible and is unambiguous. We propose targeting the open source simulator NS-3 as a prototype. NS-3 is an NSF sponsored project that aspires to replace NS-2 which is the most widely used open source network simulator. The first version of NS-3 was released June 30, 2008 and will soon have a Python interface. Two initial tools will be developed: a tool to create an NS-3 simulation from NetDMF and a tool to generate NetDMF from NS-3 output files.

Qualnet is a commercial network simulator used extensively in the FCS program. After a working version of NetDMF has been integrated with NS-3, it will then be integrated with Qualnet. This will help with analysis for data from the Mobile Network Modeling Institute as well as C4ISR-OTM and FCS projects. It is not proposed to enhance physical layer calculations in Qualnet at this time.

The heavy data from a network simulation or experiment is the actual packets of data. They are typically stored in “pcap” files; a commonly used binary format for packet data. Programs like Wireshark (formerly Ethereal) can read these files, or read data directly from a network interface, display the different protocols being used and perform some analysis of the results. In addition, Wireshark can be used to translate the binary pcap files into an XML format known as “Packet Details Markup Language” (PDML). NetDMF will include portions of the PDML format in order to capture information about the protocols but will leave the payload data behind, in either pcap or MySQL format.

First Implementation

The first attempt at specifying NetDMF will extend Xdmf. Since the physical domain (terrain, foliage, etc.) will need to be specified we will use the current Xdmf specification. Additionally, the current Xdmf implementation provides access to heavy data via HDF5 and MySQL.

Xdmf alone, however, is not sufficient. For example, there needs to be the concept of devices and events that is currently not available in Xdmf. So NetDMf will be a superset of Xdmf. Initially, several XML elements will be added:

  • Addresses - Raw network addresses like MAC and IP
  • Device – either a communication “channel” or “Mobility”
  • Protocol – communications protocol
  • Communication – specifies the device being used and data characteristics
  • Stack – reference a set of protocols
  • Node – an entity in a simulation or experiment
  • Event – movement or communication


  • Core Data Types

Xdmf uses a <DataItem> to define a homogeneous arrays for various scientific number types. This will be useful for defining terrain and any physics based simulation model data like electromagnetic fields. The simplest DataItem type is Uniform that specifies a single array. As with all XDMF elements, there are reasonable defaults wherever possible. So the simplest DataItem would be :

 <DataItem Dimensions="3">
   1.0 2.0 3.0
 </DataItem>

Since no ItemType has been specified, Uniform has been assumed. The default Format is XML and the default NumberType is a 32 bit floating point value. So the fully qualified DataItem for the same data would be :

 <DataItem ItemType="Uniform">
   Format="XML"
   NumberType="Float" Precision="4"
   Rank="1" Dimensions="3">
   1.0 2.0 3.0
  </DataItem>

Since it is only practical to store a small amount of data values in the XML, production codes typically write their data to HDF5 and specify the location in XML. HDF5 is a hierarchical, self describing data format. So an application can open an HDF5 file without any prior knowledge of the data and determine the dimensions and number type of all the arrays stored in the file. XDMF requires that this information also be stored redundantly in the XML so that applications need not have access to the actual heavy data in order to determine storage requirements.

For example, suppose an application stored a three dimensional array of pressure values at each iteration into an HDF5 file. The XML might be :

 <DataItem ItemType="Uniform"
   Format="HDF"
   NumberType="Float" Precision="8"
   Dimensions="64 128 256">
   OutputData.h5:/Results/Iteration 100/Part 2/Pressure
 </DataItem>

Dimensions are specified with the slowest varying dimension first (i.e. KJI order). The HDF filename can be fully qualified, if it is not it is assumed to be located in the current directory or the same directory as the XML file. MySQL is also supported as a heavy data format. DataItems can be fairly complex. See the Xdmf Web site for full details. http://www.xdmf.org


For example, a <DataItem> is used for defining movement:

 <Event  EventType=”Movement” 
   <Movement NodeId=”/NetDMF/Scenario/Node[4]” MovementType=”Path” PathPositionType="Grid">
     <DataItem  NumberType=”Float” Dimensions=”3 4” Format=”XML”>
	0.0		35.0   75.0   1.0
	0.1		35.01 75.0  1.0
	1.0		35.15 75.0  1.0
     </DataItem>
   </Movement>
 </Event>

This specifies a path of 3 locations for node 2. The same structure would be used for setup scenarios and measurements. Other MovementTypes might include velocity or describe some random movement model for a simulation.

In addition to the Xdmf <DataItem> we introduce other items needed for networking. The first is <AddressItem> used for network addresses at various network layers.

 <AddressItem
    AddressType="IPV4"
    NumberOfAddresses"4"
    Format="XML">
    <DataItem Format="XML" NumberType="Int" Dimensions="4 4">
        10  11  102 23
        10  11  104 23
        192 168 0   1
        192 168 0   12
    </DataItem>
 </AddressItem>

Specify 4 IP version 4 addresses in dot notation.

 <AddressItem
    AddressType="ETH"
    NumberOfAddresses"3"
    Format="XML" AddressString="00:12:80:6D:6A:2A 00:13:1A:36:3F:E8 00:10:D6:00:0C:93"/>

Specify 3 MAC addresses. Notice we always specify 2 digit hex numbers padded if necessary.

 <AddressItem
    AddressType="ETH"
    NumberOfAddresses"3"
    Format="XDMF" >
     <DataItem 
        NumberType="UChar"
        Dimensions="3 6"
        Format="HDF">
        /data/MacAddresses.h5:/Zone1/Addresses
      </DataItem>
 </AddressItem>

If there are many addresses, we might store them in HDF5 or MySQL. Since Xdmf already handles this type of data, we take advantage of Xdmf and expect arrays of 6 columns (MAC), 4 columns (IPv4), 16 columns (IPv6), etc.

Initially, we expect network packet information to be stored in PCAP files but will support an Xdmf <DataItem>, MySQL and the Packet Details Markup Language (PDML) as well. For PCAP, we will leave the heavy data in the PCAP files and reference it in the light data.

 <PacketItem
    NumberOfPackets="100"
    PacketType="Range"
    Range="0 109"
    Format="PCAP">
    test.pcap
 </PacketItem>

Specifies the first 110 packets in test.pcap.

 <PacketItem
    NumberOfPackets="100"
    PacketType="Filter"
    Filter="src host 10.11.102.23 and dst host 10.11.104.23"
    Format="PCAP">
    test.pcap
 </PacketItem>

Specifies the first 100 packets in test.pcap between 2 IP addresses. The filter syntax is built into the pcap access library and allows a very flexible specification.

  • Scenario and Results

NetDMF is intended to handle simulations, emulations, and experimentations. The format must support both setup scenarios and results. So the initial high level format would be :

    <Scenario>
	XML for description of mobile device, communication devices and protocols
	XML for planned events: movement and communication
    </Scenario>
    <Result>
	XML for measured/simulated communications
	XML for measured/simulated movements
    </Result> 

A scenario consists of Platforms or Nodes. Platforms are a collection of Nodes that have the same mobility. Nodes contain mobility and communication Devices. Nodes can be connected with a communication Channel. This creates a graph structure of the wired network topology only; not the wireless devices.

First define the protocols that will be used :

    <Protocol Name="Tcp" ProtocolType="Transport" />
    <Protocol Name="Udp" ProtocolType="Transport" />
    <Protocol Name="Ipv4" ProtocolType="Network" />
    <Protocol Name="Ipv6" ProtocolType="Network" />
    <Protocol Name="Ethernet" ProtocolType="Link" />
    <Protocol Name="Token Ring" ProtocolType="Link" />
    <Protocol Name="Wifi" ProtocolType="Link" />
    <Protocol Name="Wimax" ProtocolType="Link" />
    <Protocol Name="Ppp" ProtocolType="Link" />
    <Protocol Name="FDDI" ProtocolType="Link" />
    <Protocol Name="802.11" ProtocolType="Link" />
    <Protocol Name="Olsr" ProtocolType="Routing" />

A protocol stack is implemented by referencing existing protocols. The "Template" mechanism is used to reference a definition elsewhere in the XML.

  <Stack Name="SimpleOlsr" >
      <Protocol Template="/NetDMF/Protocol[@Name='Olsr']" /> 
      <Protocol Template="/NetDMF/Protocol[@Name='Udp']" /> 
      <Protocol Template="/NetDMF/Protocol[@Name='Ipv4']" /> 
      <Protocol Template="/NetDMF/Protocol[@Name='Ppp']" />
  </Stack>

Then define physical devices. Currently, there are two types of devices: Communications and Mobility. Devices can contain other devices (for example, a mobility device can have two communication devices). Communication devices contain one or more protocol stacks. Devices have parameters that are only of importance to that particular device :

 <Device Name="RadioType1" DeviceType="Communication">
        <Parameter Name="DataRate" Value="300000" />
        <Parameter Name="Delay" Value="2.5" />
 </Device>
 <Device Name="HMMWV" DeviceType="Mobility">
        <Parameter Name="Wheels" Value="4" />
        <Parameter Name="TopSpeed" Value="20.0" />
        <Device Template="/NetDMF/Device[@Name='RadioType1']" >
              <Stack Template="/NetDMF/Stack[@Name='SimpleOlsr']" />
              <Stack Template="/NetDMF/Stack[@Name='MyExperimentalProtocol']" />
        </Device>
 </Device>

Devices can also contain Addresses:

    <Device Name="wifi0" DeviceType="Communication">
        <AddressItem AddressType="ETH" NumberOfAddresses="1" Format="XML" AddressString="f0:01:02:03:04:05"/>
        <AddressItem AddressType="IPV4" NumberOfAddresses="1" Format="XML" AddressString="1.2.3.4/>
    </Device>

The xi:include XML is used to include some predefined devices or protocols that may be common to many simulations. Here are some examples :

    <xi:include href="DefaultCommDevices.xmn" xpointer="xpointer(//NetDMF/Protocol)" />

Would include all of the protocols in the file DefaultCommDevices.xmn

    <xi:include href="DefaultCommDevices.xmn" xpointer="xpointer(//NetDMF/Protocol[@ProtocolType='Link'])" />

Would include all of the protocols in the file DefaultCommDevices.xmn that have a ProtocolType="Link" attribute.

Now entities in the simulation or experiment can be defined. You modify or add particular parameters as needed :

 <Platform Name="Humvee1">
    <Node Name="Driver">
        <Device Name="Radio" DeviceType="Communication>
            <AddressItem AddressType="IPV4" NumberOfAddresses="1" Format="XML" AddressString="1.2.3.4/>
        </Device>
    </Node>
    <Node Name="Passenger">
        <Device Name="Radio" DeviceType="Communication>
            <AddressItem AddressType="IPV4" NumberOfAddresses="1" Format="XML" AddressString="1.2.3.5/>
        </Device>
    </Node>
 </Platform>
 <Node Name="Person1">
    <Device Name="eth0" DeviceType="Communication">
        <Parameter Name="ChannelType" Value="CSMA"/>
        <Parameter Name="GID" Value="0:0"/>
        <Parameter Name="IfIndex" Value="0"/>
        <Parameter Name="MTU" Value="1500"/>
        <Parameter Name="FrameSize" Value="1514"/>
        <Parameter Name="DataRate" Value="100000000bps"/>
        <AddressItem AddressType="ETH" NumberOfAddresses="1" Format="XML" AddressString="f0:00:00:00:00:01"/>
        <AddressItem AddressType="IPV4" NumberOfAddresses="1" Format="XML" AddressString="10.1.1.1"/>
    </Device>
    <Stack Template="/NetDMF/Stack[@Name='SimpleOlsr']" />
 </Node>
 <Node Name="Person2">
    <Device Name="wifi0" DeviceType="Communication">
        <Parameter Name="ChannelType" Value="Wifi"/>
        <Parameter Name="GID" Value="1:2"/>
        <Parameter Name="IfIndex" Value="2"/>
        <Parameter Name="MTU" Value="2304"/>
        <Parameter Name="WifiMac:Type" Value="ap"/>
        <Parameter Name="WifiMac:ssid" Value="field-test-1"/>
        <AddressItem AddressType="ETH" NumberOfAddresses="1" Format="XML" AddressString="f0:00:00:00:00:06"/>
        <AddressItem AddressType="IPV4" NumberOfAddresses="1" Format="XML" AddressString="10.1.3.2/>
    </Device>
    <Stack Template="/NetDMF/Stack[@Name='SimpleOlsr']" />
 </Node>
 <Node Name="Person3">
    <Device Name="wifi0" DeviceType="Communication">
        <Parameter Name="ChannelType" Value="Wifi"/>
        <Parameter Name="GID" Value="3:0"/>
        <Parameter Name="IfIndex" Value="0"/>
        <Parameter Name="MTU" Value="2304"/>
        <Parameter Name="WifiMac:Type" Value="sta"/>
        <Parameter Name="WifiMac:ssid" Value="field-test-1"/>
        <AddressItem AddressType="ETH" NumberOfAddresses="1" Format="XML" AddressString="f0:00:00:00:00:44"/>
        <AddressItem AddressType="IPV4" NumberOfAddresses="1" Format="XML" AddressString="10.1.3.7"/>
    </Device>
    <Stack Reference="/NetDMF/Stack[@Name='SimpleOlsr']" />
    </Stack>
 </Node>

In a wired or wireless network, a channel specifies physical or logical connectivity to a particular network segment, such as a shared Ethernet or 802.11 network. The NetDMFChannel element contains a list of network devices that are considered to be 'connected' (either physically or logically). The network devices are specified by an XML reference to the definition within a NetDMFNode.

 <Device Name="Cat6" DeviceType="Communication">
    <Parameter Name="MaxRate" Value="100E06" />
 </Device>
 <Channel Name="Cables" ChannelType="Duplex">
    <Device Reference="/NetDMF/Device[@Name='Cat6']" />
      /NetDMF/Scenario/Node[3]/Device
      /NetDMF/Scenario/Node[4]/Device[2]
      /NetDMF/Scenario/Node[@Name='Server']/Device[1]
 </Channel>

This places 3 entities at different locations with different IP addresses using the same communication.


To specify planned or measured events some new data entities need to be defied.

The <Traffic> element specifies the data that flows in a conversation. It uses an XdmfDataItem to specify how many bytes/frames were transferred, in each direction or lists the raw packets using <PacketItem>. If the TrafficType is bin, an Interval is specified.

An <Event> is the high level container for discrete events. In addition to StartTime and EndTime (from the Unix epoch) an <Event> has an EventType attribute. EventType corresponds to the child element type of the <Event>. Valid EventTypes are :

  • Movement - Node Movement
  • Communication - Communication between two endpoints
  • Collection - A group of events

The Communication EventType is further broken down into the types:

  • EndPoint - used to indicate that devices have communicated, but no characteristics of the traffic
  • Conversation - describes the traffic between 2 devices

Putting this all together we can begin to see how to describe the result of a simulation or experiment with mobile nodes of radios specified at every second.

 <Result>
 <Event EventType=”Collection” CollectionType=”Temporal” Name="Simple OLSR Simulation" StartTime="0.0" Endtime="50.0">
 <Event EventType=”Collection” CollectionType=”Temporal”  StartTime="0.0" Endtime="1.0">
 <Event  EventType=”Movement” 
	NodeId=”2”
        <Movement 
            MovementType=”Path”
            PathPositionType=”GeoSpatial”
            Interval="1.0"
            PathLength=”2”>
        <DataItem  NumberType=”Float” Dimensions=”3 4” Format=”XML”>
	    35.0   75.0  1.0
	    35.15 75.0  1.0
        </DataItem>
        </Movement>
 </Event>
 <Event  EventType=”Comm” CommType=”EndPoints” EndPointType=”IPV4”>
 <AddressItem
    AddressType="IPV4"
    NumberOfAddresses"3"
    Format="XML" AddressString="10.11.102.23 10.11.104.23 192.168.0.1 192.168.0.12"/>
 </Event>
 <Event EventType=”Comm” CommType=”Conversation”>
 <Conversation
 ConversationType=”IPV4”
 EndPointA=”10.11.102.23”
 EndPointB=”10.11.104.23”>
 <PacketItem
    NumberOfPackets="100"
    PacketType="Filter"
    Filter="src host 10.11.102.23 and dst host 10.11.104.23"
    Format="PCAP">
    test.pcap
 </PacketItem>
 <Traffic
	TrafficType=”bin”
	Units=”bytes”
	Interval=”0.1”>
 <DataItem NumberType=”Float” Dimensions=”10” Format=”XML”>
 0.0 0.0 0.0 0.0 356.0 500.0 500.0 0.0 0.0 0.0
 </DataItem>
 </Traffic>
 </Conversation>
 </Event>
 </Event>
 <Event EventType=”Collection” StartTime="1.0" Endtime="2.0">
	Events for time 1.0 – 2.0
 </Event>
 Etc.......
 </Event>
 </Result>

This description (with refinement) will allow for a flexible description of events of interest for a given simulation or experiment. <EventType>s of interest can be added and automatically generated by tools that act on raw GPS and PCAP data. In addition, it’s a clear path forward to develop visualization playbacks of movement and communication data.