Hardware Common Tasks
From Comp519
Locating Files / Searching Files
To find a specific file somewhere on the comp519 server (such as rx_queue.v), use the following command
locate rx_queue.v
To search the text of all NetFPGA library files to find out where a specific variable is used or defined at, (such as the ENABLE_HEADER parameter in rx_queue.v), use the following command. Note that it is case sensitive.
grep -R "ENABLE_HEADER" $NF2_ROOT
Administrative Tools
On your NetFPGA system, there is a special program called nfsudo that allows you to perform the following functions that require administrative (root) access:
Restart machine
nfsudo reboot
Re-initialize NetFPGA PCI controller
nfsudo cpci_reprogram.pl
Run Wireshark
nfsudo wireshark
Run tcpdump
nfsudo tcpdump <<args>> nfsudo tcpdump -i ethXX
Run bitfile download utility
nfsudo nf2_download /path/to/bitfile.bit
Run dumnet packet generator
nfsudo dumbnet <<args>>
Run arp configuration tool
nfsudo arp <<args>>
Run SCONE (root access only needed for integration project)
nfsudo scone <<args>>
Creating a Project
We recommend copying the reference_nic project when creating a new project, because it automatically includes all essential NetFPGA library files. This project can be found at $NF2_ROOT/projects/reference_nic. Your new project should be saved at projects/<your_project_name>. In addition, you should also copy the CPCI_2.1 project and save it to projects/CPCI_2.1/ in your directory. The PCI controller is required for hardware simulation.
When creating a new project, be sure to update your profile's .bashrc variable to point to your new project. For example, change
export NF2_DESIGN_DIR=${HOME}/projects/reference_nic
to
export NF2_DESIGN_DIR=${HOME}/projects/<your_project_name>
Tip 1: During the course projects, you may find that you do not want to "start over" at the Ethernet NIC project again, such as when you go from the Ethernet Switch to Ethernet Router projects. You can always copy and rename any project. However, after copying, you may want to do a make clean and make really_clean in the synth/ directory of the new project to remove all compiler-generated temporary files from the previous project. Otherwise, your new project will begin its life in a cluttered state.
Tip 2: - Be *careful* when copying a project that is checked into Subversion! You do *not* want to copy the hidden .svn folders that store repository data. See the Subversion Tutorial page for more details on how to safely copy items in the repository.
Using Library Modules
You can choose to use some, all, or even none of the original library modules. The projects/<your_project_name>/include/lib_modules.txt file specifies a list of modules to use from the library. The modules are specified relative to the /exports/provided/NF2/lib/verilog/ directory. You can choose to use different modules simply by changing the module path in the lib_modules.txt file.
Tip: By copying the lib_modules.txt file including in the reference_nic project, you will include all of the modules of the basic NIC design.
Adding Custom Code
To add your own code, write the verilog files and put them in the projects/<your_project_name>/src directory.
Tip: You can choose to partition your verilog files into separate directories under src. Note that only one level of hierarchy is usable.
Overriding Library Code
You might decide that you only need to change one file of a library module. You don't need to copy all the sources and remove the library module from lib_modules.txt. Simply copy the file you wish to modify to your project's src directory and modify it there.
If you wish to completely re-write an existing library module, you can copy the library code and modify it in your project directory. Since you are not using the original library code, be sure to remove that module name from the lib_modules.txt file.
Using Xilinx CoreGen Modules
If you need to use IP cores generated with Xilinx's Coregen, simply use the graphical CoreGen utility to generate your core.
coregen &
You'll need to create a new temporary project somewhere in your home directory, and set it to use the same FPGA type/size/speed/package as the real NetFPGA board. The file synth/nf2_top.map should provide the necessary specifications. This temporary project does not need to be part of Subversion, since you won't use it after the core is generated.
After creating your new temporary project, you'll be able to browse the list of available cores for your FPGA family and instantiate them. Once you have instantiated your core(s), copy the .xco file that was generated to the projects/<your_project_name>/synth directory. You don't need any of the other files, as the Make process will automatically create the netlist from the .xco file.
Tip: If using a memory, you might want to create a .coe file (coefficient file) that will set initial values. In this case, you'll need to copy the .coe file to the /synth directory as well, and update the .xco file to point to it. Due to the way the Makefile changes its place in the directory structure, you might need an unusual path in the .xco file, such as
CSET coefficient_file=../my_coefficient_file.coe
Tip: If you are unable to open the PDF documents for each core (by right-clicking on the core name and choosing "View Data Sheet"), you need to specify the PDF program used to view the file. Go to File->Preferences, uncheck the box labeled "Use default PDF viewer", and enter /usr/bin/evince
Using Xilinx Timing Analyzer
The Xilinx Timing Analyzer tool can be used to view all the clock domains in your design and see what signals do not meet the timing requirements placed on them. To run the timing analyzer, set an environment variable (which can be saved in your .bashrc file)
export XKEYSYMDB=/Xilinx91i/data/XKeysymDB
and then execute the program
timingan &
To analyze your design, go to the File->Open Design menu, and enter the paths to your design. For example:
Design File:
/exports/home/<username>/projects/ethernet_switch/synth/nf2_top.ncd
Physical Constraints File:
/exports/home/<username>/projects/ethernet_switch/synth/nf2_top.pcf
Click OK to open your design. After processing, choose Analyze->Against Timing Constraints. The following options are important:
- "Timing Constraints" tab:
- You can choose what timing constraints you wish to view. The "TS_core_clk_int" is the main 125MHz domain, but you probably want to include all of the constraints to make sure there aren't any other errors.
- You can view all signals (pass & fail) for a domain, or just failing signals. Select "Report failing paths against timing constraints", because that is normally what we are interested in.
- "Options" tab:
- You can choose how many failing signals you want to print (under "Limit Report To:"). The default is 3, but we recommend viewing at least 50 to get a better idea of the scope of the problem.
Click OK to analyze your design based on the options you selected. This should take a minute or two to process before displaying the results. The report will show your clock domains and selected signals from those domains. All signals in RED TEXT have failed their timing constraint.
Using Xilinx Floorplanner
The Xilinx Floorplanner tool can be used to view the design implementation on the FPGA and see where each module is physically placed. You can use this information to gain an understanding of which areas of the FPGA are congested and where optimization effort should be applied. To run the floorplanner, set an environment variable (which can be saved in your .bashrc file)
export XKEYSYMDB=/Xilinx91i/data/XKeysymDB
and then execute the program
floorplanner &
Choose File->Open, select your design (e.g. synth/nf2_top.ngd), and select OK twice. It will take a minute or two to process your design before displaying it.
In the Floorplanner, you can select modules in the Design Hierarchy window, and they will be visually highlighted in the FPGA placement window. If you zoom in far enough, you can select individual FPGA blocks, and by clicking/hovering your mouse, learn what specific part of your design it corresponds to. This can be useful in optimizing your projects or identifying why signals are slow. If you see that your module is widely dispersed across the FPGA (perhaps because it produces or consumes data from modules on the other side of the chip), it might be having problems meeting timing constraints. Thus, you'll need to add additional registers to buffer the signal.
Tip: By default, the Floorplanner placement window highlights both the FPGA elements *and* draws wires connecting them together. This is a poor choice as a program default since the wires make it very difficult to view the underlying design. To turn off these wires, click the toolbar button with the text label "Toggle Ratsnest" (it's the 8th button from the right with the pink wire icon)
Using Xilinx Output Files
The /synth directory of your project is filled with compiler-generated text files containing important project information.
- nf2_top.map and nf2_top.mrp - Output files for the Xilinx Map tool (which converts your logic into actual FPGA devices such as LUTs). These reports summarize your design and shows you how many FPGA resources (like Block RAMs, slices, etc.) are occupied and how many are still available for your design.
- nf2_top.srp - Output file from the HDL synthesis tool. This usually has "helpful" descriptions of any syntax errors in your HDL, as well as warnings for any latches that you might have accidentally created.
- nf2_top_par.twr - Output file from the timing analysis program, showing all your clock domains and counting the number of signals in each domain that failed timing. The ideal phrase to have in this file is the magic line "All constraints were met."
- Note: You should ignore the Timing Summary section at the bottom of this file. The "Maximum frequency" statistic is not useful for this design because it has multiple clock domains. You should only be concerned with the number of timing errors found for each individual timing constraint.
- nf2_top_par.par - Outfile file from the place&route program, showing how long it took for the route to be calculated, and how many iterations it took to meet timing. The more iterations, the harder it is working to fit your design.
- synth.txt - Outfile file containing the entire command-line output of the compilation process (in case you missed it as it was scrolling past)
Reading / Writing Hardware Registers
Two utilities are provided to allow you to read and write to registers from the command line: regread and regwrite. To use these tools, you need to provide one or more register addresses, and one or more values to write.
regread [addr1] [addr2] ... regwrite [addr1] [val1] [addr2] [val2] ...
For example:
regread 0x0600004 regwrite 0x0600004 0x101
To compile these tools for your own use, you need to copy them from the provided library into your own directory. The sw/ subdirectory of your project folder would be a good place. For example, if your project is ethernet_switch, copy both the reg_access and common directories to your own project's sw/ directory.
cp -R $NF2_ROOT/lib/C/common ~/projects/ethernet_switch/sw/ cp -R $NF2_ROOT/lib/C/reg_access ~/projects/ethernet_switch/sw/
Then, enter the reg_access directory and compile the utilities:
cd ~/projects/ethernet_switch/src/reg_access make
Note: These tools must be compiled on your group's netfpga system (a 32-bit machine) instead of on the comp519 server (a 64-bit machine). If you accidentally compile the utilities on the wrong system, simply do a make clean in both the reg_access and common directories and recompile it again.
Building / Cleaning a Project
- To build your hardware project, go into the projects/<your_project_name>/synth directory and run the Makefile.
make
The Makefile will invoke the Xilinx ISE compiler, build your project, and produce an output bitfile named nf2_top_par.bit in the /synth directory. In addition, it will rename that bitfile to the name of your project and copy it to the top level of your specific project directory.
Tip: Xilinx Coregen needs X to run, even from the command line as part of the Makefile. Make sure that you have an X server and have enabled X11 forwarding if IP cores are being built (for example, the first time you are simulating or implementing a design.)
Tip: To check the timing performance of your design, look at the nf2_top_par.twr file.
- To clean your hardware project (i.e. delete all compiler-produced files), go into the projects/<your_project_name>/synth directory and issue the following command:
make clean
- To really clean your hardware project (including deleting all of the hardware cores generated by CoreGen such as FIFOs, the Ethernet MAC, and PCI core), go into the projects/<your_project_name>/synth directory and issue the command:
make really_clean
Simulating your Design
The simulation environment and capabilities are described in detail on the Simulation Tutorial page. Each project can have multiple tests that can be run individually or in sequence.
You do not need to build the hardware before simulating it. The simulator will produce all the files it needs, which is only a small subset of the work required to build the hardware.
The simulator initialization script is normally run with the following arguments, where X and Y are major and minor categories, respectively:
nf21_run_test.pl --major <X> --minor <Y>
Common options:
- --major <string>
- Specify the string to match on the first part of the test directory name. This is a perl regular expression. Default is to match all major tests.
- --minor <string>
- Specify the string to match on the last part of the test directory name. This is a perl regular expression. Default is to match all minor tests
- --gui
- Run the simulator interactively with a GUI. (Without this command, the simulator can still be used to create text output files and validate your design.)
Tip: Xilinx Coregen needs X to run, even from the command line as part of the Simulator Makefile. Make sure that you have an X server and have enabled X11 forwarding if IP cores are being built (for example, the first time you are simulating or implementing a design.)
Resetting your FPGA
A CPCI utility is provided that reloads the PCI controller bitstream on the Spartan FPGA. (You are not responsible for the PCI controller or this FPGA). As a side effect of this process, the main Virtex FPGA is also completely reset. Thus, if you need to completely clear the Virtex FPGA for some reason, you can run the command:
nfsudo cpci_reprogram.pl
Downloading your Design
To download your compiled bitfile to the NetFPGA board, first SSH to your NetFPGA machine. Then enter:
nfsudo nf2_download /path/to/bitfile.bit
Verilog to HTML Converter
v2html is a free Perl script that converts verilog designs into webpages, and allows you to visually navigate and search your source code. It can be a useful tool in helping you understand the larger NetFPGA project.
To use this utility, make a new directory, download the tool, and unpack it.
mkdir v2html cd v2html wget http://www.burbleland.com/v2html/v2html.tar.gz tar -xzf v2html.tar.gz
Make the following control script in the directory. Call it build.sh:
emacs build.sh &
Be sure to update SRC_DIR_2 to point to the project you wish to analyze. (You might set it to $NF2_DESIGN_DIR)
#!/bin/bash
# Specify:
# (1) The read-only NetFPGA library
# (2) Your project source directory
# (3) The destination directory (in /tmp)
SRC_DIR_1=/exports/provided/NF2
SRC_DIR_2=/exports/home/test-student/projects/ethernet_hub/
DST_DIR=/tmp/${USER}/v2html_src
## Check if X11 forwarding is enabled on client.
if [ "${DISPLAY}" = "" ];
then
echo "Error: X11 tunneling must be enabled before running this command"
exit 1
fi
# See http://technofreakatchennai.wordpress.com/2007/10/04/bash-copy-with-find/
# for BASH example
# Search for all .v files in source directories, and copy to /tmp
# It is important to copy the USER PROJECT files SECOND, because they
# overwrite the standard library files of the same name that were copied first.
rm -rf ${DST_DIR}
mkdir ${DST_DIR}
find ${SRC_DIR_1}/ -name \*.v -exec cp {} ${DST_DIR}/ \;
find ${SRC_DIR_2}/ -name \*.v -exec cp {} ${DST_DIR}/ \;
# Convert to HTML, and open with Firefox
./v2html -F $DST_DIR/*.v
firefox frame.html &
Make your script executable:
chmod +x build.sh
Run the script, which will parse the verilog files and open a Firefox browser
./build.sh
In the Firefox window that appears, you can navigate your project. The top frame is the hierarchy, the middle frame is the source viewer, and the bottom frame shows the definition. Not all of the files included have a direct place in the hierarchy. (For example, the PCI core running the Spartan FPGA won't be part of the main hierarchy on the Virtex FPGA) To find, for example, the output_port_looup module, scroll down in the top frame to u_board, and click the folder icon to expand the following path:
u_board->nf2_top->nf2_core->user_data_path->output_port_lookup
Note: If a module is instantiated inside a Verilog generate loop, then v2html does not correctly place the module in the hierarchy, and instead displays at at the top level. For example, the nf2_mac_grp module is shown at the top level, but 4 copies of it are actually generated at at u_board->nf2_top->nf2_core.
Transmitting Raw Packets with Dumbnet
Dumbnet (dnet) is a utility to create and send raw packets across the network. You can write simple scripts that construct and send pre-determined packets to your router, which might allow you to construct reproducible tests.
The following BASH script uses dumbnet to create and send some packets. You can run this script and monitor the connection in wireshark to see that these ARP and ECHO packets get a correct response from the network. Be sure the update the destination IP and MAC address in the examples to reflect your group's network, and make sure that the script is executable (chmod +x <script name>.sh).
#!/bin/bash # Dumbnet / dnet documentation: # http://libdnet.sourceforge.net/dnet.8.txt # http://www.networksorcery.com/ # For debugging: # set -x ####################################### # Environment setup ####################################### # Network device to transmit packets on # (For example, this is the local loopback NIC on the NetFPGA machine # that connects into the hub/switch/router) ETH_DEV="eth1" # MAC address of this network device ETH_SRC=`ifconfig $ETH_DEV | awk '/eth/ {print $5}'` # IP address of this network device IP_SRC=`ifconfig $ETH_DEV | grep "inet addr" | awk -F : ' {print $2}' | awk ' {print $1}'` # MAC Address of broadcast "device ETH_BCAST="FF:FF:FF:FF:FF:FF" #echo $ETH_DEV #echo $ETH_SRC #echo $IP_SRC ####################################### # Examples ####################################### # # Notes: # sha = Source Hardware (MAC) Address # spa = Source Protocol (IP) Address # tha = Target Hardware (MAC) Address # tpa = Target Protocol (IP) Address # Notes: # * Only the final command (dumbnet send) should be called with nfsudo # * You can use Wireshark to monitor packets you transmit. # (It will warn you about "Malformed Packets" if there is an error) # Example 1: # Send an ARP request from this machine asking about 10.143.201.66 # First we create the ARP section, and wrap it in an Ethernet frame dumbnet arp op req sha $ETH_SRC spa $IP_SRC tpa 10.143.201.66 | \ dumbnet eth type arp src $ETH_SRC dst $ETH_BCAST | \ nfsudo dumbnet send $ETH_DEV # Example 2: # Send an ICMP echo request to 10.143.201.66 # First we create the ICMP request (with a made-up string comprising # the identifier/sequence#/data portion of the ICMP request), then we # wrap it in an IP header and an Ethernet header # Note that we had to provide the MAC address corresponding to the destination IP. echo "echo-packet-data-filling-space" | \ dumbnet icmp type 8 code 0 | \ dumbnet ip proto icmp src $IP_SRC dst 10.143.201.66 | \ dumbnet eth type ip src $ETH_SRC dst 00:10:18:1A:0B:0A | \ nfsudo dumbnet send $ETH_DEV
Adding Static ARP Entries
The hardware router will be unable to respond to or initiate ARP requests before it is integrated with the software. Thus, to test the router with applications such as ping, you will need to add static ARP entries to ensure that ARP packets are not sent. Refer to the Network Topology page to determine the exact entries needed for your group. Each group will need to perform the following tasks:
- On server2 machine: Add a static ARP entry defining IP address of router port 3 = MAC address yyy
- On server1 machine: Add a static ARP entry defining IP address of router port 2 = MAC address xxx
- On netfpga01 machine: Add a static ARP entry defining IP address of router port 1 = MAC address zzz
- On your hardware router: Initialize the hardware ARP table with the IP and MAC addresses of netfpga0x, server1, and server2. (Type ifconfig on those machines to find the MAC address of the desired interfaces.)
The following ARP commands can be used:
Show current entries in ARP cache
arp
Remove ARP entry
nfsudo arp -d 10.143.201.1
Add a new (static) ARP entry
nfsudo arp -s 10.143.201.1 00:00:00:00:06:0x
Note: You must choose MAC addresses that do not conflict with other groups! We recommend the following scheme:
00:00:00:0g:0i where g=Group number and i=interface number. So, for group 1: 00:00:00:01:00 (for router eth0) 00:00:00:01:01 (for router eth1) 00:00:00:01:02 (for router eth2) 00:00:00:01:03 (for router eth3)
Note: You can add or remove entries for other groups with these commands. Please be careful to only edit MAC and IP addresses assigned to your group!
Compile Specified Verilog Files
Go to the synth directory and open nf2_top.prj. Remove all the entries listed and add "verilog work <filepath>" for each file you want to try compiling. For example:
verilog work ../src/output_port_lookup/<source_file1>.v verilog work ../src/output_port_lookup/<source_file2>.v
Next type the following in the synth directory:
xst -intstyle ise -ifn nf2_top.scr
The synthesis tool will complain that no top module was found, but you can ignore this since we're just testing compilation.
