HTSim

htsim is an open-source network simulator optimized for large-scale datacenter topology simulation. It was initially developed at University College London, than University Politehnica of Bucharest and more recently by Broadcom.

The github page for htsim is here: https://github.com/Broadcom/csg-htsim

To get started with htsim, first the repo and run make in the sim directory. This will build the entire codebase. If you want to extend htsim, you can find information about the source code in the associated wiki.

Running experiments with htsim

The main files to run experiments are in the datacenter directory. There is a htsim_* executable for all the transport protocols htsim currently supports. Each of these setup a variant of a FatTree topology (with configurable number of Nodes), create transport connections and then run a traffic pattern that is either fixed (a permutation for htsim_tcp) or given as a parameter (htsim_ndp, htsim_eqds, htsim_roce, htsim_swift). Each of these main files accepts a range of parameters that control the queue size, initial window, and many more. Have a look at the source code for a definitive guide. Creating a connection matrix

The connection_matrices directory contains python files to generate connection matrix files which can be used as inputs to the experiments.

For instance, to generate a permutation traffic matrix for a network of 16 nodes, with 2MB flows you can run:

python3 gen_permutation.py perm_16n_16c_2MB.cm 16 16 2000000 0 0

To generate the same permutation traffic matrix but with infinite flows run:

python3 gen_permutation.py perm_16n_16c.cm 16 16 0 0 0

Run the script without any parameters to see how to generate custom permutation cms.

After this, we have two traffic matrices one with 2MB flows and one with infinite (0 sized) flows. Examine the files to understand their format.

Running NDP

To run an NDP experiment, a required argument is the strategy for packet-level multipathing which can be perm (source routed, near-perfect), ecmp (virtual paths given with -paths X), ecmp_host (similar but with actual switches doing hashes), ecmp_ar - adaptive routing.

To mimic ECMP, we will be using ecmp_host . Example to run permutation with ECMP, 16 paths,mtu 4K,initial window of 50 packets, 50 packet switch queues:

./htsim_ndp -nodes 16 -tm connection_matrices/perm_16n_16c.cm -cwnd 50 -strat ecmp_host -paths 16 -log sink -q 50 -end 1000 -mtu 4000

To see the average flow throughputs use * ../parse_output logout.dat -ndp -show

Exercise 1. Plot the flow throughput distributions when using 1,4,8,16 and 32 paths.

In our next exercise, we will run the same traffic matrix but with limited size flows of 2MB. We will be using the first TM file we generated, the one named _2MB.cm:

./htsim_ndp -nodes 16 -tm connection_matrices/perm_16n_16c.cm -cwnd 50 -strat ecmp_host -paths 16 -log sink -q 50 -end 1000 -mtu 4000

Grep "finished" in the output to find the flow completion times.

Exercise 2. Plot the distribution of flow completion times when using 1,4,8,16 and 32 paths.

Exercise 3. Run the same experiment for Swift as in Exercise 1 and compare the results to NDP with 1 path.

Exercise 4. Run a permutation using htsim_tcp. Study the parameters to understand how to obtain similar results to the ones above.