When computing the reward, how do you usually choose the value of epsilon? I noticed that in your work, the coefficients for throughput and RTT are both set to 1, and it seems they’re not normalized. In that case, when taking the difference between the new and old utility, the change in throughput could be much larger than the change in delay.
After varying epsilon over a wide range, we set it to -10, which yielded the highest average throughput. Throughput always exhibits slight fluctuations, so even during high-throughput periods, Unew – Uold can become negative. To accommodate this, epsilon must be set to a negative value. Furthermore, normalization was not performed this time. The values of beta and gamma were determined based on parameter tuning results under the evaluation conditions used here. This implies that throughput is prioritized over latency, which is correct as you pointed out. Note that during congestion, Unew - Uold < 0. We think that a key future challenge is the need to reproduce stable, high-throughput results independent of network parameters. We are therefore exploring normalizing beta, gamma, and epsilon based on the throughput and RTT terms.
