Skip to content

Commit

Permalink
Performance tests
Browse files Browse the repository at this point in the history 
Signed-off-by: Maarit Härkönen <[email protected]>
  • Loading branch information
@maaharko
maaharko committed Nov 25, 2024
1 parent a4f253f commit 149c6db
Show file tree
Hide file tree
Showing 2 changed files with 190 additions and 64 deletions.
8 changes: 4 additions & 4 deletions Robot-Framework/lib/PerformanceDataProcessing.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,10 @@ def write_mem_to_csv(self, test_name, mem_data):

@keyword
def write_speed_to_csv(self, test_name, speed_data):
data = [self.build_number,
speed_data['tx'],
speed_data['rx'],
self.device]
data = [self.build_number]
for key in speed_data.keys():
data.append(speed_data[key])
data.append(self.device)
self._write_to_csv(test_name, data)

def write_fileio_data_to_csv(self, test_name, data):
Expand Down
246 changes: 186 additions & 60 deletions Robot-Framework/test-suites/performance-tests/network.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,15 @@ Library ../../lib/output_parser.py
Library Process
Library ../../lib/PerformanceDataProcessing.py ${DEVICE} ${BUILD_ID} ${JOB}
Library Collections
Library JSONLibrary
Suite Setup Common Setup
Suite Teardown Close All Connections


*** Variables ***
${PERF_TEST_TIME} 10


*** Test Cases ***

TCP speed test
Expand All @@ -25,80 +30,151 @@ TCP speed test
Run iperf server on DUT
&{tcp_speed} Run TCP test
${statistics} Save Speed Data ${TEST NAME} ${tcp_speed}
${statistics_tx} Get From Dictionary ${statistics} tx
${statistics_rx} Get From Dictionary ${statistics} rx

Log <img src="${DEVICE}_${TEST NAME}.png" alt="TCP Plot" width="1200"> HTML
Report statistics ${statistics}
[Teardown] Stop iperf server

${fail_msg}= Set Variable ${EMPTY}
IF "${statistics_tx}[flag]" == "-1"
${add_msg} Create fail message ${statistics_tx}
${fail_msg}= Set Variable TX:\n${add_msg}
END
IF "${statistics_rx}[flag]" == "-1"
${add_msg} Create fail message ${statistics_rx}
${fail_msg}= Set Variable ${fail_msg}\nRX:\n${add_msg}
END
IF "${statistics_tx}[flag]" == "-1" or "${statistics_rx}[flag]" == "-1"
FAIL ${fail_msg}
END

${pass_msg}= Set Variable ${EMPTY}
IF "${statistics_tx}[flag]" == "1"
${add_msg} Create improved message ${statistics_tx}
${pass_msg}= Set Variable TX:\n${add_msg}
END
IF "${statistics_rx}[flag]" == "1"
${add_msg} Create improved message ${statistics_rx}
${pass_msg}= Set Variable ${pass_msg}\nRX:\n${add_msg}
END
IF "${statistics_tx}[flag]" == "1" or "${statistics_rx}[flag]" == "1"
Pass Execution ${pass_msg}
END


UDP speed test
[Documentation] Measure RX and TX speed for UDP
[Tags] udp SP-T87 nuc orin-agx orin-nx riscv lenovo-x1
Run iperf server on DUT
&{udp_speed} Run UDP test
${statistics} Save Speed Data ${TEST NAME} ${udp_speed}
${statistics_tx} Get From Dictionary ${statistics} tx
${statistics_rx} Get From Dictionary ${statistics} rx

Report statistics ${statistics}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="UDP Plot" width="1200"> HTML
[Teardown] Stop iperf server

${fail_msg}= Set Variable ${EMPTY}
IF "${statistics_tx}[flag]" == "-1"
${add_msg} Create fail message ${statistics_tx}
${fail_msg}= Set Variable TX:\n${add_msg}
END
IF "${statistics_rx}[flag]" == "-1"
${add_msg} Create fail message ${statistics_tx}
${fail_msg}= Set Variable ${fail_msg}RX:\n${add_msg}
END
IF "${statistics_tx}[flag]" == "-1" or "${statistics_rx}[flag]" == "-1"
FAIL ${fail_msg}
END
Measure TCP Throughput Small Packets
[Documentation] Start server on DUT. Send data from agent PC in reverse mode to get tx speed
[Tags] tcp nuc orin-agx orin-nx riscv lenovo-x1 SSRCSP-T227
[Setup] Run iperf server on DUT
[Teardown] Stop iperf server
&{speed_data} Create Dictionary
# DUT sends
${output1} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -f M -t ${PERF_TEST_TIME} -R shell=True
Log ${output1.stdout}
# DUT receives
${output2} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -f M -t ${PERF_TEST_TIME} shell=True
Log ${output2.stdout}
${bps_tx} Get Throughput Values ${output1.stdout}
${bps_rx} Get Throughput Values ${output2.stdout} direction=receiver
Set To Dictionary ${speed_data} tx ${bps_tx} rx ${bps_rx}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="TCP Transfer Small Packets" width="1200"> HTML
${statistics} Save Speed Data ${TEST NAME} ${speed_data}
Report Statistics ${statistics}

${pass_msg}= Set Variable ${EMPTY}
IF "${statistics_tx}[flag]" == "1"
${add_msg} Create improved message ${statistics_tx}
${pass_msg}= Set Variable TX:\n${add_msg}
END
IF "${statistics_rx}[flag]" == "1"
${add_msg} Create improved message ${statistics_tx}
${pass_msg}= Set Variable ${pass_msg}\nRX:\n${add_msg}
END
IF "${statistics_tx}[flag]" == "1" or "${statistics_rx}[flag]" == "1"
Pass Execution ${pass_msg}
END
Measure TCP Bidir Throughput Small Packets
[Documentation] Start server on DUT. Send data from agent PC in bidir mode to get bi-directional speed
[Tags] tcp nuc orin-agx orin-nx riscv lenovo-x1 SSRCSP-T228
[Setup] Run iperf server on DUT
[Teardown] Stop iperf server
&{speed_data} Create Dictionary
${output} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -f M -t ${PERF_TEST_TIME} --bidir shell=True
Log ${output.stdout}
${bps_tx} Get Throughput Values ${output.stdout} bidir=True
${bps_rx} Get Throughput Values ${output.stdout} direction=receiver bidir=True
Set To Dictionary ${speed_data} tx ${bps_tx} rx ${bps_rx}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="TCP Bidir Transfer Small Packets" width="1200"> HTML
${statistics} Save Speed Data ${TEST NAME} ${speed_data}
Report Statistics ${statistics}

Measure TCP Throughput Big Packets
[Documentation] Start server on DUT. Send data from agent PC in reverse mode to get tx speed
[Tags] tcp nuc orin-agx orin-nx riscv lenovo-x1 SSRCSP-T229
[Setup] Run iperf server on DUT
[Teardown] Stop iperf server
&{speed_data} Create Dictionary
${output1} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -M 9000 -f M -t ${PERF_TEST_TIME} -R shell=True
${output2} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -M 9000 -f M -t ${PERF_TEST_TIME} shell=True
Log ${output1.stdout}
${bps_tx} Get Throughput Values ${output1.stdout}
${bps_rx} Get Throughput Values ${output2.stdout} direction=receiver
Set To Dictionary ${speed_data} tx ${bps_tx} rx ${bps_rx}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="TCP Transfer Big Packets" width="1200"> HTML
${statistics} Save Speed Data ${TEST NAME} ${speed_data}
Report Statistics ${statistics}

*** Keywords ***
Measure TCP Bidir Throughput Big Packets
[Documentation] Start server on DUT. Send data from agent PC in bidir mode to get bi-directional speed
[Tags] tcp nuc orin-agx orin-nx riscv lenovo-x1 SSRCSP-T230
[Setup] Run iperf server on DUT
[Teardown] Stop iperf server
&{speed_data} Create Dictionary
${output} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -M 9000 -f M -t ${PERF_TEST_TIME} --bidir shell=True
Log ${output.stdout}
${bps_tx} Get Throughput Values ${output.stdout} bidir=True
${bps_rx} Get Throughput Values ${output.stdout} direction=receiver bidir=True
Set To Dictionary ${speed_data} tx ${bps_tx} rx ${bps_rx}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="TCP Bidir Transfer Big Packets" width="1200"> HTML
${statistics} Save Speed Data ${TEST NAME} ${speed_data}
Report Statistics ${statistics}

Measure UDP TX Throughput Small Packets
[Documentation] Start server on DUT. Send data from agent PC in reverse mode to get tx speed
[Tags] tcp nuc orin-agx orin-nx riscv lenovo-x1 SSRCSP-T231
[Setup] Run iperf server on DUT
[Teardown] Stop iperf server
&{speed_data} Create Dictionary
${output1} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -u -b 100G -f M -t ${PERF_TEST_TIME} -R shell=True
Log ${output1.stdout}
${output2} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -u -b 100G -f M -t ${PERF_TEST_TIME} shell=True
Log ${output2.stdout}
${bps_tx} Get Throughput Values ${output1.stdout}
${bps_rx} Get Throughput Values ${output2.stdout} direction=receiver
Set To Dictionary ${speed_data} tx ${bps_tx} rx ${bps_rx}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="UDP Transfer Small Packets" width="1200"> HTML
${statistics} Save Speed Data ${TEST NAME} ${speed_data}
Report Statistics ${statistics}

Measure UDP Bidir Throughput Small Packets
[Documentation] Start server on DUT. Send data from agent PC in bidir mode to get bi-directional speed
[Tags] tcp nuc orin-agx orin-nx riscv lenovo-x1 SSRCSP-T232
[Setup] Run iperf server on DUT
[Teardown] Stop iperf server
&{speed_data} Create Dictionary
${output} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -u -b 100G -f M -t ${PERF_TEST_TIME} --bidir shell=True
Log ${output.stdout}
${bps_tx} Get Throughput Values ${output.stdout} bidir=True
${bps_rx} Get Throughput Values ${output.stdout} direction=receiver bidir=True
Set To Dictionary ${speed_data} tx ${bps_tx} rx ${bps_rx}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="UDP" Bidir Transfer Small Packets" width="1200"> HTML
${statistics} Save Speed Data ${TEST NAME} ${speed_data}
Report Statistics ${statistics}

Measure UDP Throughput Big Packets
[Documentation] Start server on DUT. Send data from agent PC in reverse mode to get tx speed
[Tags] udp nuc orin-agx orin-nx riscv lenovo-x1 SSRCSP-T233
[Setup] Run iperf server on DUT
[Teardown] Stop iperf server
&{speed_data} Create Dictionary
${output1} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -l 9000 -u -b 100G -f M -t ${PERF_TEST_TIME} -R shell=True
Log ${output1.stdout}
${output2} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -l 9000 -u -b 100G -f M -t ${PERF_TEST_TIME} shell=True
Log ${output2.stdout}
${bps_tx} Get Throughput Values ${output1.stdout}
${bps_rx} Get Throughput Values ${output2.stdout} direction=receiver
Set To Dictionary ${speed_data} tx ${bps_tx} rx ${bps_rx}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="UDP Transfer Big Packets" width="1200"> HTML
${statistics} Save Speed Data ${TEST NAME} ${speed_data}
Report Statistics ${statistics}

Measure UDP Bidir Throughput Big Packets
[Documentation] Start server on DUT. Send data from agent PC in bidir mode to get bi-directional speed
[Tags] udp nuc orin-agx orin-nx riscv lenovo-x1 SSRCSP-T234
[Setup] Run iperf server on DUT
[Teardown] Stop iperf server
&{speed_data} Create Dictionary
${output} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -l 9000 -u -b 10000G -f M -t ${PERF_TEST_TIME} --bidir shell=True
Log ${output.stdout}
${bps_tx} Get Throughput Values ${output.stdout} bidir=True
${bps_rx} Get Throughput Values ${output.stdout} direction=receiver bidir=True
Set To Dictionary ${speed_data} tx ${bps_tx} rx ${bps_rx}
Log <img src="${DEVICE}_${TEST NAME}.png" alt="UDP Bidir Transfer Big Packets" width="1200"> HTML
${statistics} Save Speed Data ${TEST NAME} ${speed_data}
Report Statistics ${statistics}


*** Keywords ***
Common Setup
Set Variables ${DEVICE}
Run Keyword If "${DEVICE_IP_ADDRESS}" == "NONE" Get ethernet IP address
Expand All @@ -113,14 +189,14 @@ Run iperf server on DUT

Run TCP test
[Documentation] Run network test on agent machine
${output} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -f M -t 10 shell=True
${output} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -f M -t ${PERF_TEST_TIME} shell=True
Should Contain ${output.stdout} iperf Done.
Log ${output.stdout}
&{tcp_speed} Parse iperf output ${output.stdout}
RETURN &{tcp_speed}

Run UDP test
${output} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -u -b 100G -f M -t 10 shell=True
${output} Run Process iperf3 -c ${DEVICE_IP_ADDRESS} -u -b 100G -f M -t ${PERF_TEST_TIME} shell=True
Should Contain ${output.stdout} iperf Done.
Log ${output.stdout}
&{udp_speed} Parse iperf output ${output.stdout}
Expand Down Expand Up @@ -150,3 +226,53 @@ Check iperf was started
Sleep 1
END
IF ${status} == False FAIL Iperf server was not started

Get Throughput Values
[Documentation] Return MB per second value
[Arguments] ${output} ${direction}=sender ${bidir}=False
IF ${bidir}
IF '${direction}' == 'sender'
${MBps} Get Regexp Matches ${output} (?im).*TX-C.*\\s(\\d+(\\.\\d+)?) MBytes\\/sec.*${direction} 1
ELSE
${MBps} Get Regexp Matches ${output} (?im).*RX-C.*\\s(\\d+(\\.\\d+)?) MBytes\\/sec.*${direction} 1
END
ELSE
${MBps} Get Regexp Matches ${output} (?im)\\s(\\d+(\\.\\d+)?) MBytes\\/sec.*${direction} 1
END
${expected_throughput} Set Variable If ${bidir} ${85} ${90}
Run Keyword And Continue On Failure Should Be True ${MBps}[0] > ${expected_throughput}
RETURN ${MBps}[0]

Report statistics
[Documentation] Check deviation of iperf results
[Arguments] ${statistics}
${statistics_tx} Get From Dictionary ${statistics} tx
${statistics_rx} Get From Dictionary ${statistics} rx

${fail_msg}= Set Variable ${EMPTY}
IF "${statistics_tx}[flag]" == "-1"
${add_msg} Create fail message ${statistics_tx}
${fail_msg}= Set Variable TX:\n${add_msg}
END
IF "${statistics_rx}[flag]" == "-1"
${add_msg} Create fail message ${statistics_tx}
${fail_msg}= Set Variable ${fail_msg}RX:\n${add_msg}
END
IF "${statistics_tx}[flag]" == "-1" or "${statistics_rx}[flag]" == "-1"
FAIL ${fail_msg}
END

${pass_msg}= Set Variable ${EMPTY}
IF "${statistics_tx}[flag]" == "1"
${add_msg} Create improved message ${statistics_tx}
${pass_msg}= Set Variable TX:\n${add_msg}
END
IF "${statistics_rx}[flag]" == "1"
${add_msg} Create improved message ${statistics_tx}
${pass_msg}= Set Variable ${pass_msg}\nRX:\n${add_msg}
END
IF "${statistics_tx}[flag]" == "1" or "${statistics_rx}[flag]" == "1"
Pass Execution ${pass_msg}
END

${msg} Set Variable ${EMPTY}

0 comments on commit 149c6db

Please sign in to comment.