download.automotivelinux.org needs more cores / io
Description
Environment
Attachments
- 10 Dec 2018, 02:26 AM
- 10 Dec 2018, 02:25 AM
Activity
Ryan Day May 8, 2019 at 1:51 PM
We are experiencing a DDOS to some of our properties in Vexxhost that may be having some indirect impact on AGL.
However, when I look at the latency and performance of download.automotivelinux.org now, it seems to be performing normally. Latency from my location is averaging 115ms. Do you see any problems when in a traceroute or mtr that you can post?
Jan-Simon Moeller May 8, 2019 at 10:01 AM
[11:13:43] <OutBackDingo> leon-anavi: ok ill give this one a shot also but the download speed is crazy slow from there 4.05M 445KB/s eta 82m 58s
[11:49:56] <leon-anavi> hm... slow download again...
Leon Anavi May 7, 2019 at 2:08 PM
@Ryan Day thank you for the quick feedback. Now things are back to normal and I enjoyed 2+ MB/s download speed.
Thanks,
Leon
Ryan Day May 7, 2019 at 1:59 PM
Today's slowdown of the download server appears to be related to outages that Vexxhost experienced last night. I found that latency was high to all the AGL instances (build, download, etc). I was able to restart the networking gateway and latency dropped back to normal. We will be adding monitoring for this condition to catch it in the future.
Thanks for your report.
Leon Anavi May 7, 2019 at 1:51 PM(edited)
Hi @Ryan Day,
Today I have experienced again very slow download:
$ wget https://download.automotivelinux.org/AGL/snapshots/master/latest/raspberrypi3/deploy/images/raspberrypi3/agl-demo-platform-crosssdk-raspberrypi3.wic.xz
--2019-05-07 15:08:08-- https://download.automotivelinux.org/AGL/snapshots/master/latest/raspberrypi3/deploy/images/raspberrypi3/agl-demo-platform-crosssdk-raspberrypi3.wic.xz
Resolving download.automotivelinux.org (download.automotivelinux.org)... 199.19.213.77
Connecting to download.automotivelinux.org (download.automotivelinux.org)|199.19.213.77|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 693565544 (661M) [application/x-xz]
Saving to: ‘agl-demo-platform-crosssdk-raspberrypi3.wic.xz’agl-demo-platform-crosssdk-raspb 43%[========================> ] 290,95M 267KB/s in 69m 19s 2019-05-07 16:17:29 (71,6 KB/s) - Connection closed at byte 305083259. Retrying.--2019-05-07 16:17:30-- (try: 2) https://download.automotivelinux.org/AGL/snapshots/master/latest/raspberrypi3/deploy/images/raspberrypi3/agl-demo-platform-crosssdk-raspberrypi3.wic.xz
Connecting to download.automotivelinux.org (download.automotivelinux.org)|199.19.213.77|:443... failed: Connection timed out.
Resolving download.automotivelinux.org (download.automotivelinux.org)... 199.19.213.77
Connecting to download.automotivelinux.org (download.automotivelinux.org)|199.19.213.77|:443... failed: No route to host.
$ mtr download.automotivelinux.org -i 2 -c 30 --report-wide
Start: Tue May 7 16:51:35 2019
HOST: leon-desktop-G1 Loss% Snt Last Avg Best Wrst StDev
1.|-- router.asus.com 0.0% 30 0.3 0.3 0.2 1.9 0.2
2.|-- ge-0.aggregation.cooolbox.bg 0.0% 30 11.0 15.2 1.9 60.6 13.0
3.|-- 94.155.94.22 0.0% 30 3.2 3.1 2.9 5.6 0.5
4.|-- henet.bix.bg 0.0% 30 3.0 3.1 2.9 4.2 0.2
5.|-- 100ge10-1.core1.vie1.he.net 0.0% 30 19.6 22.6 19.6 106.6 15.9
6.|-- 100ge13-1.core1.par2.he.net 0.0% 30 35.1 36.3 35.0 53.2 3.9
7.|-- 100ge14-1.core1.nyc4.he.net 0.0% 30 105.6 105.7 105.5 107.8 0.4
8.|-- 100ge10-1.core1.ymq1.he.net 0.0% 30 118.1 117.7 117.4 120.0 0.4
9.|-- estruxture-data-centers.10gigabitethernet1-1-40.switch3.ymq1.he.net 0.0% 30 117.6 117.7 117.5 119.2 0.0
10.|-- 64.15.78.6 0.0% 30 117.6 117.5 117.3 118.0 0.0
11.|-- compute-199-19-213-77.ymq-zone1.thenebulacloud.com 0.0% 30 159.0 242.3 126.1 557.6 111.2
Could you please have a look what might be causing it?
Thanks,
Leon
We need to improve performance of the download server when multiple jenkins jobs run and external hosts download items.
Usually we have good speeds also for external downloads. But when slaves spin-up, external downloads go down to 100k/sec and below.