download.automotivelinux.org needs more cores / io

Fixed

Description

We need to improve performance of the download server when multiple jenkins jobs run and external hosts download items.

Usually we have good speeds also for external downloads. But when slaves spin-up, external downloads go down to 100k/sec and below.

Environment

None

Attachments

2

Activity

Show:

Ryan Day 
May 8, 2019 at 1:51 PM

We are experiencing a DDOS to some of our properties in Vexxhost that may be having some indirect impact on AGL.

However, when I look at the latency and performance of download.automotivelinux.org now, it seems to be performing normally.  Latency from my location is averaging 115ms.  Do you see any problems when in a traceroute or mtr that you can post?

Jan-Simon Moeller 
May 8, 2019 at 10:01 AM

[11:13:43] <OutBackDingo> leon-anavi: ok ill give this one a shot also but the download speed is crazy slow from there 4.05M 445KB/s eta 82m 58s [11:49:56] <leon-anavi> hm... slow download again...

Leon Anavi 
May 7, 2019 at 2:08 PM

thank you for the quick feedback. Now things are back to normal and I enjoyed 2+ MB/s download speed.

Thanks,
Leon

Ryan Day 
May 7, 2019 at 1:59 PM

Today's slowdown of the download server appears to be related to outages that Vexxhost experienced last night.  I found that latency was high to all the AGL instances (build, download, etc).  I was able to restart the networking gateway and latency dropped back to normal.  We will be adding monitoring for this condition to catch it in the future.

Thanks for your report.

Leon Anavi 
May 7, 2019 at 1:51 PM
(edited)

Hi ,

Today I have experienced again very slow download:

$ wget https://download.automotivelinux.org/AGL/snapshots/master/latest/raspberrypi3/deploy/images/raspberrypi3/agl-demo-platform-crosssdk-raspberrypi3.wic.xz --2019-05-07 15:08:08-- https://download.automotivelinux.org/AGL/snapshots/master/latest/raspberrypi3/deploy/images/raspberrypi3/agl-demo-platform-crosssdk-raspberrypi3.wic.xz Resolving download.automotivelinux.org (download.automotivelinux.org)... 199.19.213.77 Connecting to download.automotivelinux.org (download.automotivelinux.org)|199.19.213.77|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 693565544 (661M) [application/x-xz] Saving to: ‘agl-demo-platform-crosssdk-raspberrypi3.wic.xz’agl-demo-platform-crosssdk-raspb 43%[========================> ] 290,95M 267KB/s in 69m 19s 2019-05-07 16:17:29 (71,6 KB/s) - Connection closed at byte 305083259. Retrying.--2019-05-07 16:17:30-- (try: 2) https://download.automotivelinux.org/AGL/snapshots/master/latest/raspberrypi3/deploy/images/raspberrypi3/agl-demo-platform-crosssdk-raspberrypi3.wic.xz Connecting to download.automotivelinux.org (download.automotivelinux.org)|199.19.213.77|:443... failed: Connection timed out. Resolving download.automotivelinux.org (download.automotivelinux.org)... 199.19.213.77 Connecting to download.automotivelinux.org (download.automotivelinux.org)|199.19.213.77|:443... failed: No route to host.
$ mtr download.automotivelinux.org -i 2 -c 30 --report-wide Start: Tue May 7 16:51:35 2019 HOST: leon-desktop-G1 Loss% Snt Last Avg Best Wrst StDev 1.|-- router.asus.com 0.0% 30 0.3 0.3 0.2 1.9 0.2 2.|-- ge-0.aggregation.cooolbox.bg 0.0% 30 11.0 15.2 1.9 60.6 13.0 3.|-- 94.155.94.22 0.0% 30 3.2 3.1 2.9 5.6 0.5 4.|-- henet.bix.bg 0.0% 30 3.0 3.1 2.9 4.2 0.2 5.|-- 100ge10-1.core1.vie1.he.net 0.0% 30 19.6 22.6 19.6 106.6 15.9 6.|-- 100ge13-1.core1.par2.he.net 0.0% 30 35.1 36.3 35.0 53.2 3.9 7.|-- 100ge14-1.core1.nyc4.he.net 0.0% 30 105.6 105.7 105.5 107.8 0.4 8.|-- 100ge10-1.core1.ymq1.he.net 0.0% 30 118.1 117.7 117.4 120.0 0.4 9.|-- estruxture-data-centers.10gigabitethernet1-1-40.switch3.ymq1.he.net 0.0% 30 117.6 117.7 117.5 119.2 0.0 10.|-- 64.15.78.6 0.0% 30 117.6 117.5 117.3 118.0 0.0 11.|-- compute-199-19-213-77.ymq-zone1.thenebulacloud.com 0.0% 30 159.0 242.3 126.1 557.6 111.2

Could you please have a look what might be causing it?

Thanks,
Leon

Details

Assignee

Reporter

Priority

Created December 10, 2018 at 2:24 AM
Updated May 17, 2019 at 12:28 PM
Resolved May 7, 2019 at 2:08 PM

Flag notifications