After some digging and looking at the logs, it appears for some reason Windows 11 re-initiates the VPN connection during the download. I see this pop up in the log:
23:19:45 charon: 14[IKE] no acceptable proposal found
23:19:45 charon: 14[IKE] failed to establish CHILD_SA, keeping IKE_SA
23:19:45 charon: 14[ENC] generating CREATE_CHILD_SA response 4 [ N(NO_PROP) ]
23:19:45 charon: 14[NET] sending packet: from 192.168.1.2[4500] to xx.xx.xx.xx[4500] (65 bytes)
23:20:05 charon: 10[IKE] sending keep alive to xx.xx.xx.xx[4500]
23:20:25 charon: 13[IKE] sending keep alive to xx.xx.xx.xx[4500]
23:20:45 charon: 05[IKE] sending keep alive to xx.xx.xx.xx[4500]
23:20:45 charon: 07[IKE] sending DPD request
23:20:45 charon: 07[ENC] generating INFORMATIONAL request 0 [ ]
23:20:45 charon: 07[NET] sending packet: from 192.168.1.2[4500] to xx.xx.xx.xx[4500] (57 bytes)
23:20:45 charon: 06[NET] received packet: from xx.xx.xx.xx[4500] to 192.168.1.2[4500] (72 bytes)
23:20:45 charon: 06[ENC] parsed INFORMATIONAL response 0 [ ]
23:20:49 charon: 13[NET] received packet: from xx.xx.xx.xx[4500] to 192.168.1.2[4500] (440 bytes)
23:20:49 charon: 13[ENC] parsed CREATE_CHILD_SA request 5 [ SA No TSi TSr ]
23:20:49 charon: 13[CFG] received proposals: ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ, ESP:AES_CBC_128/HMAC_SHA1_96/NO_EXT_SEQ, ESP:3DES_CBC/HMAC_SHA1_96/NO_EXT_SEQ, ESP:DES_CBC/HMAC_SHA1_96/NO_EXT_SEQ, ESP:NULL/HMAC_SHA1_96/NO_EXT_SEQ
23:20:49 charon: 13[CFG] configured proposals: ESP:CHACHA20_POLY1305/CURVE_448/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/CURVE_25519/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/ECP_521/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/ECP_384/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_4096/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_3072/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_2048/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_1024/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/CURVE_448/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/CURVE_25519/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/ECP_521/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/ECP_384/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_4096/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_3072/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_2048/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_1024/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/CURVE_448/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/CURVE_25519/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/ECP_521/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/ECP_384/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_4096/NO_EXT_SEQ, ESP:CHACHA20_POLY1305/MODP_3072/NO_EXT_SEQ, ES
And after a bit of pot luck searching I came across this answer from Microsoft:
In Powershell:
Set-VpnConnectionIPsecConfiguration -ConnectionName <String - your VPN connection name> -AuthenticationTransformConstants SHA256128 -CipherTransformConstants AES128 -DHGroup Group14 -EncryptionMethod AES128 -IntegrityCheckMethod SHA256 -PfsGroup PFS2048 -Force
Fix for Windows 11 VPN IPSEC Disconnection issues
Once applied, it fixed the issue.