DevOps teams: BGP security is BAD. But you can fix it

The security of the Border Gateway Protocol (BGP) is laughable. But we all rely on it every day. For everything.

BGP spoofing can let attackers pretend to be your infrastructure — or that of your cloud provider. But there are things you can do to mitigate the risks.

Are you doing them? Do you want to know how? In this week’s Secure Software Blogwatch, we point you in the right directions.

Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: It’s time for RPKI.
 

“It’s always DNS” — unless it’s BGP

What’s the craic? Dan Goodin reports — “3 hours of inaction from Amazon”:

“It’s not the first time”
Amazon recently lost control of IP addresses it uses to host cloud services and took more than three hours to regain control, a lapse that allowed hackers to steal $235,000. … The hackers seized control of roughly 256 IP addresses through BGP hijacking. … Despite its crucial function in routing wholesale amounts of data across the globe in real time, BGP still largely relies on the Internet equivalent of word of mouth.
…
Last month, [AS209243] suddenly began announcing its infrastructure was the proper path for other ASNs to access … a /24 block of IP addresses belonging to AS16509, one of at least three ASNs operated by Amazon. … A new announcement such as this … should have triggered an immediate investigation by the cloud provider. For reasons that remain unknown, Amazon didn’t start announcing the correct path for the /24 block until … more than three hours after the rogue announcements began.
…
It’s not the first time a BGP attack on an Amazon IP address ended in huge losses of cryptocurrency. … In fairness, Amazon is hardly the only cloud operator to lose control of its IP addresses in a BGP attack. The vulnerability of BGP to ham-fisted misconfigurations and outright fraud has been evident for more than two decades now.

How did it go down? Peter Kacherginsky offers an “Incident analysis”:

“The attacker created a valid certificate”
On August 17, 2022 … users were targeted in a front-end hijacking attack which lasted approximately 3 hours and resulted in 32 impacted victims …  with a single victim losing $156K. … BGP hijacking is a unique attack vector exploiting weakness and trust relationships in the Internet’s core routing architecture. It was used earlier this year to target other cryptocurrency projects such as KLAYswap.
…
The attacker performed initial preparation on August 12, 2022 by deploying a series of malicious smart contracts. … Preparation for the BGP route hijacking took place on August 16th, 2022 and culminated with the attack on August 17, 2022 by taking over a subdomain responsible for serving … bridge contract addresses.
…
The attack targeted the cbridge-prod2.celer.network subdomain which hosted critical smart contract configuration data for the Celer Bridge … UI. Prior to the attack [it] was served by AS-16509 (Amazon) with a 44.224.0.0/11 route. … A new route started propagating for the more specific 44.235.216.0/24 route. … In order to intercept rerouted traffic, the attacker created a valid certificate for the target domain [at] an SSL certificate provider based in Latvia.

And what can we learn? Doug Madory ponders “What can be learned”:

“Raised some eyebrows among the Amazon NetOps team”
Companies looking to secure their internet-facing infrastructures need to deploy robust BGP and DNS monitoring of their infrastructure as well as that of any internet-based dependencies they may have. … Additionally, strict RPKI configuration can also increase the difficulty for someone to hijack your routes.
…
DNS monitoring … uses agents around the world to check that queries for a specified domain return expected results. If a response contains something other than what was expected, it will fire off an alert. … BGP monitoring could have alerted that a new /24 of Amazon address space was being announced. … When this new /24 appeared with an unexpected upstream … that should have triggered an alert [and] raised some eyebrows among the Amazon NetOps team.
…
Amazon had an ROA for the prefix that was hijacked, so why didn’t RPKI ROV help here? … It could have still helped if the ROA were set up [as] other networks such as Cloudflare and Comcast have done: Set the origin and maximum prefix length to be identical to how the prefix is routed. … Had Amazon created a ROA specifically for 44.224.0.0/11 with an origin of AS16509 and a max-prefin-len of 11, then the attacker would have [failed].

ELI5? u/grauenwolf explains like we’re five:

Imagine you could change the road signs so that an armored car delivered money directly to your house instead of the bank. The Border Gateway Protocol is the road signs.

So we all rely on BGP, but it’s fundamentally unsafe? icedchai isn’t shocked:

Not a surprise. [If it was safe], most of the internet would be unreachable, because most routes (60%) are not signed. … Certainly some percentage of this is people being lazy. Though many routes will never be signed because the owners of these routes simply do not have the capability.

It rather sounds like the fault of the TLS cert issuer. hizonner agrees:

Lax CA verification practice and inappropriate reliance on the public CA infrastructure cost cryptocurrency holders $235,000. IP addresses are not an authentication mechanism, folks. The blame here belongs on the X.509 infrastructure and its operators.

However, with a tiny bit more nuance, u/aaaaaaaarrrrrgh screams in frustration:

Any CA would have issued the certificate, as the attacker was able to prove ownership. Better CAs would check from multiple perspectives (network locations) but if the hijack is effective worldwide that wouldn't stop it.

A CAA record restricting the authorized CAs would also not have stopped it unless it was restricted to a set of CAs that won't issue a domain validated cert for that host without additional authentication.

Still, it’s yet another crypto-FAIL for people to point and laugh at. Lest we forget, RuralNinja reminds us that it could happen to any service:

At least this time nothing of value was lost. But it's certainly frustrating seeing time and again that the entire internet is a house of cards supported by a series of gentleman's agreements to not bring it down.

Meanwhile, what of Amazon’s liability here? u/doctorcrimson sees red: [You’re fired—Ed.]

I have a feeling that Amazon will try to not pay reparations citing some obscure policy the user hit Accept on.

And Finally:

RPKwhatnow?

Previously in And finally

Keep learning

• Learn more about Secure Dev & DevSecOps
• Get up to speed on the SBOM's evolution
• Find out why the NVD needs to evolve to include software supply chain threatss
• See survey report: Tampering top of mind for dev — but detection lags

You have been reading Secure Software Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or ssbw@richi.uk. Ask your doctor before reading. Your mileage may vary. Past performance is no guarantee of future results. Do not stare into laser with remaining eye. E&OE. 30.

Image sauce: Kody Goodson (via Unsplash; leveled and cropped)