This is Part 2 in our 2-part mini-series guide on the Log4j vulnerability that hit the globe in December 2021. For the first post in the series, please go here:
- Part 1 – Log4j Vulnerability – What you need to know (click to open in another tab)
- Part 2 – Log4shell – How it is being dealt with (this post)
Welcome to Part 2 of Nexon’s Guide to Log4shell – The COVID of Applications.
In Part 1, we brought you up to speed with everything you need to know about the Log4j vulnerability, including what it is and how it affects you and your organisation. If you need to catch up, you can go read Part 1 first: Log4j Vulnerability – What you need to know.
Now it’s time to get down to Part 2. We’re going to break down – step by step – how Nexon’s Cloud, Network, and Security teams worked to ensure that not only we were protected, but of course, that our clients were too.
Part 2 – How Nexon dealt with Log4shell
First, we need to outline the vendor response to the log4j vulnerability, because their response largely dictated how our teams were able to proceed.
NOTE: “Vendor” refers to the organisations that leverage open-source software to build and maintain products to sell, irrespective of organisational size.
Code flaws are inevitable, irrespective of whether using proprietary or open-source code. Thus, a key item to bring to light here is that a vendor’s responsibility is to ensure their customers have the resources needed to rectify a situation like this. However, unless a customer has purchased a support package, or a higher level of service or subscription, the vendor’s responsibility ends there. From that point, it is the responsibility of the owner/lessee of the device or software to deploy the patches/fixes provided.
Apache released a list of all the major vendors and their products that leveraged log4j in some way shape or form. This list enabled organisations to quickly go through and see if and how they may be affected, and to what extent they would need to possibly cut off outside access to their devices.
The list also served as a gate way to quickly access the vendors’ update pages so that IT teams could ensure that they were seeing the most current updates and information in regard to either new workarounds provided, or new patched versions released.
As a partner of many of the vendors on the list who had at-risk products, we use those products in the delivery of our solutions every day. Proudly. Because we believe that they are the best fit for clients in need of the solutions we provide.
In this particular situation, due to both the hardware and software affected by this vulnerability, and the severity of it, we had three teams working around the clock to ensure that not only Nexon was kept secure, but that our many clients were as well. Our Cloud, Network, and Security teams worked together, and we ended up with a surprising outcome.
Cybersecurity knows no weekends, or holidays either
News of Log4shell reached our SOC team leader, Shivanthan, on a Saturday. He quickly rallied the troops, and the team was on it. A level 10 threat is no laughing matter.
The timing of this whole situation probably couldn’t have been much worse. December? Socialising, parties, family get-togethers, weekends, annual leave? Probably not this year everyone! To their IMMENSE credit, our teams diligently worked when needed and asked. Of course, they had time off, but essentially the Christmas/New Year break for many of them was spent on call – ready to sprint back to their PCs to ensure the security of Nexon and our clients. Truly, these folks are heroes – because without their tireless efforts, Nexon’s outcomes in this situation could have been very different.
Time to work
Internal communications were sent out, and the teams brought up to speed, but it very quickly became apparent that this wasn’t something that would be solved overnight. Actions were assigned to ensure that three fronts were being covered:
- Assess – not only Nexon’s custom infrastructure, but those of our clients – what elements are at risk?
- Protect – deploy workarounds and patched versions of various software, increase threat level protection.
- Detection enablement – through our proactive monitoring already in place are we able to detect when threat actors are attempting to leverage this vulnerability? What other indicators can we utilise to increase our ability to shut these attacks down, and/or can we increase our capability in this area?
Nexon custom monitoring platform affected
The monitoring platform our NOC uses to monitor all faults (need clarification here – is this for all clients, or just clients on Nexon cloud?) – security and otherwise – was one of the systems unsurprisingly affected by the log4j vulnerability. It’s a distributed system with hundreds of nodes deployed everywhere to ensure the best coverage. Yes. Hundreds.
This platform is part of our frontline defense system for all faults, including communications systems, so needless to say, it was a top priority. SOC not only deployed the requisite patches for log4j across the entire system, but at the same time also upgraded the system by deploying the IP block list and known attack signatures at the same time. This new, extensive upgrade can be used with any kinds of firewalls and network devices, and settings can be adjusted from the individual device all the way to the ISP level, and our SOC can manage and update it regularly – without needing any downtime for the system.
Key Takeaway – Evolving Security Monitoring
New world IT has given us something that is very important to understand, and this includes – especially – the realm of security. Gone are the days of “set and forget:” when engineers would just set up a system and be able to leave it to protect the network unless the next time they get budget to commit to upgrades. Continuous and consistent monitoring and updates ensure that your IT environment is secure, no matter what latest threats come out of the woodwork.
The security guard and the janitor
Or, Nexon’s response to the vulnerability-fix cycle, and how to proactively maintain an IT environment
No surprise here that we were among those hit by the vulnerability-fix cycle. How many times? While log4j had 3 versions released, each vendor had different patches and workarounds with each consecutive release. So while the easy answer is three times we went through the cycle, it was actually a daily task for our teams across various products. However, as mentioned, in order to mitigate the potential risk, it was worth the extra effort to ensure that both Nexon and our clients were protected right out of the gate, to the highest level they could be, until everything was stable.
Daily, our team would check the vendor update pages to see if any further progress had been made, changes to work-arounds, or further patches released off the back of the latest version released by Apache. It was very typical of what a building security guard might do – doing through rounds through the systems, checking all the different points and seeing if they needed to be adjusted. Once the team was confident that our customers were 100% secure, some of the remaining upgrades were able to be held off until our clients had staff back in the offices because the patches and workarounds were holding sufficiently.
As the team was doing the daily rounds, they also used this time to proactively clean up things like ports and lists and emails – stuff that wasn’t getting used anymore that could very well end up being a point to be leveraged in future flaws. Think of the janitor: mopping and cleaning up spills to ensure no one slips and gets hurt.
Both of these are key factors in a solid security hygiene strategy.
Safe and secure
We are happy to say that we (Nexon) and all of our clients are safe and protected, thanks to the work of the NOC, SOC, and Cloud teams. It was a lot of work, and relaxation time sacrificed, to ensure this outcome, and we’re grateful for their efforts.
However, what we didn’t expect from all of this was that we, and in turn our clients, would get two additional benefits from this.
Higher-level, more future-proofed protection
First, we now have a suite of assets and upgrades to proactively monitor and detect not just log4shell attacks, but future ones as well. These assets include the upgrading of our monitoring platform, refined basic IoCs (Indicators of Compromise), but also the global block list, and a database of attack signatures provided by vendors and other sources. And finally, the way we protect from these various aspects has also shifted, so that our monitoring system is far more refined, easily updateable, and capable of blocking attacks at the routers, not just the individual devices, blanketing our entire infrastructure in a much more solid net of security.
Nexon internal task force
Second, once the immediate tasks were under way, our teams realised that something permanent should be created to address this and other future vulnerabilities in a more cohesive, less siloed way. And thus, our Threat Protection Task Force was created, including our SOC (Security Operations Centre), NOC (Network Operations Centre), and Cloud Team. Going forward, this task force will ensure that all 3 teams are across everything they can be, and additionally, enable them to work more efficiently to provide the service we strive for.
While situations like this aren’t ideal, it’s reassuring to see the systems put in place by everyone involved are actually working as expected. What could have been a hack that brought the globe to its knees, ended up not nearly as bad as it could have been. Swift action from 10 December onward has prevented this vulnerability from exploding initially in the way that it was predicted – but that’s not to say it’s still not coming down the track.
Yes, the major patches have been released, but to date, there are still millions of devices, organisations, and individuals at risk – and you may be counted among them.
Not sure if your organisation is protected from Log4shell? Let’s chat about what your next steps are.