Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Improved the Summary.

...

If you have noticed slow services (e.g. Jira, Confluence, Jenkins, etc.) on Technology Nursery lately, I suspect the server was hit with an opportunistic malware attack that uses its CPU for bitcoin mining!  Here is what I did to identify and fix the problem.  Additional   

The malware has been removed.  However, additional work is needed to backup and upgrade Confluence to minimize the attack surface risk by backing up and upgrading Confluence to the latest version.

Symptom

Today, Confluence was very slow to respond.  Looking at Grafana and top, CPU utilization was at 100%.  The following processes where taking up most of the CPU:

...

Stack Overflow has this thread on Jenkins High CPU Usage khugepageds.

Solution

  •  Clean up any errant cron jobs.
    No errant cron jobs found.
  •  Kill all of the errant processes.
    CPU came back down to normal, but crept back up as those processes were restarted by some cron job script.
  •  Kill all errant processes and immediately restart server s14 (the server that hosts jenkins and confluence).
    This time, the processes stayed down.
  •  Rebuild Jenkins image with the latest LTS version 2.164.2
  •  Start Technology Nursery services: technologynursery.start
  •  On Jenkins, upgrade all plugins
    CPU utilization has stayed normal (>2%) for the past two hours.  Confluence is responsive again.

...