Fixing “No space left on device” – Linux

Yesterday evening our Jenkins Dev Build job started failing whenever it tried to transfer the artifacts to Dev node stating the reason – “No space left on device” as follows:

mkdir: cannot create directory `/apps/jboss/dev-deployment/backup/28November2016_15:07:22': No space left on device

Failure message itself made us understand that there was a memory issue on the Dev node which was making the build job to fail. So we quickly logged into the machine and started diagnosing it following below command:

$ df -h

Above command lists the information regarding Filesystem along with total size, used and available of memory as below:

Filesystem            Size Used Avail Use% Mounted on
/dev/mapper/VG00-root 2.0G 510M 1.4G 28% /
tmpfs                 16G 12K 16G 1%
/dev/shm/dev/sda1     190M 86M 94M 48% /boot
/dev/mapper/VG00-home 4.8G 11M 4.6G 1% /home
/dev/mapper/VG00-opt  20G 313M 19G 2% /opt
/dev/mapper/VG00-tmp  4.8G 110M 4.5G 3% /tmp
/dev/mapper/VG00-usr  9.8G 1.6G 7.7G 18% /usr
/dev/mapper/VG00-var  4.8G 857M 3.8G 19% /var
/dev/mapper/VG_APPS-apps 36G 36G 17M 100% /apps
/dev/mapper/nb_vg-lv_openv 9.8G 3.3G 6.0G 36% /usr/openv

Looking over the result made us understand that /dev/mapper/VG_APPS-apps is mounted with 36 GB which is totally occupied and eventually making our Jenkins job fail.

Provided information was not enough to know the exact place which was occupied, in order to find out we further diagnosed it following below command:

$ du -h /apps/ --max-depth=2

which listed below result:

300K /apps/jboss/.activator
4.9G /apps/jboss/jboss-eap-6.4.0
171M /apps/jboss/.sbt
16K /apps/jboss/.fontconfig
8.0K /apps/jboss/.vim
12K /apps/jboss/.oracle_jre
120M /apps/jboss/tools
353M /apps/jboss/jdk1.8.0_71
26GB /apps/jboss/dev-deployment
48K /apps/jboss/.jmc
8.0K /apps/jboss/.ssh
298M /apps/jboss/.ivy2
8.4G /apps/jboss

Looking over the result again we got an idea that there was something inside /apps/jboss/dev-deployment which occupied almost all of the space mounted to the directory. Getting more deeper

$ cd /apps/jboss/dev-deployment/
$ du -h --max-depth=1

Apparently helped to know that it was a log folder which occupied the complete directory size

32K ./data
8.0K ./lib
600K ./configuration
577M ./tmp
26GB ./log
297M ./deployments
945M .

We deleted unwanted logs from log directory and restarted Jenkins Dev Build job assuming it should work but it again failed stating the same reason.

We ran the same commands again this time with more curiosity:

$  df -h /apps/

Apparently result showed us the disk was still full:

Filesystem              Size Used Avail  Use% Mounted on
/dev/mapper/VG_APPS-apps 36G 36G  17M    100% /apps

Whereas checking the disk space used by files

$  du -h /apps/

showed only 8.4GB is consumed:

300K /apps/jboss/.activator
4.9G /apps/jboss/jboss-eap-6.4.0
171M /apps/jboss/.sbt
16K /apps/jboss/.fontconfig
8.0K /apps/jboss/.vim
12K /apps/jboss/.oracle_jre
120M /apps/jboss/tools
353M /apps/jboss/jdk1.8.0_71
200M /apps/jboss/dev-deployment
48K /apps/jboss/.jmc
8.0K /apps/jboss/.ssh
298M /apps/jboss/.ivy2
8.4G /apps/jboss

These results makes us worried where the space was occupied, then we came across this answer on  which says there might be chances that file is still opened by some process.

And as suggested in the solution, we restarted Dev node along with Jenkins Dev Build job and were able to turn our build job “GREEN”.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s