No JSON object could be decoded IOError: [Errno 28] No space left on device

Hi,

This morning I wasn’t able to access to a site. I thought it was an issue with SSL certificate. When I tried to renew, this error was showed: IOError: [Errno 28] No space left on device.

I delete the biggest file in /tmp: wkhtmltox.tar.xz

For a momentary lapse I was able to enter in the site but then go down again. I tried with bench setup nginx, but now besides the ‘No space left on device’ error, is ‘ValueError: No JSON object could be decoded’.

Searching some solutions in forum, I tried bench setup nginx, bench migrate and bench restart, but stills ‘ValueError: No JSON object could be decoded’ or ‘IOError: [Errno 28] No space left on device’.

[Errno 28] No space left on device

root@vps#####:/home/frappe/frappe-bench# bench restart
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 885, in emit
    self.flush()
  File "/usr/lib/python2.7/logging/__init__.py", line 845, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file utils.py, line 148
INFO:bench.utils:sudo supervisorctl restart frappe-bench-workers: frappe-bench-web:
frappe-bench-workers:frappe-bench-frappe-schedule: stopped
frappe-bench-workers:frappe-bench-frappe-default-worker-0: stopped
frappe-bench-workers:frappe-bench-frappe-long-worker-0: stopped
frappe-bench-workers:frappe-bench-frappe-short-worker-0: stopped
frappe-bench-web:frappe-bench-node-socketio: stopped
frappe-bench-workers:frappe-bench-frappe-schedule: started
frappe-bench-workers:frappe-bench-frappe-default-worker-0: started
frappe-bench-workers:frappe-bench-frappe-long-worker-0: started
frappe-bench-workers:frappe-bench-frappe-short-worker-0: started
frappe-bench-web:frappe-bench-frappe-web: started
frappe-bench-web:frappe-bench-node-socketio: started
root@vps158813:/home/frappe/frappe-bench# bench restart
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 885, in emit
    self.flush()
  File "/usr/lib/python2.7/logging/__init__.py", line 845, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file utils.py, line 148
INFO:bench.utils:sudo supervisorctl restart frappe-bench-workers: frappe-bench-web:
frappe-bench-workers:frappe-bench-frappe-schedule: stopped
frappe-bench-workers:frappe-bench-frappe-default-worker-0: stopped
frappe-bench-workers:frappe-bench-frappe-long-worker-0: stopped
frappe-bench-workers:frappe-bench-frappe-short-worker-0: stopped
frappe-bench-web:frappe-bench-node-socketio: stopped
frappe-bench-workers:frappe-bench-frappe-schedule: started
frappe-bench-workers:frappe-bench-frappe-default-worker-0: started
frappe-bench-workers:frappe-bench-frappe-long-worker-0: started
frappe-bench-workers:frappe-bench-frappe-short-worker-0: started
frappe-bench-web:frappe-bench-frappe-web: started
frappe-bench-web:frappe-bench-node-socketio: started

No JSON object could be decoded

root@vps######:/home/frappe/frappe-bench# bench setup nginx
Traceback (most recent call last):
  File "/usr/local/bin/bench", line 11, in <module>
    load_entry_point('bench', 'console_scripts', 'bench')()
  File "/home/frappe/.bench/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/.bench/bench/commands/setup.py", line 22, in setup_nginx
    make_nginx_conf(bench_path=".", yes=yes)
  File "/home/frappe/.bench/bench/config/nginx.py", line 13, in make_nginx_conf
    sites = prepare_sites(config, bench_path)
  File "/home/frappe/.bench/bench/config/nginx.py", line 100, in prepare_sites
    sites_configs = get_sites_with_config(bench_path=bench_path)
  File "/home/frappe/.bench/bench/config/nginx.py", line 190, in get_sites_with_config
    site_config = get_site_config(site, bench_path=bench_path)
  File "/home/frappe/.bench/bench/config/site_config.py", line 11, in get_site_config
    return json.load(f)
  File "/usr/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I hadn’t make any changes in ‘common_site_config.json’, anyway I validated. But when I look at ‘site_config.json’ it’s completely blank and I didn’t make any changes on it.

Is there a way to rebuild a ‘site_config.json’ file?
Or should I create a new site and restore database from backup files?

At /tmp are these files, even some folders are empty, can be all deleted?

Tmp

Sounds like you are seeing many random errors due to one of your partitions being full quite likely the / (root) partition.

Most files in /tmp won’t be big and won’t affect the server being full. It’s more likely vthat your /var partition is full or / partition normally. I think you need to try and run

df -h

to see what space is available or not in what partitions. If you have LVM you can expand your space to give some breathing space to your system.

You’re right Julian.

Anyway I can’t figure why the ‘site_config.json’ is completely blank and can’t find how to restore. I still trying creating a new site to restore a backup.

I had something like this happen on a ERPNext site. I presume when the site_config.json was going to be written to, there was no space and thus the file couldn’t be written.

You’re best bet is to restore from backups but it’s likely that other files may also be corrupted so I think you should try and restore the whole site to be sure not just this one missing file

1 Like

Always take backup of site_config.json.
I also observe some scenario, where if space is not left, then site_config.json will be black