ERROR: Auto Attendance Background Job 'Failed'

ERPNext
1

The Auto Attendance background job “shift_type.process_auto_attendance_for_all_shifts” keep failing and giving me the following error:

Traceback (most recent call last):
  File "apps/frappe/frappe/core/doctype/scheduled_job_type/scheduled_job_type.py", line 85, in execute
    frappe.get_attr(self.method)()
  File "apps/erpnext/erpnext/hr/doctype/shift_type/shift_type.py", line 174, in process_auto_attendance_for_all_shifts
    doc.process_auto_attendance()
  File "apps/erpnext/erpnext/hr/doctype/shift_type/shift_type.py", line 67, in process_auto_attendance
    self.mark_absent_for_dates_with_no_attendance(employee)
  File "apps/erpnext/erpnext/hr/doctype/shift_type/shift_type.py", line 142, in mark_absent_for_dates_with_no_attendance
    attendance = mark_attendance(employee, date, "Absent", self.name)
  File "apps/erpnext/erpnext/hr/doctype/attendance/attendance.py", line 195, in mark_attendance
    attendance.insert()
  File "apps/frappe/frappe/model/document.py", line 255, in insert
    self._validate_links()
  File "apps/frappe/frappe/model/document.py", line 899, in _validate_links
    invalid_links, cancelled_links = self.get_invalid_links()
  File "apps/frappe/frappe/model/base_document.py", line 633, in get_invalid_links
    values = frappe.db.get_value(doctype, docname, values_to_fetch, as_dict=True)
  File "apps/frappe/frappe/database/database.py", line 459, in get_value
    ret = self.get_values(
  File "apps/frappe/frappe/database/database.py", line 531, in get_values
    out = self._get_values_from_table(
  File "apps/frappe/frappe/database/database.py", line 726, in _get_values_from_table
    r = self.sql(
  File "apps/frappe/frappe/database/database.py", line 174, in sql
    self._cursor.execute(query, values)
  File "env/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "env/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "env/lib/python3.8/site-packages/pymysql/connections.py", line 547, in query
    self._execute_command(COMMAND.COM_QUERY, sql)
  File "env/lib/python3.8/site-packages/pymysql/connections.py", line 814, in _execute_command
    self._write_bytes(packet)
  File "env/lib/python3.8/site-packages/pymysql/connections.py", line 754, in _write_bytes
    self._sock.settimeout(self._write_timeout)
  File "env/lib/python3.8/site-packages/rq/timeouts.py", line 63, in handle_death_penalty
    raise self._exception('Task exceeded maximum timeout value '
rq.timeouts.JobTimeoutException: Task exceeded maximum timeout value (300 seconds)
2

Now it become this error:

Traceback (most recent call last):
  File "apps/frappe/frappe/core/doctype/scheduled_job_type/scheduled_job_type.py", line 85, in execute
    frappe.get_attr(self.method)()
  File "apps/erpnext/erpnext/hr/doctype/shift_type/shift_type.py", line 174, in process_auto_attendance_for_all_shifts
    doc.process_auto_attendance()
  File "apps/erpnext/erpnext/hr/doctype/shift_type/shift_type.py", line 67, in process_auto_attendance
    self.mark_absent_for_dates_with_no_attendance(employee)
  File "apps/erpnext/erpnext/hr/doctype/shift_type/shift_type.py", line 144, in mark_absent_for_dates_with_no_attendance
    frappe.get_doc(
  File "apps/frappe/frappe/model/document.py", line 291, in insert
    self.run_post_save_methods()
  File "apps/frappe/frappe/model/document.py", line 1100, in run_post_save_methods
    self.save_version()
  File "apps/frappe/frappe/model/document.py", line 1217, in save_version
    version.insert(ignore_permissions=True)
  File "apps/frappe/frappe/model/document.py", line 291, in insert
    self.run_post_save_methods()
  File "apps/frappe/frappe/model/document.py", line 1096, in run_post_save_methods
    self.notify_update()
  File "apps/frappe/frappe/model/document.py", line 1124, in notify_update
    frappe.publish_realtime(
  File "apps/frappe/frappe/__init__.py", line 1928, in publish_realtime
    return frappe.realtime.publish_realtime(*args, **kwargs)
  File "apps/frappe/frappe/realtime.py", line 85, in publish_realtime
    if not params in frappe.local.realtime_log:
  File "env/lib/python3.8/site-packages/rq/timeouts.py", line 63, in handle_death_penalty
    raise self._exception('Task exceeded maximum timeout value '
rq.timeouts.JobTimeoutException: Task exceeded maximum timeout value (300 seconds)
3

Hi all,

I just wanted to bump up this thread since I am facing the same issue.

Any progress or breakthrough in solving this?

4

Increase your site workers maybe?

5

Hi @flexy2ky ,

Thank you for the reply. I went ahead and tried to increase the workers as per your suggestion. In fact, I also checked many of the threads here, and it would seem increasing the number of workers is the way to go. I changed the worker amount in the “common_site_config.json”, and ran the necessary commands:

bench setup supervisor
sudo supervisorctl reread
sudo supervisorctl update

But unfortunately, it did not seem to solve the issue. Please check the attached screenshots.

The kicker is that the jobs used to get completed fine. It just started failing suddenly for the last 3 weeks or so. And I am at my wits end here. I am using ERPNext version 14.

Any suggestion as to what is going wrong?

Screenshot from 2023-06-12 16-46-14
Screenshot from 2023-06-12 16-46-141038×190 21.8 KB

Screenshot from 2023-06-12 16-47-20
Screenshot from 2023-06-12 16-47-201200×552 84.6 KB

This is the exact issue I am getting:

Traceback (most recent call last):
  File "apps/frappe/frappe/core/doctype/scheduled_job_type/scheduled_job_type.py", line 88, in execute
    frappe.get_attr(self.method)()
  File "apps/hrms/hrms/hr/doctype/shift_type/shift_type.py", line 218, in process_auto_attendance_for_all_shifts
    doc.process_auto_attendance()
  File "apps/hrms/hrms/hr/doctype/shift_type/shift_type.py", line 71, in process_auto_attendance
    self.mark_absent_for_dates_with_no_attendance(employee)
  File "apps/hrms/hrms/hr/doctype/shift_type/shift_type.py", line 132, in mark_absent_for_dates_with_no_attendance
    shift_details = get_employee_shift(employee, timestamp, True)
  File "apps/hrms/hrms/hr/doctype/shift_assignment/shift_assignment.py", line 296, in get_employee_shift
    if shift_details and is_holiday_date(employee, shift_details):
  File "apps/hrms/hrms/hr/doctype/shift_assignment/shift_assignment.py", line 364, in is_holiday_date
    return holiday_list_name and is_holiday(holiday_list_name, shift_details.start_datetime.date())
  File "apps/erpnext/erpnext/setup/doctype/holiday_list/holiday_list.py", line 118, in is_holiday
    return bool(frappe.get_all("Holiday List", dict(name=holiday_list, holiday_date=date)))
  File "apps/frappe/frappe/__init__.py", line 1898, in get_all
    return get_list(doctype, *args, **kwargs)
  File "apps/frappe/frappe/__init__.py", line 1870, in get_list
    return frappe.model.db_query.DatabaseQuery(doctype).execute(*args, **kwargs)
  File "apps/frappe/frappe/model/db_query.py", line 172, in execute
    result = self.build_and_run()
  File "apps/frappe/frappe/model/db_query.py", line 212, in build_and_run
    return frappe.db.sql(
  File "apps/frappe/frappe/database/database.py", line 260, in sql
    self.log_query(query, values, debug, explain)
  File "apps/frappe/frappe/database/mariadb/database.py", line 205, in log_query
    self._log_query(query, debug, explain)
  File "apps/frappe/frappe/database/database.py", line 293, in _log_query
    if frappe.conf.allow_tests and frappe.cache().get_value("flag_print_sql"):
  File "env/lib/python3.10/site-packages/werkzeug/local.py", line 308, in __get__
    def __get__(self, instance: "LocalProxy", owner: t.Optional[type] = None) -> t.Any:
  File "env/lib/python3.10/site-packages/rq/timeouts.py", line 61, in handle_death_penalty
    raise self._exception('Task exceeded maximum timeout value '
rq.timeouts.JobTimeoutException: Task exceeded maximum timeout value (300 seconds)
6

BUMP

I investigated further, looked into the RQ-Job list and found that the “hrms.hr.doctype.shift_type.shift_type.process_auto_attendance_for_all_shifts” is failing with the following exception message:

Traceback (most recent call last):
  File "/home/erp2/bench/erpnext/env/lib/python3.10/site-packages/rq/worker.py", line 1075, in perform_job
    rv = job.perform()
  File "/home/erp2/bench/erpnext/env/lib/python3.10/site-packages/rq/job.py", line 854, in perform
    self._result = self._execute()
  File "/home/erp2/bench/erpnext/env/lib/python3.10/site-packages/rq/job.py", line 877, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/home/erp2/bench/erpnext/apps/frappe/frappe/utils/background_jobs.py", line 189, in execute_job
    frappe.db.commit()
  File "/home/erp2/bench/erpnext/apps/frappe/frappe/database/database.py", line 997, in commit
    self.sql("commit")
  File "/home/erp2/bench/erpnext/apps/frappe/frappe/database/database.py", line 218, in sql
    self._cursor.execute(query, values)
  File "/home/erp2/bench/erpnext/env/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/erp2/bench/erpnext/env/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/erp2/bench/erpnext/env/lib/python3.10/site-packages/pymysql/connections.py", line 547, in query
    self._execute_command(COMMAND.COM_QUERY, sql)
  File "/home/erp2/bench/erpnext/env/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
    raise err.InterfaceError(0, "")
pymysql.err.InterfaceError: (0, '')

I cannot seem to figure this, and it has become a burning issue for my production.

Any help will be highly appreciated.

7

Hello,

I was finally able to solve the issue for the Timeout error. I had to increase the default timeout value (which is set to 300 seconds). For my instance, the job successfully completed at around 9 minutes; so I had to change mine to 10 minutes.

You should be able to find the default timeout value in the “background_jobs.py” file:
/bench/{{instance_name}}/apps/frappe/frappe/utils/background_jobs.py
It is in line 36:

@lru_cache
def get_queues_timeout():
	common_site_config = frappe.get_conf()
	custom_workers_config = common_site_config.get("workers", {})
	default_timeout = 300

	return {
		"default": default_timeout,
		"short": default_timeout,
		"long": 1500,
		**{
			worker: config.get("timeout", default_timeout)
			for worker, config in custom_workers_config.items()
		},
	}

You may need to restart the service since the change was made to a .py file.

sudo service supervisor stop
sudo service supervisor start

Hope it helps! Cheers!