frontend / nginx looks normal
queue-long looks like no significant activity:
# docker logs -f erpnext-one-queue-long-1
04:44:48 Worker rq:worker:b1c7d51978a44e75afcdf31fffaf2c7f.5cbf164610e2.7.long: started, version 1.8.1
04:44:48 Subscribing to channel rq:pubsub:b1c7d51978a44e75afcdf31fffaf2c7f.5cbf164610e2.7.long
04:44:48 *** Listening on long...
04:58:18 Cleaning registries for queue: long
05:00:06 long: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.stock.doctype.repost_item_valuation.repost_item_valua..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (0cdb9fe4-c450-4e19-bf91-8a79049297a3)
05:00:09 long: Job OK (0cdb9fe4-c450-4e19-bf91-8a79049297a3)
05:00:09 Result is kept for 500 seconds
^C
scheduler have no activity
Websocket have some errors, some about chat, some about my activity (payment entry) that cannot be saved:
# docker logs -f erpnext-one-websocket-1
listening on *: 9000
Unable to join chat room. Error: Bad Gateway
Unable to join chat room. Error: Bad Gateway
Unable to join chat room. Error: Bad Gateway
Unable to join chat room. Error: Bad Gateway
Unable to join chat room. Error: Bad Gateway
Unable to join chat room. Error: Bad Gateway
Unable to join chat room. Error: Bad Gateway
Error: Not Found
at Request.callback (/home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/node/index.js:883:15)
at /home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/node/index.js:1126:20
at IncomingMessage.<anonymous> (/home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/node/parsers/json.js:22:7)
at IncomingMessage.emit (node:events:539:35)
at endReadableNT (node:internal/streams/readable:1345:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
status: 404,
response: <ref *1> Response {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
res: IncomingMessage {
_readableState: [ReadableState],
_events: [Object: null prototype],
_eventsCount: 4,
_maxListeners: undefined,
socket: [TLSSocket],
httpVersionMajor: 1,
httpVersionMinor: 1,
httpVersion: '1.1',
complete: true,
rawHeaders: [Array],
rawTrailers: [],
aborted: false,
upgrade: false,
url: '',
method: null,
statusCode: 404,
statusMessage: 'Not Found',
client: [TLSSocket],
_consuming: false,
_dumped: false,
req: [ClientRequest],
text: '{"exc_type":"DoesNotExistError","_server_messages":"[\\"{\\\\\\"message\\\\\\": \\\\\\"Payment Entry new-payment-entry-1 not found\\\\\\", \\\\\\"title\\\\\\": \\\\\\"Message\\\\\\", \\\\\\"indicator\\\\\\": \\\\\\"red\\\\\\", \\\\\\"raise_exception\\\\\\": 1}\\"]"}',
[Symbol(kCapture)]: false,
[Symbol(kHeaders)]: [Object],
[Symbol(kHeadersCount)]: 20,
[Symbol(kTrailers)]: null,
[Symbol(kTrailersCount)]: 0,
[Symbol(RequestTimeout)]: undefined
},
request: Request {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
_enableHttp2: false,
_agent: false,
method: 'GET',
url: 'https://mydomain.com/api/method/frappe.realtime.can_subscribe_doc?sid=a36076b007b0d9c8429973781b18ddfd7e91497796f77d40c9aa43dc&doctype=Payment%20Entry&docname=new-payment-entry-1',
_header: [Object],
header: [Object],
writable: true,
_redirects: 1,
_maxRedirects: 5,
cookies: '',
qs: {},
_query: [],
qsRaw: [],
_redirectList: [Array],
_streamRequest: false,
protocol: 'https:',
host: 'mydomain.com',
_endCalled: true,
_callback: [Function (anonymous)],
res: [IncomingMessage],
_data: null,
req: [ClientRequest],
_resBuffered: true,
response: [Circular *1],
called: true,
[Symbol(kCapture)]: false
},
req: ClientRequest {
_events: [Object: null prototype],
_eventsCount: 3,
_maxListeners: undefined,
outputData: [],
outputSize: 0,
writable: true,
destroyed: false,
_last: true,
chunkedEncoding: false,
shouldKeepAlive: false,
maxRequestsOnConnectionReached: false,
_defaultKeepAlive: true,
useChunkedEncodingByDefault: false,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: 0,
_hasBody: true,
_trailer: '',
finished: true,
_headerSent: true,
_closed: false,
socket: [TLSSocket],
_header: 'GET /api/method/frappe.realtime.can_subscribe_doc?sid=a36076b007b0d9c8429973781b18ddfd7e91497796f77d40c9aa43dc&doctype=Payment%20Entry&docname=new-payment-entry-1 HTTP/1.1\r\n' +
'Host: mydomain.com\r\n' +
'accept-encoding: gzip, deflate\r\n' +
'Connection: close\r\n' +
'\r\n',
_keepAliveTimeout: 0,
_onPendingData: [Function: nop],
agent: [Agent],
socketPath: undefined,
method: 'GET',
maxHeaderSize: undefined,
insecureHTTPParser: undefined,
path: '/api/method/frappe.realtime.can_subscribe_doc?sid=a36076b007b0d9c8429973781b18ddfd7e91497796f77d40c9aa43dc&doctype=Payment%20Entry&docname=new-payment-entry-1',
_ended: true,
res: [IncomingMessage],
aborted: false,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
reusedSocket: false,
host: 'mydomain.com',
protocol: 'https:',
[Symbol(kCapture)]: false,
[Symbol(kNeedDrain)]: false,
[Symbol(corked)]: 0,
[Symbol(kOutHeaders)]: [Object: null prototype]
},
text: '{"exc_type":"DoesNotExistError","_server_messages":"[\\"{\\\\\\"message\\\\\\": \\\\\\"Payment Entry new-payment-entry-1 not found\\\\\\", \\\\\\"title\\\\\\": \\\\\\"Message\\\\\\", \\\\\\"indicator\\\\\\": \\\\\\"red\\\\\\", \\\\\\"raise_exception\\\\\\": 1}\\"]"}',
body: {
exc_type: 'DoesNotExistError',
_server_messages: '["{\\"message\\": \\"Payment Entry new-payment-entry-1 not found\\", \\"title\\": \\"Message\\", \\"indicator\\": \\"red\\", \\"raise_exception\\": 1}"]'
},
files: undefined,
buffered: true,
headers: {
'content-length': '222',
'content-type': 'application/json',
date: 'Tue, 05 Jul 2022 04:51:27 GMT',
server: 'nginx/1.23.0',
'set-cookie': [Array],
connection: 'close'
},
header: {
'content-length': '222',
'content-type': 'application/json',
date: 'Tue, 05 Jul 2022 04:51:27 GMT',
server: 'nginx/1.23.0',
'set-cookie': [Array],
connection: 'close'
},
statusCode: 404,
status: 404,
statusType: 4,
info: false,
ok: false,
redirect: false,
clientError: true,
serverError: false,
error: Error: cannot GET /api/method/frappe.realtime.can_subscribe_doc?sid=a36076b007b0d9c8429973781b18ddfd7e91497796f77d40c9aa43dc&doctype=Payment%20Entry&docname=new-payment-entry-1 (404)
at Response.toError (/home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/node/response.js:98:13)
at ResponseBase._setStatusProperties (/home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/response-base.js:119:48)
at new Response (/home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/node/response.js:44:8)
at Request._emitResponse (/home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/node/index.js:930:18)
at /home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/node/index.js:1126:42
at IncomingMessage.<anonymous> (/home/frappe/frappe-bench/apps/frappe/node_modules/superagent/lib/node/parsers/json.js:22:7)
at IncomingMessage.emit (node:events:539:35)
at endReadableNT (node:internal/streams/readable:1345:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
status: 404,
text: '{"exc_type":"DoesNotExistError","_server_messages":"[\\"{\\\\\\"message\\\\\\": \\\\\\"Payment Entry new-payment-entry-1 not found\\\\\\", \\\\\\"title\\\\\\": \\\\\\"Message\\\\\\", \\\\\\"indicator\\\\\\": \\\\\\"red\\\\\\", \\\\\\"raise_exception\\\\\\": 1}\\"]"}',
method: 'GET',
path: '/api/method/frappe.realtime.can_subscribe_doc?sid=a36076b007b0d9c8429973781b18ddfd7e91497796f77d40c9aa43dc&doctype=Payment%20Entry&docname=new-payment-entry-1'
},
created: false,
accepted: false,
noContent: false,
badRequest: false,
unauthorized: false,
notAcceptable: false,
forbidden: false,
notFound: true,
unprocessableEntity: false,
type: 'application/json',
links: {},
setEncoding: [Function: bound ],
redirects: [
'https://mydomain.com/api/method/frappe.realtime.can_subscribe_doc?sid=a36076b007b0d9c8429973781b18ddfd7e91497796f77d40c9aa43dc&doctype=Payment%20Entry&docname=new-payment-entry-1'
],
pipe: [Function (anonymous)],
[Symbol(kCapture)]: false
}
}
^C
backend looks normal
# docker logs -f erpnext-one-backend-1
[2022-07-05 04:44:49 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2022-07-05 04:44:49 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2022-07-05 04:44:49 +0000] [1] [INFO] Using worker: gthread
[2022-07-05 04:44:49 +0000] [8] [INFO] Booting worker with pid: 8
[2022-07-05 04:44:49 +0000] [9] [INFO] Booting worker with pid: 9
^C
queue-default looks normal to me:
04:56:03 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.crm.doctype.social_media_post.social_media_post.proce..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (5b323ba9-3f45-41ee-8b54-28d267db6070)
04:56:04 default: Job OK (5b323ba9-3f45-41ee-8b54-28d267db6070)
04:56:04 Result is kept for 500 seconds
04:56:04 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.healthcare.doctype.patient_appointment.patient_appoin..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (38e4d79b-f319-4d3b-abf7-806bed0b99fc)
04:56:04 default: Job OK (38e4d79b-f319-4d3b-abf7-806bed0b99fc)
04:56:04 Result is kept for 500 seconds
04:56:04 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.projects.doctype.project.project.project_status_updat..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (62e4e8f6-c78b-4426-acd8-ea1cfab75caa)
04:56:05 default: Job OK (62e4e8f6-c78b-4426-acd8-ea1cfab75caa)
04:56:05 Result is kept for 500 seconds
04:56:05 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.monitor.flush'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (e9a5283d-481b-40de-b602-12c0cb58be05)
04:56:06 default: Job OK (e9a5283d-481b-40de-b602-12c0cb58be05)
04:56:06 Result is kept for 500 seconds
04:56:06 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.utils.global_search.sync_global_search'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (1b20225a-2b93-428b-814f-7f9dbc8fbc96)
04:56:06 default: Job OK (1b20225a-2b93-428b-814f-7f9dbc8fbc96)
04:56:06 Result is kept for 500 seconds
04:56:06 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.integrations.doctype.razorpay_settings.razorpay_settin..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (287c8711-dc81-49d0-9047-5da35ec372cc)
04:56:07 default: Job OK (287c8711-dc81-49d0-9047-5da35ec372cc)
04:56:07 Result is kept for 500 seconds
04:56:07 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.email.doctype.email_account.email_account.notify_unrep..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (eab42aea-4732-41ca-9d3e-f12bdbd608fd)
04:56:08 default: Job OK (eab42aea-4732-41ca-9d3e-f12bdbd608fd)
04:56:08 Result is kept for 500 seconds
04:56:08 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.email.doctype.email_account.email_account.pull'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (b3e726c3-4797-4d4f-bba2-f800e011b703)
04:56:09 default: Job OK (b3e726c3-4797-4d4f-bba2-f800e011b703)
04:56:09 Result is kept for 500 seconds
04:56:09 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.email.queue.flush'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (54be4fef-9af4-445d-802b-fe90e67332bd)
04:56:10 default: Job OK (54be4fef-9af4-445d-802b-fe90e67332bd)
04:56:10 Result is kept for 500 seconds
05:00:06 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.manufacturing.doctype.bom_update_log.bom_update_log.r..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (2d9313c9-ada5-4497-999a-33b0b2240d98)
05:00:09 default: Job OK (2d9313c9-ada5-4497-999a-33b0b2240d98)
05:00:09 Result is kept for 500 seconds
05:00:09 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.hr.doctype.interview.interview.send_interview_reminde..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (3d9121fd-7362-44a3-b379-b78b92e4edf0)
05:00:10 default: Job OK (3d9121fd-7362-44a3-b379-b78b92e4edf0)
05:00:10 Result is kept for 500 seconds
05:00:10 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.erpnext_integrations.connectors.shopify_connection.sy..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (57f6648a-1979-4c35-8d5f-168b875427b9)
05:00:11 default: Job OK (57f6648a-1979-4c35-8d5f-168b875427b9)
05:00:11 Result is kept for 500 seconds
05:00:11 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.support.doctype.issue.issue.set_service_level_agreeme..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (98f08392-8677-42c1-8773-9464ef27fcba)
05:00:12 default: Job OK (98f08392-8677-42c1-8773-9464ef27fcba)
05:00:12 Result is kept for 500 seconds
05:00:12 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.hr.doctype.shift_type.shift_type.process_auto_attenda..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (8d1359fd-a74a-4351-821c-4bd8f599fab7)
05:00:12 default: Job OK (8d1359fd-a74a-4351-821c-4bd8f599fab7)
05:00:12 Result is kept for 500 seconds
05:00:12 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.projects.doctype.project.project.collect_project_stat..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (db6bc22f-e461-4491-85e4-fbfc51e7ca8b)
05:00:13 default: Job OK (db6bc22f-e461-4491-85e4-fbfc51e7ca8b)
05:00:13 Result is kept for 500 seconds
05:00:13 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.projects.doctype.project.project.hourly_reminder'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (750f6c59-4d25-4e7e-b49c-207050773263)
05:00:14 default: Job OK (750f6c59-4d25-4e7e-b49c-207050773263)
05:00:14 Result is kept for 500 seconds
05:00:14 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.erpnext_integrations.doctype.plaid_settings.plaid_set..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (de8b4ef5-022f-47b8-84f2-8f44aad84e9c)
05:00:16 default: Job OK (de8b4ef5-022f-47b8-84f2-8f44aad84e9c)
05:00:16 Result is kept for 500 seconds
05:00:16 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.accounts.doctype.gl_entry.gl_entry.rename_gle_sle_doc..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (4659c878-83e0-437c-bb6d-f39923749dce)
05:00:17 default: Job OK (4659c878-83e0-437c-bb6d-f39923749dce)
05:00:17 Result is kept for 500 seconds
05:00:17 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.erpnext_integrations.doctype.amazon_mws_settings.amaz..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (aa59e81b-10c8-4ca1-ab4d-1bef0357df32)
05:00:18 default: Job OK (aa59e81b-10c8-4ca1-ab4d-1bef0357df32)
05:00:18 Result is kept for 500 seconds
05:00:18 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.accounts.doctype.subscription.subscription.process_al..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (cb468a81-f137-4df8-b2d6-504159bed586)
05:00:19 default: Job OK (cb468a81-f137-4df8-b2d6-504159bed586)
05:00:19 Result is kept for 500 seconds
05:00:19 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.hr.doctype.daily_work_summary_group.daily_work_summar..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (d336c72b-fb41-4378-a1c5-6356ec36e034)
05:00:20 default: Job OK (d336c72b-fb41-4378-a1c5-6356ec36e034)
05:00:20 Result is kept for 500 seconds
05:00:20 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.email.doctype.newsletter.newsletter.send_scheduled_ema..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (a82ec4e5-a88a-4c78-a16a-499f01464cda)
05:00:21 default: Job OK (a82ec4e5-a88a-4c78-a16a-499f01464cda)
05:00:21 Result is kept for 500 seconds
05:00:21 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.integrations.doctype.google_calendar.google_calendar.s..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (ea3e98a3-7e07-4c4e-b0a7-e644e8a1180d)
05:00:22 default: Job OK (ea3e98a3-7e07-4c4e-b0a7-e644e8a1180d)
05:00:22 Result is kept for 500 seconds
05:00:22 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.desk.form.document_follow.send_hourly_updates'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (b86cebaa-4b38-40d3-8287-518df53cd0ec)
05:00:23 default: Job OK (b86cebaa-4b38-40d3-8287-518df53cd0ec)
05:00:23 Result is kept for 500 seconds
05:00:23 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.deferred_insert.save_to_db'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (80d6a7a3-817b-42ad-9960-7f474a67cec7)
05:00:24 default: Job OK (80d6a7a3-817b-42ad-9960-7f474a67cec7)
05:00:24 Result is kept for 500 seconds
05:00:24 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.desk.page.backups.backups.delete_downloadable_backups'..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (756cb2a9-cb86-4a9d-a392-2bebc3588388)
05:00:25 default: Job OK (756cb2a9-cb86-4a9d-a392-2bebc3588388)
05:00:25 Result is kept for 500 seconds
05:00:25 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.utils.error.collect_error_snapshots'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (49224a18-db85-4029-bde7-6494d2adea7b)
05:00:26 default: Job OK (49224a18-db85-4029-bde7-6494d2adea7b)
05:00:26 Result is kept for 500 seconds
05:00:26 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.model.utils.user_settings.sync_user_settings'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (a43188d4-b454-4f09-8089-a9da0fc46b9f)
05:00:27 default: Job OK (a43188d4-b454-4f09-8089-a9da0fc46b9f)
05:00:27 Result is kept for 500 seconds
05:00:27 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.model.utils.link_count.update_link_count'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (78b14e65-a767-4425-8bcc-df15088c95b2)
05:00:27 default: Job OK (78b14e65-a767-4425-8bcc-df15088c95b2)
05:00:27 Result is kept for 500 seconds
05:00:27 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.crm.doctype.social_media_post.social_media_post.proce..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (093ab3eb-bc29-4135-87d8-bcb76c1e5b03)
05:00:28 default: Job OK (093ab3eb-bc29-4135-87d8-bcb76c1e5b03)
05:00:28 Result is kept for 500 seconds
05:00:28 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.healthcare.doctype.patient_appointment.patient_appoin..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (05e10c62-f473-4875-94e7-675fced17c1d)
05:00:28 default: Job OK (05e10c62-f473-4875-94e7-675fced17c1d)
05:00:28 Result is kept for 500 seconds
05:00:28 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.projects.doctype.project.project.project_status_updat..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (1ec6c250-0714-4c14-b9d1-98816744198f)
05:00:29 default: Job OK (1ec6c250-0714-4c14-b9d1-98816744198f)
05:00:29 Result is kept for 500 seconds
05:00:29 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.monitor.flush'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (9d2d3446-93af-4b05-87fe-dca0a28ff8ee)
05:00:30 default: Job OK (9d2d3446-93af-4b05-87fe-dca0a28ff8ee)
05:00:30 Result is kept for 500 seconds
05:00:30 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.utils.global_search.sync_global_search'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (bebabd63-d644-48cd-a07c-f3a749c76a0f)
05:00:30 default: Job OK (bebabd63-d644-48cd-a07c-f3a749c76a0f)
05:00:30 Result is kept for 500 seconds
05:00:30 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.integrations.doctype.razorpay_settings.razorpay_settin..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (ff3465f5-c632-41aa-824e-aa5a632eda5f)
05:00:31 default: Job OK (ff3465f5-c632-41aa-824e-aa5a632eda5f)
05:00:31 Result is kept for 500 seconds
05:00:31 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.email.doctype.email_account.email_account.notify_unrep..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (171850d6-3540-4dd9-95cc-8afd4e3fb20e)
05:00:31 default: Job OK (171850d6-3540-4dd9-95cc-8afd4e3fb20e)
05:00:31 Result is kept for 500 seconds
05:00:31 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.email.doctype.email_account.email_account.pull'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (4a53160b-e4c7-410e-889b-8dc4af2c0086)
05:00:32 default: Job OK (4a53160b-e4c7-410e-889b-8dc4af2c0086)
05:00:32 Result is kept for 500 seconds
05:00:32 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.email.queue.flush'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (0ffb0220-ca40-4726-ace9-97e7a3939cd5)
05:00:33 default: Job OK (0ffb0220-ca40-4726-ace9-97e7a3939cd5)
05:00:33 Result is kept for 500 seconds
05:00:33 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'erpnext.utilities.doctype.video.video.update_youtube_data'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (a6b14539-1012-4a8d-8476-c5c5ffe49e50)
05:00:34 default: Job OK (a6b14539-1012-4a8d-8476-c5c5ffe49e50)
05:00:34 Result is kept for 500 seconds
05:00:34 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.twofactor.delete_all_barcodes_for_users'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (f855a3e8-1d48-4410-8b54-11cfbd5ae620)
05:00:35 default: Job OK (f855a3e8-1d48-4410-8b54-11cfbd5ae620)
05:00:35 Result is kept for 500 seconds
05:00:35 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.website.doctype.web_page.web_page.check_publish_status..., method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (6b751408-9fdb-4988-9493-1abb3d605261)
05:00:36 default: Job OK (6b751408-9fdb-4988-9493-1abb3d605261)
05:00:36 Result is kept for 500 seconds
05:00:36 default: frappe.utils.background_jobs.execute_job(event=None, is_async=True, job_name='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., kwargs={'job_type': 'frappe.oauth.delete_oauth2_data'}, method='frappe.core.doctype.scheduled_job_type.scheduled_job_type.run_scheduled_jo..., site='mydomain.com', user='Administrator') (38449372-e81e-40cc-84b4-52d2e2979000)
05:00:37 default: Job OK (38449372-e81e-40cc-84b4-52d2e2979000)
05:00:37 Result is kept for 500 seconds
^C
queue-short has no activity
redis looks normal to me:
1:M 05 Jul 2022 04:54:30.154 * Background saving terminated with success
1:M 05 Jul 2022 04:59:31.049 * 100 changes in 300 seconds. Saving...
1:M 05 Jul 2022 04:59:31.053 * Background saving started by pid 18
18:C 05 Jul 2022 04:59:31.153 * DB saved on disk
18:C 05 Jul 2022 04:59:31.155 * RDB: 0 MB of memory used by copy-on-write
1:M 05 Jul 2022 04:59:31.254 * Background saving terminated with success
^C
traefik looks normal, mostly telling me that there are some unseen error logs
