where is your custom script (Python, I assume)? Be aware that in ERPNext, custom script refers to JavaScript inserts that can be attached to DocTypes. If you run the console (or in your case, maybe bench execute might even be better), you will be executing server-side Python code.
Assuming your app is apps/yourapp, then your code might be e.g. located in apps/yourapp/yourapp/yourapp/doctype/yourdoctype/yourdoctype.py and might be function def do_stuff. Note that the path is apps/(repo name)/(app name)/(module name)/doctype/(doctype name)/(doctype name).py. So you could run it with
The directory in which you are working also plays a major role. You should be currently in your custom app directory i think.
Thank you @Tropicalrambler and @lasalesi for your timely response. This is one stop solution for those who want to make trial and error to run their custom python scripts via console.