Django是一款经典的Python Web开发框架，也是最受欢迎的Python开源项目之一。不同于Flask框架，Django是高度集成的，可以帮助开发者快速搭建一个Web项目。 从本周开始，我们一起进入Djaong项目的源码解析，加深对django的理解，熟练掌握python web开发。django采用的源码版本是: 4.0.0，我们首先采用概读法，大概了解django项目的创建和启动过程，包括下面几个部分:
- django简单回顾
- django项目概况
- django命令脚手架
- startproject和startapp命令实现
- runserver命令实现
- app的setup过程
- 小结
- 小技巧
django简单回顾
创建好虚拟环境，安装django的 4.0.0 版本。这个版本和官方的最新文档一致，而且有完整的中文翻译版本。我们可以跟随「快速安装指南」创建一个django项目，感受一下其魅力。首先使用 startproject 命令创建一个名叫 hello 的项目，django会在本地搭建一个基础的项目结构，进入hello项目后，可以直接使用 runserver 命令启动项目。
1
2
|
python3 -m django startproject hello
cd hello && python3 manage.py runserver
|
django提供很好的模块化支持，可以利用它做大型web项目的开发。在project下的模块称为app，一个project可以包括多个app模块, 和flask的blue-print类似。我们继续使用 startapp 命令来创建一个叫做 api 的app。
1
|
python3 -m django startapp api
|
对api-app的 views.py，我们需要完善一下，增加下面内容：
1
2
3
4
5
6
7
8
9
|
from django.shortcuts import render
# Create your views here.
from django.http import HttpResponse
def index(request):
return HttpResponse("Hello, Game404. You're at the index.")
|
然后再创建一个urls.py的模块文件，定义url和view的映射关系，填写:
1
2
3
4
5
6
7
|
from django.urls import path
from . import views
urlpatterns = [
path('', views.index, name='index'),
]
|
完成api-app的实现后，我们需要在project中添加上自定义的api模块。这需要两步，先是在hello-project的setting.py中配置这个api-app:
1
2
3
4
5
6
7
8
9
|
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'api',
]
|
再在hello-project的urls.py导入api-app中定义的url和视图:
1
2
3
4
5
6
7
|
from django.contrib import admin
from django.urls import path, include
urlpatterns = [
path('admin/', admin.site.urls),
path('api/', include('api.urls')),
]
|
剩下的内容，都可以使用模版的标准实现。然后我们再次启动项目，访问下面路径:
1
2
|
➜ ~ curl http://127.0.0.1:8000/api/
Hello, Game404. You're at the index.%
|
我们基本完成了一个最简单的django项目创建和启动，接下来我们一起了解这个流程是如何实现的。
django项目概况
django项目源码大概包括下面一些包:
| 包 |
功能描述 |
| apps |
django的app管理器 |
| conf |
配置信息，主要有项目模版和app模版等 |
| contrib |
django默认提供的标准app |
| core |
django核心功能 |
| db |
数据库模型实现 |
| dispatch |
信号，用于模块解耦 |
| forms |
表单实现 |
| http |
http协议和服务相关实现 |
| middleware |
django提供的标准中间件 |
| template && templatetags |
模版功能 |
| test |
单元测试的支持 |
| urls |
一些url的处理类 |
| utils |
工具类 |
| views |
视图相关的实现 |
我也简单对比了一下django和flask的代码情况:
1
2
3
4
5
|
-------------------------------------------------------------------------------
Project files blank comment code
-------------------------------------------------------------------------------
django 716 19001 25416 87825
flask 20 1611 3158 3587
|
仅从文件数量和代码行数可以看到，django是一个庞大的框架，不同于flask是一个简易框架，有700多个模块文件和近9万行代码。在阅读flask源码前，我们还需要先了解sqlalchemy，werkzeug等依赖框架，从pyproject.toml中的依赖项可发现，django默认就没有其它依赖，可以直接开始。这有点类似ios和Android系统，flask需要各种插件的支持，更为开放；django则是全部集成，使用默认的框架就已经可以处理绝大部分web开放需求了。
django命令脚手架
django提供了一系列脚手架命令，协助开发者创建和管理django项目。可以使用 help 参数查看命令清单:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
python3 -m django --help
Type 'python -m django help <subcommand>' for help on a specific subcommand.
Available subcommands:
[django]
check
compilemessages
createcachetable
dbshell
diffsettings
dumpdata
flush
inspectdb
loaddata
makemessages
makemigrations
migrate
runserver
sendtestemail
shell
showmigrations
sqlflush
sqlmigrate
sqlsequencereset
squashmigrations
startapp
startproject
test
testserver
|
前面使用到的 startproject ， startapp 和 runserver 三个命令是本篇重点介绍的命令，其它的二十多个命令我们在使用到的时候再行介绍。这是 概读法 的精髓，只关注主干，先建立全局视野，再逐步深入。
django模块的main函数在 __main__.py 中提供:
1
2
3
4
5
6
7
8
9
|
"""
Invokes django-admin when the django module is run as a script.
Example: python -m django check
"""
from django.core import management
if __name__ == "__main__":
management.execute_from_command_line()
|
可以看到 django.core.management 模块提供脚手架的功能实现。同样在project的 manager.py 中也是通过调用management模块来实现项目启动:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hello.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
...
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
|
ManagementUtility的主要结构如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
class ManagementUtility:
"""
Encapsulate the logic of the django-admin and manage.py utilities.
"""
def __init__(self, argv=None):
# 命令行参数
self.argv = argv or sys.argv[:]
self.prog_name = os.path.basename(self.argv[0])
if self.prog_name == '__main__.py':
self.prog_name = 'python -m django'
self.settings_exception = None
def main_help_text(self, commands_only=False):
"""Return the script's main help text, as a string."""
pass
def fetch_command(self, subcommand):
"""
Try to fetch the given subcommand, printing a message with the
appropriate command called from the command line (usually
"django-admin" or "manage.py") if it can't be found.
"""
pass
...
def execute(self):
"""
Given the command-line arguments, figure out which subcommand is being
run, create a parser appropriate to that command, and run it.
"""
pass
|
- init函数解析argv的命令行参数
- main_help_text 输出命令的帮助信息
- fetch_command 查找django模块的子命令
- execute 执行子命令
一个好的命令行工具，离不开清晰的帮助输出。默认的帮助信息是调用main_help_text函数:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
def main_help_text(self, commands_only=False):
"""Return the script's main help text, as a string."""
usage = [
"",
"Type '%s help <subcommand>' for help on a specific subcommand." % self.prog_name,
"",
"Available subcommands:",
]
commands_dict = defaultdict(lambda: [])
for name, app in get_commands().items():
commands_dict[app].append(name)
style = color_style()
for app in sorted(commands_dict):
usage.append("")
usage.append(style.NOTICE("[%s]" % app))
for name in sorted(commands_dict[app]):
usage.append(" %s" % name)
# Output an extra note if settings are not properly configured
if self.settings_exception is not None:
usage.append(style.NOTICE(
"Note that only Django core commands are listed "
"as settings are not properly configured (error: %s)."
% self.settings_exception))
return '\n'.join(usage)
|
main_help_text主要利用了下面4个函数去查找命令清单:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
def find_commands(management_dir):
"""
Given a path to a management directory, return a list of all the command
names that are available.
"""
command_dir = os.path.join(management_dir, 'commands')
return [name for _, name, is_pkg in pkgutil.iter_modules([command_dir])
if not is_pkg and not name.startswith('_')]
def load_command_class(app_name, name):
"""
Given a command name and an application name, return the Command
class instance. Allow all errors raised by the import process
(ImportError, AttributeError) to propagate.
"""
module = import_module('%s.management.commands.%s' % (app_name, name))
return module.Command()
def get_commands():
commands = {name: 'django.core' for name in find_commands(__path__[0])}
if not settings.configured:
return commands
for app_config in reversed(list(apps.get_app_configs())):
path = os.path.join(app_config.path, 'management')
commands.update({name: app_config.name for name in find_commands(path)})
return commands
def call_command(command_name, *args, **options):
pass
|
find_commands查找management/commands下的命令文件，load_command_class使用 import_module 这个动态加载模块的方法导入命令。
子命令的执行是通过fetch_command找到子命令，然后执行子命令的run_from_argv方法。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
def execute(self):
...
self.fetch_command(subcommand).run_from_argv(self.argv)
def fetch_command(self, subcommand):
"""
Try to fetch the given subcommand, printing a message with the
appropriate command called from the command line (usually
"django-admin" or "manage.py") if it can't be found.
"""
# Get commands outside of try block to prevent swallowing exceptions
commands = get_commands()
try:
app_name = commands[subcommand]
except KeyError:
...
if isinstance(app_name, BaseCommand):
# If the command is already loaded, use it directly.
klass = app_name
else:
klass = load_command_class(app_name, subcommand)
return klass
|
在django的core中包含了下面这些命令，多数都直接继承自BaseCommand:
BaseCommand的主要代码结构:
最重要的run_from_argv和execute方法都是命令的执行入口:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
def run_from_argv(self, argv):
...
parser = self.create_parser(argv[0], argv[1])
options = parser.parse_args(argv[2:])
cmd_options = vars(options)
# Move positional args out of options to mimic legacy optparse
args = cmd_options.pop('args', ())
handle_default_options(options)
try:
self.execute(*args, **cmd_options)
except CommandError as e:
...
def execute(self, *args, **options):
output = self.handle(*args, **options)
return output
|
handler方法则留给子类覆盖实现:
1
2
3
4
5
6
|
def handle(self, *args, **options):
"""
The actual logic of the command. Subclasses must implement
this method.
"""
raise NotImplementedError('subclasses of BaseCommand must provide a handle() method')
|
startproject && startapp 命令实现
startproject和startapp两个命令分别创建项目和app，都派生自TemplateCommand。主要功能实现都在TemplateCommand中。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
# startproject
def handle(self, **options):
project_name = options.pop('name')
target = options.pop('directory')
# Create a random SECRET_KEY to put it in the main settings.
options['secret_key'] = SECRET_KEY_INSECURE_PREFIX + get_random_secret_key()
super().handle('project', project_name, target, **options)
# startapp
def handle(self, **options):
app_name = options.pop('name')
target = options.pop('directory')
super().handle('app', app_name, target, **options)
|
我们使用startproject命令后的项目结构大概如下:
1
2
3
4
5
6
7
|
├── hello
│ ├── __init__.py
│ ├── asgi.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
└── manage.py
|
这和conf下的project_template目录中的模版文件一致：
1
2
3
4
5
6
7
8
|
.
├── manage.py-tpl
└── project_name
├── __init__.py-tpl
├── asgi.py-tpl
├── settings.py-tpl
├── urls.py-tpl
└── wsgi.py-tpl
|
manage.py-tpl模版文件内容:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
"""Run administrative tasks."""
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '{{ project_name }}.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
....
if __name__ == '__main__':
main()
|
可见startproject命令的功能是接收开发者输入的project_name，然后渲染到模版文件中，再生成项目文件。
TemplateCommand中模版处理的函数主要内容如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
from django.template import Context, Engine
def handle(self, app_or_project, name, target=None, **options):
...
base_name = '%s_name' % app_or_project
base_subdir = '%s_template' % app_or_project
base_directory = '%s_directory' % app_or_project
camel_case_name = 'camel_case_%s_name' % app_or_project
camel_case_value = ''.join(x for x in name.title() if x != '_')
...
context = Context({
**options,
base_name: name,
base_directory: top_dir,
camel_case_name: camel_case_value,
'docs_version': get_docs_version(),
'django_version': django.__version__,
}, autoescape=False)
...
template_dir = self.handle_template(options['template'],
base_subdir)
...
for root, dirs, files in os.walk(template_dir):
for filename in files:
if new_path.endswith(extensions) or filename in extra_files:
with open(old_path, encoding='utf-8') as template_file:
content = template_file.read()
template = Engine().from_string(content)
content = template.render(context)
with open(new_path, 'w', encoding='utf-8') as new_file:
new_file.write(content)
|
- 构建模版参数使用的Context上下文
- 遍历模版文件目录
- 使用django的模版引擎Engine渲染内容
- 用渲染的结果生成项目脚手架文件
django.template.Engine如何工作，和模版引擎mako有什么区别，以后再行介绍，本章我们只需要了解即可。
runserver 命令实现
runserver提供了一个开发测试的http服务，帮助启动django项目，也是django使用频率最高的命令。django项目遵循wsgi规范，执行启动之前需要先查找wsgi-application。(如果对wsgi规范不了解的同学，欢迎翻看之前的文章)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
def get_internal_wsgi_application():
"""
Load and return the WSGI application as configured by the user in
``settings.WSGI_APPLICATION``. With the default ``startproject`` layout,
this will be the ``application`` object in ``projectname/wsgi.py``.
This function, and the ``WSGI_APPLICATION`` setting itself, are only useful
for Django's internal server (runserver); external WSGI servers should just
be configured to point to the correct application object directly.
If settings.WSGI_APPLICATION is not set (is ``None``), return
whatever ``django.core.wsgi.get_wsgi_application`` returns.
"""
from django.conf import settings
app_path = getattr(settings, 'WSGI_APPLICATION')
if app_path is None:
return get_wsgi_application()
try:
return import_string(app_path)
except ImportError as err:
...
|
结合注释可以知道这里会载入开发者在project的settings中定义的application:
1
|
WSGI_APPLICATION = 'hello.wsgi.application'
|
默认情况下，自定义的wsgi-application是这样的:
1
2
3
4
5
6
7
|
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hello.settings')
application = get_wsgi_application()
|
继续查看runserver的执行, 主要是 inner_run 函数:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
from django.core.servers.basehttp import run
def inner_run(self, *args, **options):
try:
handler = self.get_handler(*args, **options)
run(self.addr, int(self.port), handler,
ipv6=self.use_ipv6, threading=threading, server_cls=self.server_cls)
except OSError as e:
...
# Need to use an OS exit because sys.exit doesn't work in a thread
os._exit(1)
except KeyboardInterrupt:
..
sys.exit(0)
|
http服务的功能实现由 django.core.servers.basehttp 提供，完成http服务和wsgi之间的衔接，其架构图如下:
run函数的代码:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
def run(addr, port, wsgi_handler, ipv6=False, threading=False, server_cls=WSGIServer):
server_address = (addr, port)
if threading:
httpd_cls = type('WSGIServer', (socketserver.ThreadingMixIn, server_cls), {})
else:
httpd_cls = server_cls
httpd = httpd_cls(server_address, WSGIRequestHandler, ipv6=ipv6)
if threading:
# ThreadingMixIn.daemon_threads indicates how threads will behave on an
# abrupt shutdown; like quitting the server by the user or restarting
# by the auto-reloader. True means the server will not wait for thread
# termination before it quits. This will make auto-reloader faster
# and will prevent the need to kill the server manually if a thread
# isn't terminating correctly.
httpd.daemon_threads = True
httpd.set_app(wsgi_handler)
httpd.serve_forever()
|
- 如果支持多线程，则创建一个WSGIServer和ThreadingMixIn的新类
- 创建http服务并启动
django的wsgi-application实现, http协议的实现，下一章再行详细介绍，我们也暂时跳过。在runserver中还有一个非常重要的功能: 自动重启服务 。 如果我们修改了项目的代码，服务会自动重启，可以提高开发效率。
比如我们修改api的视图功能，随便增加几个字符，可以在控制台看到大概下面的输出:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
# python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
March 05, 2022 - 09:06:07
Django version 4.0, using settings 'hello.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
/Users/yoo/tmp/django/hello/api/views.py changed, reloading.
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
March 05, 2022 - 09:12:59
Django version 4.0, using settings 'hello.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
|
runserver命令检测到 /hello/api/views.py 有修改，然后自动使用StatReloader重启服务。
inner_run有下面2中启动方式, 默认情况下使用autoreload启动:
1
2
3
4
5
6
7
8
9
|
def run(self, **options):
"""Run the server, using the autoreloader if needed."""
...
use_reloader = options['use_reloader']
if use_reloader:
autoreload.run_with_reloader(self.inner_run, **options)
else:
self.inner_run(None, **options)
|
autoreload的继承关系如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
class BaseReloader:
pass
class StatReloader(BaseReloader):
pass
class WatchmanReloader(BaseReloader):
pass
def get_reloader():
"""Return the most suitable reloader for this environment."""
try:
WatchmanReloader.check_availability()
except WatchmanUnavailable:
return StatReloader()
return WatchmanReloader()
|
当前版本优先使用WatchmanReloader实现，这依赖于pywatchman库，需要额外安装。否则使用StatReloader的实现，这个实现在之前介绍werkzeug中也有过介绍，本质上都是持续的监听文件的状态变化。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
class StatReloader(BaseReloader):
SLEEP_TIME = 1 # Check for changes once per second.
def tick(self):
mtimes = {}
while True:
for filepath, mtime in self.snapshot_files():
old_time = mtimes.get(filepath)
mtimes[filepath] = mtime
if old_time is None:
logger.debug('File %s first seen with mtime %s', filepath, mtime)
continue
elif mtime > old_time:
logger.debug('File %s previous mtime: %s, current mtime: %s', filepath, old_time, mtime)
self.notify_file_changed(filepath)
time.sleep(self.SLEEP_TIME)
yield
|
- StatReloader 以1s为间隔检查文件的时间戳变化。
- tick是生成器模式，可以使用next持续调用
当文件有变化后，退出当前进程，并使用subprocess启动新的进程:
1
2
3
4
5
6
7
8
9
10
11
|
def trigger_reload(filename):
logger.info('%s changed, reloading.', filename)
sys.exit(3)
def restart_with_reloader():
new_environ = {**os.environ, DJANGO_AUTORELOAD_ENV: 'true'}
args = get_child_arguments()
while True:
p = subprocess.run(args, env=new_environ, close_fds=False)
if p.returncode != 3:
return p.returncode
|
app的setup过程
runserver命令在执行之前还有很重要的一步就是setup: 加载和初始化开发者自定义的app内容。这是在ManagementUtility的execute函数中开始的:
1
2
3
4
5
6
7
8
9
|
def execute(self):
...
try:
settings.INSTALLED_APPS
except ImproperlyConfigured as exc:
self.settings_exception = exc
except ImportError as exc:
self.settings_exception = exc
...
|
Settings中会载下面的一些模块，主要是INSTALLED_APPS:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
class Settings:
def __init__(self, settings_module):
...
# store the settings module in case someone later cares
self.SETTINGS_MODULE = settings_module
mod = importlib.import_module(self.SETTINGS_MODULE)
tuple_settings = (
'ALLOWED_HOSTS',
"INSTALLED_APPS",
"TEMPLATE_DIRS",
"LOCALE_PATHS",
)
self._explicit_settings = set()
...
|
INSTALLED_APPS在项目的setting中定义:
1
2
3
4
5
6
7
8
9
10
11
|
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'api',
]
|
这样django框架就完成了开发者自定义的内容动态加载。
小结
Django是一个高度集成的python web开发框架，支持模块化开发。django还提供了一系列脚手架命令，比如使用startproject和startapp协助创建项目和模块模版；使用runserver辅助测试和开发项目。django项目也符合wsgi规范，其http服务的启动中创建了WSGIServer，并且支持多线程模式。django作为一个框架，可以通过约定的配置文件setting动态加载开发者的业务实现。
小技巧
django命令支持智能提示。比如我们 runserver 命令时，不小心输入错误的把字母 u 打成了 i。命令会自动提醒我们，是不是想使用 runserver 命令:
1
2
3
4
|
python -m django rinserver
No Django settings specified.
Unknown command: 'rinserver'. Did you mean runserver?
Type 'python -m django help' for usage.
|
智能提示的功能对命令行工具很有帮助，一般的实现就是比较用户输入和已知命令的重合度，从而找到最接近的命令。这是「字符串编辑距离」算法的实际应用。个人认为理解场景，这会比死刷算法更有用，我偶尔会在面试的时候使用这个例子来观测面试人的算法水准。django这里直接使用python标准库difflib提供的实现:
1
2
3
4
5
6
|
from difflib import get_close_matches
possible_matches = get_close_matches(subcommand, commands)
sys.stderr.write('Unknown command: %r' % subcommand)
if possible_matches:
sys.stderr.write('. Did you mean %s?' % possible_matches[0])
|
get_close_matches的使用示例:
1
2
|
>>> get_close_matches("appel", ["ape", "apple", "peach", "puppy"])
['apple', 'ape']
|
对算法感兴趣的同学，可以自己进一步了解其实现细节。
另外一个小技巧是一个命名的异化细节。class 一般在很多开发语言中都是关键字，如果我们要定义一个class类型的变量名时候，避免关键字冲突。一种方法是使用 klass 替代:
1
2
3
4
5
6
|
#
if isinstance(app_name, BaseCommand):
# If the command is already loaded, use it directly.
klass = app_name
else:
klass = load_command_class(app_name, subcommand)
|
另一种是使用 clazz 替代:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
if ( classes.length ) {
while ( ( elem = this[ i++ ] ) ) {
curValue = getClass( elem );
...
if ( cur ) {
j = 0;
while ( ( clazz = classes[ j++ ] ) ) {
// Remove *all* instances
while ( cur.indexOf( " " + clazz + " " ) > -1 ) {
cur = cur.replace( " " + clazz + " ", " " );
}
}
...
}
}
}
|
大家一般都用哪一种呢?
参考链接