【问题标题】:Is it possible to store the alembic connect string outside of alembic.ini?是否可以将 alembic 连接字符串存储在 alembic.ini 之外?
【发布时间】:2014-04-06 08:45:37
【问题描述】:

我将 Alembic 与 SQL Alchemy 一起使用。对于 SQL Alchemy,我倾向于遵循一种模式,即不将连接字符串与版本化代码一起存储。相反,我有包含任何机密信息的文件secret.py。我将这个文件名放在我的.gitignore 中,这样它就不会出现在 GitHub 上。

这种模式运行良好,但现在我开始使用 Alembic 进行迁移。看来我无法隐藏连接字符串。而是在 alembic.ini 中,将连接字符串放置为 configuration parameter

# the 'revision' command, regardless of autogenerate
# revision_environment = false

sqlalchemy.url = driver://user:pass@localhost/dbname

# Logging configuration
[loggers]
keys = root,sqlalchemy,alembi

我担心我会不小心为我的数据库提交包含用户名/密码信息的文件。我宁愿将此连接字符串存储在一个地方,并避免意外将其提交给版本控制的风险。

我有什么选择?

【问题讨论】:

    标签: python sqlalchemy alembic


    【解决方案1】:

    所以看起来可行的是在 env.py 中重新实现引擎创建,这显然是进行这种自定义的地方,而不是在 ini 中使用 sqlalchemy 连接字符串:

    engine = engine_from_config(
                config.get_section(config.config_ini_section),
                prefix='sqlalchemy.',
               poolclass=pool.NullPool)
    

    您可以替换并指定自己的引擎配置:

    import store
    engine = store.engine
    

    确实是文档seems to imply 这没关系:

    sqlalchemy.url - 通过 SQLAlchemy 连接到数据库的 URL。这个键实际上只在特定于“通用”配置的 env.py 文件中被引用;开发人员可以自定义的文件。多数据库配置可能会在此处响应多个键,也可能会引用文件的其他部分。

    【讨论】:

    • 使用最新的 alembic (0.8.1),这只会影响“在线”模式。 run_migrations_offline 也需要更改以使用不同的数据库 URL。
    【解决方案2】:

    另一种解决方案是创建模板 alembic.ini.dist 文件并使用您的版本化代码跟踪它,同时忽略 VCS 中的 alembic.ini。

    不要在 alembic.ini.dist 中添加任何机密信息:

    sqlalchemy.url = ...
    

    将您的代码部署到平台时,将 alembic.ini.dist 复制到 alembic.ini(您的 VCS 不会跟踪此代码)并使用平台的凭据修改 alembic.ini。

    【讨论】:

      【解决方案3】:

      我昨天遇到了同样的问题,并找到了以下解决方案。 我在alembic/env.py 中执行以下操作:

      # this is the Alembic Config object, which provides
      # access to the values within the .ini file in use.
      config = context.config
      
      # this will overwrite the ini-file sqlalchemy.url path
      # with the path given in the config of the main code
      import config as ems_config
      config.set_main_option('sqlalchemy.url', ems_config.config.get('sql', 'database'))
      

      ems_config 是一个保存我的配置数据的外部模块。

      config.set_main_option(...) 实质上覆盖了alembic.ini 文件的[alembic] 部分中的sqlalchemy.url 键。在我的配置中,我只是将其保留为黑色。

      【讨论】:

      【解决方案4】:

      As Doug T. said 您可以编辑 env.py 以从除 ini 文件之外的其他位置提供 URL。您可以向engine_from_config 函数传递一个额外的url 参数,而不是创建新引擎(kwargs 稍后会合并到从ini 文件中获取的选项)。在这种情况下,您可以例如将加密的密码存储在 ini 文件中,并在运行时通过存储在 ENV 变量中的密码对其进行解密。

      connectable = engine_from_config(                 
          config.get_section(config.config_ini_section),
          prefix='sqlalchemy.',                         
          poolclass=pool.NullPool,                      
          url=some_decrypted_endpoint)                                   
      

      【讨论】:

        【解决方案5】:

        Alembic documentation 建议使用 create_engine 和数据库 URL(而不是 modifying sqlalchemy.url in code)。

        您还应该修改 run_migrations_offline 以使用新的 URL。 Allan Simon 在他的博客上有an example,但总的来说,将 env.py 修改为:

        1. 提供一个共享函数以某种方式获取 URL(这里它来自命令行):

          def get_url():
              url = context.get_x_argument(as_dictionary=True).get('url')
              assert url, "Database URL must be specified on command line with -x url=<DB_URL>"
              return url
          
        2. 在离线模式下使用 URL:

          def run_migrations_offline():
              ...
              url = get_url()
              context.configure(
                  url=url, target_metadata=target_metadata, literal_binds=True)
              ...
          
        3. 使用create_engine而不是engine_from_config在在线模式下使用URL:

          def run_migrations_online():
              ...
              connectable = create_engine(get_url())
              with connectable.connect() as connection:
              ...
          

        【讨论】:

        【解决方案6】:

        我一直在寻找如何为多数据库管理这个

        这就是我所做的。我有两个数据库:logsohlc

        根据doc, 我已经设置了这样的蒸馏器

        alembic init --template multidb
        

        alembic.ini

        databases = logs, ohlc
        [logs]
        sqlalchemy.url = postgresql://botcrypto:botcrypto@localhost/logs
        [ohlc]
        sqlalchemy.url = postgresql://botcrypto:botcrypto@localhost/ohlc
        

        env.py

        ​​>
        [...]
        # this is the Alembic Config object, which provides
        # access to the values within the .ini file in use.
        config = context.config
        
        # Interpret the config file for Python logging.
        # This line sets up loggers basically.
        fileConfig(config.config_file_name)
        logger = logging.getLogger('alembic.env')
        
        # overwrite alembic.ini db urls from the config file
        settings_path = os.environ.get('SETTINGS')
        if settings_path:
            with open(settings_path) as fd:
                settings = conf.load(fd, context=os.environ) # loads the config.yml
            config.set_section_option("ohlc", "sqlalchemy.url", settings["databases"]["ohlc"])
            config.set_section_option("logs", "sqlalchemy.url", settings["databases"]["logs"])
        else:
            logger.warning('Environment variable SETTINGS missing - use default alembic.ini configuration')
        [...]
        

        config.yml

        databases:
            logs: postgresql://botcrypto:botcrypto@127.0.0.1:5432/logs
            ohlc: postgresql://botcrypto:botcrypto@127.0.0.1:5432/ohlc
        

        用法

        SETTINGS=config.yml alembic upgrade head
        

        希望对你有帮助!

        【讨论】:

          【解决方案7】:

          为了避免提交我的用户/密码,我能想到的最简单的方法是 a) 将插值字符串添加到 alembic.ini 文件中,b) 在 env.py 中设置这些插值

          alembic.ini

          sqlalchemy.url = postgresql://%(DB_USER)s:%(DB_PASS)s@35.197.196.146/nozzle-website
          

          env.py

          import os
          
          from logging.config import fileConfig
          
          from sqlalchemy import engine_from_config
          from sqlalchemy import pool
          
          from alembic import context
          
          # this is the Alembic Config object, which provides
          # access to the values within the .ini file in use.
          config = context.config
          
          # here we allow ourselves to pass interpolation vars to alembic.ini
          # fron the host env
          section = config.config_ini_section
          config.set_section_option(section, "DB_USER", os.environ.get("DB_USER"))
          config.set_section_option(section, "DB_PASS", os.environ.get("DB_PASS"))
          
          ...
          

          【讨论】:

          • 这对我有用,而且似乎比最佳答案更容易 - 有什么缺点吗?
          • 由于 env.py 现在依赖于正确设置的环境变量,如果未设置 DB_USER 或 DB_PASS,您可能需要进行一些防御性编码并提供有用的“使用”消息。
          • 迄今为止最新的解决方案。到目前为止,您可能无论如何都在受保护的 .env 文件中设置密码并使用 dotenv 加载它。所以只有重用这个环境变量才有意义。如果你只写 os.environ["DB_USER"],你也可能不需要“深度编码”,因为由此产生的关键错误已经非常明确了。
          【解决方案8】:

          我在这里尝试了所有答案,但没有奏效。然后我尝试自己处理,如下:

          .ini 文件:

          # A generic, single database configuration.
          
          [alembic]
          # path to migration scripts
          script_location = alembic
          
          # template used to generate migration files
          file_template = %%(rev)s_%%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d_%%(minute).2d_%%(second).2d
          
          # timezone to use when rendering the date
          # within the migration file as well as the filename.
          # string value is passed to dateutil.tz.gettz()
          # leave blank for localtime
          # timezone =
          
          # max length of characters to apply to the
          # "slug" field
          #truncate_slug_length = 40
          
          # set to 'true' to run the environment during
          # the 'revision' command, regardless of autogenerate
          # revision_environment = false
          
          # set to 'true' to allow .pyc and .pyo files without
          # a source .py file to be detected as revisions in the
          # versions/ directory
          # sourceless = false
          
          # version location specification; this defaults
          # to alembic/versions.  When using multiple version
          # directories, initial revisions must be specified with --version-path
          # version_locations = %(here)s/bar %(here)s/bat alembic/versions
          
          # the output encoding used when revision files
          # are written from script.py.mako
          # output_encoding = utf-8
          
          databases = auth_engine
          
          
          [auth_engine]
          sqlalchemy.url = mysql+mysqldb://{}:{}@{}:{}/auth_db
          
          # Logging configuration
          [loggers]
          keys = root,sqlalchemy,alembic
          
          [handlers]
          keys = console
          
          [formatters]
          keys = generic
          
          [logger_root]
          level = WARN
          handlers = console
          qualname =
          
          [logger_sqlalchemy]
          level = WARN
          handlers =
          qualname = sqlalchemy.engine
          
          [logger_alembic]
          level = INFO
          handlers =
          qualname = alembic
          
          [handler_console]
          class = StreamHandler
          args = (sys.stderr,)
          level = NOTSET
          formatter = generic
          
          [formatter_generic]
          format = %(levelname)-5.5s [%(name)s] %(message)s
          datefmt = %H:%M:%S
          

          .env 文件(它在我项目的根文件夹中):

          DB_USER='root'
          DB_PASS='12345678'
          DB_HOST='127.0.0.1'
          DB_PORT='3306'
          

          env.py 文件:

          from __future__ import with_statement
          
          import os
          import re
          import sys
          from logging.config import fileConfig
          
          from sqlalchemy import engine_from_config
          from sqlalchemy import pool
          
          from alembic import context
          
          DB_USER = os.getenv("DB_USER")
          DB_PASS = os.getenv("DB_PASS")
          DB_HOST = os.getenv("DB_HOST")
          DB_PORT = os.getenv("DB_PORT")
          
          # this is the Alembic Config object, which provides
          # access to the values within the .ini file in use.
          config = context.config
          
          # Interpret the config file for Python logging.
          # This line sets up loggers basically.
          fileConfig(config.config_file_name)
          
          # gather section names referring to different
          # databases.  These are named "engine1", "engine2"
          # in the sample .ini file.
          db_names = config.get_main_option('databases')
          
          # add your model's MetaData object here
          # for 'autogenerate' support
          # from myapp import mymodel
          # target_metadata = mymodel.Base.metadata
          
          sys.path.append(os.path.join(os.path.dirname(__file__), "../../../"))
          from db_models.auth_db import auth_db_base
          
          target_metadata = {
              'auth_engine': auth_db_base.auth_metadata
          }
          
          
          # other values from the config, defined by the needs of env.py,
          # can be acquired:
          # my_important_option = config.get_main_option("my_important_option")
          # ... etc.
          
          
          def run_migrations_offline():
              """Run migrations in 'offline' mode.
          
              This configures the context with just a URL
              and not an Engine, though an Engine is acceptable
              here as well.  By skipping the Engine creation
              we don't even need a DBAPI to be available.
          
              Calls to context.execute() here emit the given string to the
              script output.
          
              """
              engines = {}
              for name in re.split(r',\s*', db_names):
                  engines[name] = rec = {}
                  section = context.config.get_section(name)
                  url = section['sqlalchemy.url'].format(DB_USER, DB_PASS, DB_HOST, DB_PORT)
                  section['sqlalchemy.url'] = url
                  rec['url'] = url
                  # rec['url'] = context.config.get_section_option(name, "sqlalchemy.url")
          
              for name, rec in engines.items():
                  print("Migrating database %s" % name)
                  file_ = "%s.sql" % name
                  print("Writing output to %s" % file_)
                  with open(file_, 'w') as buffer:
                      context.configure(url=rec['url'], output_buffer=buffer,
                                        target_metadata=target_metadata.get(name),
                                        compare_type=True,
                                        compare_server_default=True
                                        )
          
                      with context.begin_transaction():
                          context.run_migrations(engine_name=name)
          
          
          def run_migrations_online():
              """Run migrations in 'online' mode.
          
              In this scenario we need to create an Engine
              and associate a connection with the context.
          
              """
              engines = {}
              for name in re.split(r',\s*', db_names):
                  engines[name] = rec = {}
                  section = context.config.get_section(name)
                  url = section['sqlalchemy.url'].format(DB_USER, DB_PASS, DB_HOST, DB_PORT)
                  section['sqlalchemy.url'] = url
                  rec['engine'] = engine_from_config(
                      section,
                      prefix='sqlalchemy.',
                      poolclass=pool.NullPool)
          
              for name, rec in engines.items():
                  engine = rec['engine']
                  rec['connection'] = conn = engine.connect()
                  rec['transaction'] = conn.begin()
          
              try:
                  for name, rec in engines.items():
                      print("Migrating database %s" % name)
                      context.configure(
                          connection=rec['connection'],
                          upgrade_token="%s_upgrades" % name,
                          downgrade_token="%s_downgrades" % name,
                          target_metadata=target_metadata.get(name),
                          compare_type=True,
                          compare_server_default=True
                      )
                      context.run_migrations(engine_name=name)
          
                  for rec in engines.values():
                      rec['transaction'].commit()
              except:
                  for rec in engines.values():
                      rec['transaction'].rollback()
                  raise
              finally:
                  for rec in engines.values():
                      rec['connection'].close()
          
          
          if context.is_offline_mode():
              run_migrations_offline()
          else:
              run_migrations_online()
          

          Wish 可以帮助别人。

          【讨论】:

          • 这在很多层面上都是错误和危险的,无法解开,所以我只想指出,永远不会发生篡改 syspath 的三个文件夹。
          【解决方案9】:

          我也遇到了这个问题,因为我们正在从本地机器运行迁移。我的解决方案是将环境部分放在存储数据库配置(减去凭据)的alembic.ini 中:

          [local]
          host = localhost
          db = dbname
          
          [test]
          host = x.x.x.x
          db = dbname
          
          [prod]
          host = x.x.x.x
          db = dbname
          

          然后我将以下内容放在env.py 中,以便用户可以选择他们的环境并提示输入凭据:

          from alembic import context
          from getpass import getpass
          
          ...
          
          envs = ['local', 'test', 'prod']
          
          print('Warning: Do not commit your database credentials to source control!')
          print(f'Available migration environments: {", ".join(envs)}')
          
          env = input('Environment: ')
          
          if env not in envs:
              print(f'{env} is not a valid environment')
              exit(0)
          
          env_config = context.config.get_section(env)
          host = env_config['host']
          db = env_config['db']
          
          username = input('Username: ')
          password = getpass()
          connection_string = f'postgresql://{username}:{password}@{host}/{db}'
          
          context.config.set_main_option('sqlalchemy.url', connection_string)
          

          您应该将您的凭据存储在整个团队都可以访问的密码管理器中,或者您可以使用的任何配置/秘密存储中。但是,通过这种方法,密码会暴露给您的本地剪贴板 - 更好的方法是让 env.py 直接连接到您的配置/秘密存储 API 并直接提取用户名/密码,但这会增加第三方依赖.

          【讨论】:

            【解决方案10】:

            env.py 中添加

            config.set_main_option('sqlalchemy.url', os.environ['DB_URL'])
            

            之后

            config = context.config
            

            喜欢

            config = context.config
            config.set_main_option('sqlalchemy.url', os.environ['DB_URL'])
            

            然后像这样执行:

            DB_URL="mysql://atuamae:de4@127.0.0.1/db" \
              alembic upgrade head
            

            【讨论】:

              【解决方案11】:

              env.py:

              from alembic.config import Config
              
              alembic_cfg = Config()
              alembic_cfg.set_main_option("sqlalchemy.url", getenv('PG_URI'))
              

              https://alembic.sqlalchemy.org/en/latest/api/config.html

              【讨论】:

                【解决方案12】:

                对于MultiDB设置(SingleDB也一样),你可以使用config.set_section_option('section_name', 'variable_name', 'db_URL') 修改 alembic.ini 文件中数据库 URL 的值。

                例如:

                alembic.init

                [engine1]
                sqlalchemy.url = 
                
                [engine2]
                sqlalchemy.url = 
                

                那么,

                env.py

                config = context.config
                
                config.set_section_option('engine1', 'sqlalchemy.url', os.environ.get('URL_DB1'))
                config.set_section_option('engine2', 'sqlalchemy.url', os.environ.get('URL_DB2'))
                

                【讨论】:

                  猜你喜欢
                  • 1970-01-01
                  • 1970-01-01
                  • 1970-01-01
                  • 1970-01-01
                  • 2011-09-20
                  • 2011-04-29
                  • 1970-01-01
                  • 1970-01-01
                  • 1970-01-01
                  相关资源
                  最近更新 更多