【问题标题】:Setup custom cleanup for docker images in nexus repository为 nexus 存储库中的 docker 图像设置自定义清理
【发布时间】:2021-11-16 02:44:25
【问题描述】:

我一直在使用cleanup policies 处理存储在我们的nexus 存储库中的docker 图像。这些对基本行为有好处,在每天(或每小时或您想要的)运行的任务中配置,如下所示:

  1. 第一个任务的类型为管理员 - 使用相关策略清理存储库
  2. 第二个类型 Docker - 删除未使用的清单和图像
  3. 第三种类型管理员 - 紧凑型 blob 存储

清理策略有一个正则表达式,以避免删除以某种方式标记的某个图像(例如: build-latest),以及最后一次下载的位置(eg: 5 天)。

现在这有助于每 X 天删除一次映像,但只要不存在其他映像就需要保留一些映像,即如果唯一存在的映像是 build-99,请不要删除它,即仅靠政策是无法做到的。

我想要实现的 repo 看起来如何:

my-repository 只是一个文件夹名称,默认情况下采用存储库名称,只是为了演示。

那么你是如何管理的呢?

注意:有关此处所做操作的信息可以在不同的 SO 帖子或 github 中找到

【问题讨论】:

    标签: docker groovy nexus


    【解决方案1】:

    使用每天自动运行的 groovy 脚本,我能够做到这一点。 该脚本设置在 Admin - Execute script 的任务中,在 nexus 较新版本中默认禁用,我在常见问题部分中的 Scripting Nexus Repository Manager 3 以及 How to Determine the Location of the Nexus 3 Data Directory 解决了这个问题。

    该脚本基于来自不同地方的文档、问题和代码(例如: StorageTxImpl.java 是您可以找到获取/删除资产、组件等的方法的地方)。它也受到这些启发 Using the Nexus3 API how do I get a list of artifacts in a repositoryNEXUS-14837Nexus 3 Groovy Script development environment setup

    脚本:

    脚本必须在第二个任务之前运行(即等于第一个,之前或之后都没有关系)。这些策略也不再需要,因此不再分配给存储库。

    它是如何工作或做什么的:

    • 获取存储库
    • 获取 repo 的组件
    • 按名称对它们进行分组(例如: repository/my-repository/some-project/service-A)
    • 为每个服务循环其组件并获取其资产
    • 按资产的last_downloaded 过滤资产,并仅保留与最近的 3 个不匹配的资产,例如
    • 删除与资产相关的组件(nexus deleteComponent(cp) 在内部删除资产及其 blob)

    注意:我看到脚本可以参数化,但在我的情况下不需要

    注意:这可以更新以循环所有存储库,但我只需要一个

    import org.sonatype.nexus.repository.storage.Asset
    import org.sonatype.nexus.repository.storage.Query
    import org.sonatype.nexus.repository.storage.StorageFacet
    
    import groovy.json.JsonOutput
    import groovy.json.JsonSlurper
    import org.sonatype.nexus.repository.Repository
    
    class RepositoryProcessor {
        private final log
        private final repository
        private final String repoName = 'my-repository'
        private final String[] ignoreVersions = ['build-latest']
        private final int processIfSizeGt = 3
        private final int delAllButMostRecentNImages = 2
    
        RepositoryProcessor(log, repository) {
            this.log = log
            this.repository = repository
        }
    
        void processRepository() {
            def repo = repository.repositoryManager.get(repoName)
            log.debug("found repository: {}", repo)
            // will use default of sonatype
            // https://github.com/sonatype/nexus-public/blob/master/components/nexus-repository/src/main/java/org/sonatype/nexus/repository/storage/StorageFacetImpl.java
            StorageFacet storageFacet = repo.facet(StorageFacet)
            log.debug("initiated storage facet: {}", storageFacet.toString())
            // tx of type https://github.com/sonatype/nexus-public/blob/master/components/nexus-repository/src/main/java/org/sonatype/nexus/repository/storage/StorageTxImpl.java $$EnhancerByGuice ??
            def transaction = storageFacet.txSupplier().get()
            log.debug("initiated transaction instance: {}", transaction.toString())
    
            try {
                transaction.begin()
    
                log.info("asset count {}", transaction.countAssets(Query.builder().build(), [repo]))
                log.info("components count {}", transaction.countComponents(Query.builder().build(), [repo]))
    
                // queried db is orientdb, syntax is adapted to it
                def components = transaction.findComponents(Query.builder()
                        // .where("NOT (name LIKE '%service-A%')")
                        // .and("NOT (name LIKE '%service-B%')")
                        .build(), [repo])
                // cp and cpt refers to component
                // group by name eg: repository/my-repository/some-project/service-A
                def groupedCps = components.groupBy{ it.name() }.collect()
    
                // fetch assets for each cp
                // and set them in maps to delete the old ones
                groupedCps.each{ cpEntry ->
                    // process only if its greater than the minimum amount of images per service
                    if (cpEntry.value.size > processIfSizeGt) {
                        // single component processing (i.e this would be done for each service)
                        def cpMap = [:] // map with key eq id
                        def cpAssetsMap = [:] // map of cp assets where key eq cp id
                        // process service cpts
                        cpEntry.value.each { cp ->
                            // cp id of type https://github.com/sonatype/nexus-public/blob/master/components/nexus-orient/src/main/java/org/sonatype/nexus/orient/entity/AttachedEntityId.java
                            def cpId = cp.entityMetadata.id.identity
                            // asset of type: https://github.com/sonatype/nexus-public/blob/master/components/nexus-repository/src/main/java/org/sonatype/nexus/repository/storage/Asset.java
                            def cpAssets = transaction.browseAssets(cp).collect()
                           
                            // document of type https://github.com/joansmith1/orientdb/blob/master/core/src/main/java/com/orientechnologies/orient/core/record/impl/ODocument.java
                            // _fields of type: https://github.com/joansmith1/orientdb/blob/master/core/src/main/java/com/orientechnologies/orient/core/record/impl/ODocumentEntry.java
                            // any field is of type ODocumentEntry.java
                            // append to map if it does not belong to the ignored versions
                            if (!(cp.entityMetadata.document._fields.version.value in ignoreVersions)) {
                                cpMap.put(cpId, cp)
                                cpAssetsMap.put(cpId, cpAssets)
                            }
                        }
                        // log info about the affected folder/service
                        log.info("cp map size: {}, versions: {}",
                                cpMap.values().size(),
                                cpMap.values().entityMetadata.document._fields.version.value)
                        // order desc by last_downloaded (default is asc)
                        log.debug("cp map assets of size: {}", cpAssetsMap.values().size())
                        def sortedFilteredList = cpAssetsMap.values()
                                .sort { it.entityMetadata.document._fields.last_downloaded?.value }
                                .reverse(true)
                                .drop(delAllButMostRecentNImages)
                        // list of cp ids from the assets that going to be deleted
                        def sortedAssetsCps = sortedFilteredList.entityMetadata.document._fields.component?.value?.flatten()
                        log.info("cp map assets size after filtering {}", sortedFilteredList.size())
                        // this will print the cps ids to delete
                        log.debug("elements to delete : sorted assets cps list {}", sortedAssetsCps)
                        // deleting components and their assets
                        cpMap.findAll { it.key in sortedAssetsCps }
                                .each { entry ->
                                    log.info("deleting cp version {}", entry.value.entityMetadata.document._fields.version?.value)
                                    // this will call delete asset internally, and by default will delete blob
                                    transaction.deleteComponent(entry.value)
                                }
                    }
                }
                transaction.commit();
            } catch (Exception e) {
                log.warn("transaction failed {}", e.toString())
                transaction.rollback()
            } finally {
                transaction.close();
            }
        }
    }
    
    new RepositoryProcessor(log, repository).processRepository()
    
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 2021-05-03
      • 1970-01-01
      • 2015-04-18
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2017-08-19
      相关资源
      最近更新 更多