Replace storage.config and volume.config with storage.yaml (#13031)

* Replace storage.config & volume.config by storage.yaml

* Doc: storage.yaml

Co-authored-by: Alan M. Carroll <amc@apache.org>

* Fix unit tests with ASan

* Fix cache_volume_features AuTest

* Fix CacheProcessor::diskInitialized

* Minor fixes

* Configuration Parsing Library Support

* Add 'traffic_ctl config convert storage' cmd

* Claenup

---------

Co-authored-by: Alan M. Carroll <amc@apache.org>

Author: masaori335

.gitignore

@@ -47,7 +47,7 @@ m4/lt~obsolete.m4
 /build*
 
 configs/records.yaml.default
-configs/storage.config.default
+configs/storage.yaml.default
 
 doc/_build
 doc/docbuild/

CMakeLists.txt

@@ -759,7 +759,7 @@ file(REMOVE "${PROJECT_SOURCE_DIR}/include/tscore/ink_config.h")
 file(REMOVE "${PROJECT_SOURCE_DIR}/include/ts/apidefs.h")
 file(REMOVE "${PROJECT_SOURCE_DIR}/include/ink_autoconf.h")
 
-configure_file(configs/storage.config.default.in configs/storage.config.default)
+configure_file(configs/storage.yaml.default.in configs/storage.yaml.default)
 configure_file(configs/records.yaml.default.in configs/records.yaml.default)
 configure_file(include/tscore/ink_config.h.cmake.in include/tscore/ink_config.h)
 configure_file(include/ts/apidefs.h.in include/ts/apidefs.h)

configs/records.yaml.default.in

@@ -25,7 +25,7 @@ records:
 ##############################################################################
 # RAM and disk cache configurations. Docs:
 #    https://docs.trafficserver.apache.org/en/latest/admin-guide/files/records.yaml.en.html#ram-cache
-#    https://docs.trafficserver.apache.org/en/latest/admin-guide/files/storage.config.en.html
+#    https://docs.trafficserver.apache.org/en/latest/admin-guide/files/storage.yaml.en.html
 ##############################################################################
     ram_cache:
       size: -1

configs/storage.config.default.in

@@ -0,0 +1,32 @@
+#
+# storage.yaml - Span and Volume Configuration file
+#
+# Documentation:
+#    https://docs.trafficserver.apache.org/en/latest/admin-guide/files/storage.yaml.en.html
+#
+# The storage configuration is a list of all the storage to be used by the cache and volume
+#
+# cache:                  # file level key
+#   spans:                #
+#     - name:             # name of the span
+#       path:             # path to storage
+#       size:             # size in bytes, required for file system storage, optional for raw device
+#       hash_seed:        # optional, used to isolate lookup from path changes
+#   volumes:              # optional
+#     - id:               # identifier [1-255]
+#       size:             # optional, size in bytes or percentage
+#       scheme:           # optional, default to "http"
+#       ram_cache:        # optional, default to "true"
+#       ram_cache_size:   # optional, allocate a dedicated RAM cache pool for this volume
+#       ram_cache_cutoff: # optional, overrides proxy.config.cache.ram_cache_cutoff
+#       avg_obj_size:     # optional, overrides proxy.config.cache.min_average_object_size
+#       fragment_size:    # optional, overrides proxy.config.cache.target_fragment_size
+#       spans:            # optional
+#         - use:          # Span identifier
+#           size:         # size allocated to this volume
+
+cache:
+  spans:
+    - name: disk-1
+      path: @rel_cachedir@
+      size: 256M

configs/storage.yaml.default.in

@@ -29,7 +29,7 @@
 #       v1.0.3   - Check to force /mnt based cache.db
 
 REMAP_FILE=/usr/local/etc/trafficserver/remap.config
-STORAGE_FILE=/usr/local/etc/trafficserver/storage.config
+STORAGE_FILE=/usr/local/etc/trafficserver/storage.yaml
 EC2_CACHE_LOC=/mnt/trafficserver_cache
 
 # Base settings to use for testing and benchmarking

configs/volume.config.default

@@ -30,7 +30,7 @@ refer to :ref:`partitioning-the-cache`.
 
 Before you can assign cache volumes to specific
 origin servers and/or domains, you must first partition your cache
-according to size and protocol in the :file:`volume.config`
+according to size and protocol in the :file:`storage.yaml`
 file.
 
 After you modify hosting.config, navigate to the Traffic Server bin
@@ -52,7 +52,7 @@ want to store on a particular partition (for example, ``mydomain.com``);
 and ``NUMBERS`` is a comma-separated list of the partitions on
 which you want to store the content that belongs to the origin server or
 domain listed. The partition numbers must be valid numbers listed in the
-:file:`volume.config`.
+:file:`storage.yaml`.
 
 **Note:** To allocate more than one partition to an origin server or
 domain, you must enter the partitions in a comma-separated list on one

contrib/set_trafficserver.sh

@@ -37,9 +37,8 @@ Configuration Files
    splitdns.config.en
    ssl_multicert.yaml.en
    sni.yaml.en
-   storage.config.en
+   storage.yaml.en
    strategies.yaml.en
-   volume.config.en
    jsonrpc.yaml.en
 
 :doc:`cache.config.en`
@@ -83,15 +82,12 @@ Configuration Files
 :doc:`sni.yaml.en`
    Configures SNI based Layer 4 routing.
 
-:doc:`storage.config.en`
-   Configures all storage devices and paths to be used for the |TS| cache.
+:doc:`storage.yaml.en`
+   Configures all storage devices and paths to be used for the |TS| cache and defines cache space usage by individual protocols.
 
 :doc:`strategies.yaml.en`
    Configures NextHop strategies used with `remap.config` and replaces parent.config.
 
-:doc:`volume.config.en`
-    Defines cache space usage by individual protocols.
-
 :doc:`jsonrpc.yaml.en`
     Defines some of the configurable arguments of the jsonrpc endpoint.
 

doc/admin-guide/files/hosting.config.en.rst

@@ -2851,7 +2851,7 @@ RAM Cache
    **20GB** (21474836480)
 
    This global setting can be overridden on a per-volume basis using the
-   ``ram_cache_size`` parameter in :file:`volume.config`. Per-volume
+   ``ram_cache_size`` parameter in :file:`storage.yaml`. Per-volume
    allocations are subtracted from the total RAM cache size before
    distributing the remainder among volumes without explicit settings.
 
@@ -2863,7 +2863,7 @@ RAM Cache
    **4MB** (4194304)
 
    This global setting can be overridden on a per-volume basis using the
-   ``ram_cache_cutoff`` parameter in :file:`volume.config`. When set,
+   ``ram_cache_cutoff`` parameter in :file:`storage.yaml`. When set,
    the per-volume cutoff takes precedence over this global setting for
    that specific volume.
 
@@ -5701,11 +5701,11 @@ Sockets
    ``2`` Do not accept inbound connections until cache initialization has
          finished and been sufficiently successful that cache is enabled. This
          means at least one cache span is usable. If there are no spans in
-         :file:`storage.config` or none of the spans can be successfully parsed
+         :file:`storage.yaml` or none of the spans can be successfully parsed
          and initialized then |TS| will shut down.
    ``3`` Do not accept inbound connections until cache initialization has
          finished and been completely successful. This requires at least one
-         cache span in :file:`storage.config` and that every span specified is
+         cache span in :file:`storage.yaml` and that every span specified is
          valid and successfully initialized. Any error will cause |TS| to shut
          down.
    ===== ======================================================================