go.mod: Upgrade to quic-go v0.37.3

Fixes #5680 once and for all! Hopefully :)

Thank you @marten-seemann for your excellent work!

.github/CONTRIBUTING.md

@@ -1,7 +1,7 @@
 Contributing to Caddy
 =====================
 
-Welcome! Thank you for choosing to be a part of our community. Caddy wouldn't be great without your involvement!
+Welcome! Thank you for choosing to be a part of our community. Caddy wouldn't be nearly as excellent without your involvement!
 
 For starters, we invite you to join [the Caddy forum](https://caddy.community) where you can hang out with other Caddy users and developers.
 
@@ -35,19 +35,29 @@ Here are some of the expectations we have of contributors:
 
 - **Keep related commits together in a PR.** We do want pull requests to be small, but you should also keep multiple related commits in the same PR if they rely on each other.
 
-- **Write tests.** Tests are essential! Written properly, they ensure your change works, and that other changes in the future won't break your change. CI checks should pass.
+- **Write tests.** Good, automated tests are very valuable! Written properly, they ensure your change works, and that other changes in the future won't break your change. CI checks should pass.
 
-- **Benchmarks should be included for optimizations.** Optimizations sometimes make code harder to read or have changes that are less than obvious. They should be proven with benchmarks or profiling.
+- **Benchmarks should be included for optimizations.** Optimizations sometimes make code harder to read or have changes that are less than obvious. They should be proven with benchmarks and profiling.
 
 - **[Squash](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html) insignificant commits.** Every commit should be significant. Commits which merely rewrite a comment or fix a typo can be combined into another commit that has more substance. Interactive rebase can do this, or a simpler way is `git reset --soft <diverging-commit>` then `git commit -s`.
 
-- **Own your contributions.** Caddy is a growing project, and it's much better when individual contributors help maintain their change after it is merged.
+- **Be responsible for and maintain your contributions.** Caddy is a growing project, and it's much better when individual contributors help maintain their change after it is merged.
 
 - **Use comments properly.** We expect good godoc comments for package-level functions, types, and values. Comments are also useful whenever the purpose for a line of code is not obvious.
 
-- **Pull requests may still get closed.** The longer a PR stays open and idle, the more likely it is to be closed. If we haven't reviewed it in a while, it probably means the change is not a priority. Please don't take this personally, we're trying to balance a lot of tasks! If nobody else has commented or reacted to the PR, it likely means your change is useful only to you. The reality is this happens quite a bit. We don't tend to accept PRs that aren't generally helpful. For these reasons or others, the PR may get closed even after a review. We are not obligated to accept all proposed changes, even if the best justification we can give is something vague like, "It doesn't sit right." Sometimes PRs are just the wrong thing or the wrong time. Because it is open source, you can always build your own modified version of Caddy with a change you need, even if we reject it in the official repo.
+- **Pull requests may still get closed.** The longer a PR stays open and idle, the more likely it is to be closed. If we haven't reviewed it in a while, it probably means the change is not a priority. Please don't take this personally, we're trying to balance a lot of tasks! If nobody else has commented or reacted to the PR, it likely means your change is useful only to you. The reality is this happens quite a lot. We don't tend to accept PRs that aren't generally helpful. For these reasons or others, the PR may get closed even after a review. We are not obligated to accept all proposed changes, even if the best justification we can give is something vague like, "It doesn't sit right." Sometimes PRs are just the wrong thing or the wrong time. Because it is open source, you can always build your own modified version of Caddy with a change you need, even if we reject it in the official repo. Plus, because Caddy is extensible, it's possible your feature could make a great plugin instead!
 
-We often grant [collaborator status](#collaborator-instructions) to contributors who author one or more significant, high-quality PRs that are merged into the code base!
+- **You certify that you wrote and comprehend the code you submit.** The Caddy project welcomes original contributions that comply with [our CLA](https://cla-assistant.io/caddyserver/caddy), meaning that authors must be able to certify that they created or have rights to the code they are contributing. In addition, we require that code is not simply copy-pasted from Q/A sites or AI language models without full comprehension and rigorous testing. In other words: contributors are allowed to refer to communities for assistance and use AI tools such as language models for inspiration, but code which originates from or is assisted by these resources MUST be:
+
+	- Licensed for you to freely share
+	- Fully comprehended by you (be able to explain every line of code)
+	- Verified by automated tests when feasible, or thorough manual tests otherwise
+
+	We have found that current language models (LLMs, like ChatGPT) may understand code syntax and even problem spaces to an extent, but often fail in subtle ways to convey true knowledge and produce correct algorithms. Integrated tools such as GitHub Copilot and Sourcegraph Cody may be used for inspiration, but code generated by these tools still needs to meet our criteria for licensing, human comprehension, and testing. These tools may be used to help write code comments and tests as long as you can certify they are accurate and correct. Note that it is often more trouble than it's worth to certify that Copilot (for example) is not giving you code that is possibly plagiarised, unlicensed, or licensed with incompatible terms -- as the Caddy project cannot accept such contributions. If that's too difficult for you (or impossible), then we recommend using these resources only for inspiration and write your own code. Ultimately, you (the contributor) are responsible for the code you're submitting.
+
+	As a courtesy to reviewers, we kindly ask that you disclose when contributing code that was generated by an AI tool or copied from another website so we can be aware of what to look for in code review.
+
+We often grant [collaborator status](#collaborator-instructions) to contributors who author one or more significant, high-quality PRs that are merged into the code base.
 
 
 #### HOW TO MAKE A PULL REQUEST TO CADDY

.github/SECURITY.md

@@ -7,7 +7,7 @@ The Caddy project would like to make sure that it stays on top of all practicall
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 2.x     | :white_check_mark: |
+| 2.x     | ✔️ |
 | 1.x     | :x:                |
 | < 1.x   | :x:                |
 
@@ -24,7 +24,7 @@ We do not accept reports if the steps imply or require a compromised system or t
 
 Client-side exploits are out of scope. In other words, it is not a bug in Caddy if the web browser does something unsafe, even if the downloaded content was served by Caddy. (Those kinds of exploits can generally be mitigated by proper configuration of HTTP headers.) As a general rule, the content served by Caddy is not considered in scope because content is configurable by the site owner or the associated web application.
 
-Security bugs in code dependencies are out of scope. Instead, if a dependency has patched a relevant security bug, please feel free to open a public issue or pull request to update that dependency in our code.
+Security bugs in code dependencies (including Go's standard library) are out of scope. Instead, if a dependency has patched a relevant security bug, please feel free to open a public issue or pull request to update that dependency in our code.
 
 
 ## Reporting a Vulnerability
@@ -42,7 +42,7 @@ We'll need enough information to verify the bug and make a patch. To speed thing
 - Specific minimal steps to reproduce the issue from scratch
 - A working patch
 
-Please DO NOT use containers, VMs, cloud instances or services, or any other complex infrastructure in your steps. Always prefer `curl` instead of web browsers.
+Please DO NOT use containers, VMs, cloud instances or services, or any other complex infrastructure in your steps. Always prefer `curl -v` instead of web browsers.
 
 We consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems. Do not redact domain names from your reports. The actual content of your domain name affects Caddy's behavior, so we need the exact domain name(s) to reproduce with, or your report will be ignored.
 

.github/workflows/ci.yml

@@ -18,17 +18,22 @@ jobs:
       # Default is true, cancels jobs for other platforms in the matrix if one fails
       fail-fast: false
       matrix:
-        os: [ ubuntu-latest, macos-latest, windows-latest ]
-        go: [ '1.18', '1.20' ]
+        os: 
+          - ubuntu-latest
+          - macos-latest
+          - windows-latest
+        go: 
+          - '1.20'
+          # - '1.21'
 
         include:
         # Set the minimum Go patch version for the given Go minor
         # Usable via ${{ matrix.GO_SEMVER }}
-        - go: '1.18'
-          GO_SEMVER: '~1.18.4'
-
         - go: '1.20'
-          GO_SEMVER: '~1.20.0'
+          GO_SEMVER: '~1.20.6'
+
+        # - go: '1.21'
+        #   GO_SEMVER: '~1.21.0'
 
         # Set some variables per OS, usable via ${{ matrix.VAR }}
         # CADDY_BIN_PATH: the path to the compiled Caddy binary, for artifact publishing
@@ -48,15 +53,15 @@ jobs:
     runs-on: ${{ matrix.os }}
 
     steps:
+    - name: Checkout code
+      uses: actions/checkout@v3
+
     - name: Install Go
-      uses: actions/setup-go@v3
+      uses: actions/setup-go@v4
       with:
         go-version: ${{ matrix.GO_SEMVER }}
         check-latest: true
 
-    - name: Checkout code
-      uses: actions/checkout@v3
-
     # These tools would be useful if we later decide to reinvestigate
     # publishing test/coverage reports to some tool for easier consumption
     # - name: Install test and coverage analysis tools
@@ -79,23 +84,6 @@ jobs:
         # Calculate the short SHA1 hash of the git commit
         echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
 
-    - name: Cache the build cache
-      uses: actions/cache@v3
-      with:
-        # In order:
-        # * Module download cache
-        # * Build cache (Linux)
-        # * Build cache (Mac)
-        # * Build cache (Windows)
-        path: |
-          ~/go/pkg/mod
-          ~/.cache/go-build
-          ~/Library/Caches/go-build
-          ~\AppData\Local\go-build
-        key: ${{ runner.os }}-${{ matrix.go }}-go-ci-${{ hashFiles('**/go.sum') }}
-        restore-keys: |
-          ${{ runner.os }}-${{ matrix.go }}-go-ci
-
     - name: Get dependencies
       run: |
         go get -v -t -d ./...
@@ -146,7 +134,7 @@ jobs:
     if: github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]'
     continue-on-error: true  # August 2020: s390x VM is down due to weather and power issues
     steps:
-      - name: Checkout code into the Go module directory
+      - name: Checkout code
         uses: actions/checkout@v3
       - name: Run Tests
         run: |
@@ -172,7 +160,7 @@ jobs:
   goreleaser-check:
     runs-on: ubuntu-latest
     steps:
-      - name: checkout
+      - name: Checkout code
         uses: actions/checkout@v3
       
       - uses: goreleaser/goreleaser-action@v4

.github/workflows/cross-build.yml

@@ -15,20 +15,35 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        goos: ['android', 'linux', 'solaris', 'illumos', 'dragonfly', 'freebsd', 'openbsd', 'plan9', 'windows', 'darwin', 'netbsd']
-        go: [ '1.20' ]
+        goos: 
+          - 'android'
+          - 'linux'
+          - 'solaris'
+          - 'illumos'
+          - 'dragonfly'
+          - 'freebsd'
+          - 'openbsd'
+          - 'plan9'
+          - 'windows'
+          - 'darwin'
+          - 'netbsd'
+        go: 
+          - '1.20'
 
         include:
         # Set the minimum Go patch version for the given Go minor
         # Usable via ${{ matrix.GO_SEMVER }}
         - go: '1.20'
-          GO_SEMVER: '~1.20.0'
+          GO_SEMVER: '~1.20.6'
 
     runs-on: ubuntu-latest
     continue-on-error: true
     steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
       - name: Install Go
-        uses: actions/setup-go@v3
+        uses: actions/setup-go@v4
         with:
           go-version: ${{ matrix.GO_SEMVER }}
           check-latest: true
@@ -43,22 +58,6 @@ jobs:
           printf "\n\nSystem environment:\n\n"
           env
 
-      - name: Cache the build cache
-        uses: actions/cache@v3
-        with:
-          # In order:
-          # * Module download cache
-          # * Build cache (Linux)
-          path: |
-            ~/go/pkg/mod
-            ~/.cache/go-build
-          key: cross-build-go${{ matrix.go }}-${{ matrix.goos }}-${{ hashFiles('**/go.sum') }}
-          restore-keys: |
-            cross-build-go${{ matrix.go }}-${{ matrix.goos }}
-
-      - name: Checkout code into the Go module directory
-        uses: actions/checkout@v3
-
       - name: Run Build
         env:
           CGO_ENABLED: 0

.github/workflows/lint.yml

@@ -22,20 +22,31 @@ jobs:
     name: lint
     strategy:
       matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
+        os: 
+          - ubuntu-latest
+          - macos-latest
+          - windows-latest
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v3
-      - uses: actions/setup-go@v3
+      - uses: actions/setup-go@v4
         with:
-          go-version: '~1.18.4'
+          go-version: '~1.20.6'
           check-latest: true
 
+          # Workaround for https://github.com/golangci/golangci-lint-action/issues/135
+          skip-pkg-cache: true
+
       - name: golangci-lint
         uses: golangci/golangci-lint-action@v3
         with:
-          version: v1.50
+          version: v1.53
+
+          # Workaround for https://github.com/golangci/golangci-lint-action/issues/135
+          skip-pkg-cache: true
+
           # Windows times out frequently after about 5m50s if we don't set a longer timeout.
           args: --timeout 10m
+
           # Optional: show only new issues if it's a pull request. The default value is `false`.
           # only-new-issues: true

.github/workflows/release.yml

@@ -10,14 +10,16 @@ jobs:
     name: Release
     strategy:
       matrix:
-        os: [ ubuntu-latest ]
-        go: [ '1.20' ]
+        os: 
+          - ubuntu-latest
+        go: 
+          - '1.20'
 
         include:
         # Set the minimum Go patch version for the given Go minor
         # Usable via ${{ matrix.GO_SEMVER }}
         - go: '1.20'
-          GO_SEMVER: '~1.20.0'
+          GO_SEMVER: '~1.20.6'
 
     runs-on: ${{ matrix.os }}
     # https://github.com/sigstore/cosign/issues/1258#issuecomment-1002251233
@@ -29,17 +31,17 @@ jobs:
       contents: write
 
     steps:
-    - name: Install Go
-      uses: actions/setup-go@v3
-      with:
-        go-version: ${{ matrix.GO_SEMVER }}
-        check-latest: true
-
     - name: Checkout code
       uses: actions/checkout@v3
       with:
         fetch-depth: 0
 
+    - name: Install Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: ${{ matrix.GO_SEMVER }}
+        check-latest: true
+
     # Force fetch upstream tags -- because 65 minutes
     # tl;dr: actions/checkout@v3 runs this line:
     #   git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +ebc278ec98bb24f2852b61fde2a9bf2e3d83818b:refs/tags/
@@ -94,18 +96,6 @@ jobs:
         # tags are only accepted if signed by Matt's key
         git verify-tag "${{ steps.vars.outputs.version_tag }}" || exit 1
 
-    - name: Cache the build cache
-      uses: actions/cache@v3
-      with:
-        # In order:
-        # * Module download cache
-        # * Build cache (Linux)
-        path: |
-          ~/go/pkg/mod
-          ~/.cache/go-build
-        key: ${{ runner.os }}-go${{ matrix.go }}-release-${{ hashFiles('**/go.sum') }}
-        restore-keys: |
-          ${{ runner.os }}-go${{ matrix.go }}-release
     - name: Install Cosign
       uses: sigstore/cosign-installer@main
     - name: Cosign version
@@ -119,7 +109,7 @@ jobs:
       uses: goreleaser/goreleaser-action@v4
       with:
         version: latest
-        args: release --rm-dist --timeout 60m
+        args: release --clean --timeout 60m
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         TAG: ${{ steps.vars.outputs.version_tag }}

.github/workflows/release_published.yml

@@ -10,7 +10,8 @@ jobs:
     name: Release Published
     strategy:
       matrix:
-        os: [ ubuntu-latest ]
+        os: 
+          - ubuntu-latest
     runs-on: ${{ matrix.os }}
 
     steps:

.gitignore

@@ -11,6 +11,7 @@ Caddyfile.*
 # build artifacts and helpers
 cmd/caddy/caddy
 cmd/caddy/caddy.exe
+cmd/caddy/tmp/*.exe
 
 # mac specific
 .DS_Store

.golangci.yml

@@ -7,7 +7,6 @@ linters:
   disable-all: true
   enable:
     - bodyclose
-    - deadcode
     - errcheck
     - gofmt
     - goimports
@@ -18,11 +17,9 @@ linters:
     - misspell
     - prealloc
     - staticcheck
-    - structcheck
     - typecheck
     - unconvert
     - unused
-    - varcheck
   # these are implicitly disabled:
   # - asciicheck
   # - depguard

.goreleaser.yml

@@ -4,7 +4,9 @@ before:
     # This is so we can run goreleaser on tag without Git complaining of being dirty. The main.go in cmd/caddy directory 
     # cannot be built within that directory due to changes necessary for the build causing Git to be dirty, which
     # subsequently causes gorleaser to refuse running.
-    - rm -rf caddy-build caddy-dist
+    - rm -rf caddy-build caddy-dist vendor
+    # vendor Caddy deps
+    - go mod vendor
     - mkdir -p caddy-build
     - cp cmd/caddy/main.go caddy-build/main.go
     - /bin/sh -c 'cd ./caddy-build && go mod init caddy'
@@ -14,6 +16,8 @@ before:
     # as of Go 1.16, `go` commands no longer automatically change go.{mod,sum}. We now have to explicitly
     # run `go mod tidy`. The `/bin/sh -c '...'` is because goreleaser can't find cd in PATH without shell invocation.
     - /bin/sh -c 'cd ./caddy-build && go mod tidy'
+    # vendor the deps of the prepared to-build module
+    - /bin/sh -c 'cd ./caddy-build && go mod vendor'
     - git clone --depth 1 https://github.com/caddyserver/dist caddy-dist
     - mkdir -p caddy-dist/man
     - go mod download
@@ -71,7 +75,7 @@ signs:
   - cmd: cosign
     signature: "${artifact}.sig"
     certificate: '{{ trimsuffix (trimsuffix .Env.artifact ".zip") ".tar.gz" }}.pem'
-    args: ["sign-blob", "--output-signature=${signature}", "--output-certificate", "${certificate}", "${artifact}"]
+    args: ["sign-blob", "--yes", "--output-signature=${signature}", "--output-certificate", "${certificate}", "${artifact}"]
     artifacts: all
 
 sboms:
@@ -89,7 +93,8 @@ sboms:
     args: ["$artifact", "--file", "${document}", "--output", "cyclonedx-json"]
 
 archives:
-  - format_overrides:
+  - id: default
+    format_overrides:
       - goos: windows
         format: zip
     name_template: >-
@@ -100,6 +105,33 @@ archives:
       {{- with .Arm }}v{{ . }}{{ end }}
       {{- with .Mips }}_{{ . }}{{ end }}
       {{- if not (eq .Amd64 "v1") }}{{ .Amd64 }}{{ end }}
+  
+  # packge the 'caddy-build' directory into a tarball,
+  # allowing users to build the exact same set of files as ours.
+  - id: source
+    meta: true
+    name_template: "{{ .ProjectName }}_{{ .Version }}_buildable-artifact"
+    files:
+      - src: LICENSE
+        dst: ./LICENSE
+      - src: README.md
+        dst: ./README.md
+      - src: AUTHORS
+        dst: ./AUTHORS
+      - src: ./caddy-build
+        dst: ./
+
+source:
+  enabled: true
+  name_template: '{{ .ProjectName }}_{{ .Version }}_src'
+  format: 'tar.gz'
+
+  # Additional files/template/globs you want to add to the source archive.
+  #
+  # Default: empty.
+  files:
+    - vendor
+
 
 checksum:
   algorithm: sha512

README.md

@@ -70,7 +70,7 @@
 - **Stays up when other servers go down** due to TLS/OCSP/certificate-related issues
 - **Production-ready** after serving trillions of requests and managing millions of TLS certificates
 - **Scales to hundreds of thousands of sites** as proven in production
-- **HTTP/1.1, HTTP/2, and HTTP/3** supported all by default
+- **HTTP/1.1, HTTP/2, and HTTP/3** all supported by default
 - **Highly extensible** [modular architecture](https://caddyserver.com/docs/architecture) lets Caddy do anything without bloat
 - **Runs anywhere** with **no external dependencies** (not even libc)
 - Written in Go, a language with higher **memory safety guarantees** than other servers
@@ -87,7 +87,7 @@ See [our online documentation](https://caddyserver.com/docs/install) for other i
 
 Requirements:
 
-- [Go 1.18 or newer](https://golang.org/dl/)
+- [Go 1.20 or newer](https://golang.org/dl/)
 
 ### For development
 

admin.go

@@ -71,6 +71,11 @@ type AdminConfig struct {
 	// parsed by Caddy. Accepts placeholders.
 	// Default: the value of the `CADDY_ADMIN` environment variable,
 	// or `localhost:2019` otherwise.
+	//
+	// Remember: When changing this value through a config reload,
+	// be sure to use the `--address` CLI flag to specify the current
+	// admin address if the currently-running admin endpoint is not
+	// the default address.
 	Listen string `json:"listen,omitempty"`
 
 	// If true, CORS headers will be emitted, and requests to the
@@ -313,7 +318,32 @@ func (admin AdminConfig) allowedOrigins(addr NetworkAddress) []*url.URL {
 				// messages. If the requested URI does not include an Internet host
 				// name for the service being requested, then the Host header field MUST
 				// be given with an empty value."
+				//
+				// UPDATE July 2023: Go broke this by patching a minor security bug in 1.20.6.
+				// Understandable, but frustrating. See:
+				// https://github.com/golang/go/issues/60374
+				// See also the discussion here:
+				// https://github.com/golang/go/issues/61431
+				//
+				// We can no longer conform to RFC 2616 Section 14.26 from either Go or curl
+				// in purity. (Curl allowed no host between 7.40 and 7.50, but now requires a
+				// bogus host; see https://superuser.com/a/925610.) If we disable Host/Origin
+				// security checks, the infosec community assures me that it is secure to do
+				// so, because:
+				// 1) Browsers do not allow access to unix sockets
+				// 2) DNS is irrelevant to unix sockets
+				//
+				// I am not quite ready to trust either of those external factors, so instead
+				// of disabling Host/Origin checks, we now allow specific Host values when
+				// accessing the admin endpoint over unix sockets. I definitely don't trust
+				// DNS (e.g. I don't trust 'localhost' to always resolve to the local host),
+				// and IP shouldn't even be used, but if it is for some reason, I think we can
+				// at least be reasonably assured that 127.0.0.1 and ::1 route to the local
+				// machine, meaning that a hypothetical browser origin would have to be on the
+				// local machine as well.
 				uniqueOrigins[""] = struct{}{}
+				uniqueOrigins["127.0.0.1"] = struct{}{}
+				uniqueOrigins["::1"] = struct{}{}
 			} else {
 				uniqueOrigins[net.JoinHostPort("localhost", addr.port())] = struct{}{}
 				uniqueOrigins[net.JoinHostPort("::1", addr.port())] = struct{}{}
@@ -1011,9 +1041,9 @@ func handleConfigID(w http.ResponseWriter, r *http.Request) error {
 	id := parts[2]
 
 	// map the ID to the expanded path
-	currentCtxMu.RLock()
+	rawCfgMu.RLock()
 	expanded, ok := rawCfgIndex[id]
-	defer currentCtxMu.RUnlock()
+	rawCfgMu.RUnlock()
 	if !ok {
 		return APIError{
 			HTTPStatus: http.StatusNotFound,

caddy.go

@@ -156,8 +156,8 @@ func changeConfig(method, path string, input []byte, ifMatchHeader string, force
 		return fmt.Errorf("method not allowed")
 	}
 
-	currentCtxMu.Lock()
-	defer currentCtxMu.Unlock()
+	rawCfgMu.Lock()
+	defer rawCfgMu.Unlock()
 
 	if ifMatchHeader != "" {
 		// expect the first and last character to be quotes
@@ -257,8 +257,8 @@ func changeConfig(method, path string, input []byte, ifMatchHeader string, force
 // readConfig traverses the current config to path
 // and writes its JSON encoding to out.
 func readConfig(path string, out io.Writer) error {
-	currentCtxMu.RLock()
-	defer currentCtxMu.RUnlock()
+	rawCfgMu.RLock()
+	defer rawCfgMu.RUnlock()
 	return unsyncedConfigAccess(http.MethodGet, path, nil, out)
 }
 
@@ -305,7 +305,7 @@ func indexConfigObjects(ptr any, configPath string, index map[string]string) err
 // it as the new config, replacing any other current config.
 // It does NOT update the raw config state, as this is a
 // lower-level function; most callers will want to use Load
-// instead. A write lock on currentCtxMu is required! If
+// instead. A write lock on rawCfgMu is required! If
 // allowPersist is false, it will not be persisted to disk,
 // even if it is configured to.
 func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
@@ -314,7 +314,7 @@ func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
 	strippedCfgJSON := RemoveMetaFields(cfgJSON)
 
 	var newCfg *Config
-	err := strictUnmarshalJSON(strippedCfgJSON, &newCfg)
+	err := StrictUnmarshalJSON(strippedCfgJSON, &newCfg)
 	if err != nil {
 		return err
 	}
@@ -340,8 +340,10 @@ func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
 	}
 
 	// swap old context (including its config) with the new one
+	currentCtxMu.Lock()
 	oldCtx := currentCtx
 	currentCtx = ctx
+	currentCtxMu.Unlock()
 
 	// Stop, Cleanup each old app
 	unsyncedStop(oldCtx)
@@ -627,22 +629,35 @@ type ConfigLoader interface {
 // stop the others. Stop should only be called
 // if not replacing with a new config.
 func Stop() error {
+	currentCtxMu.RLock()
+	ctx := currentCtx
+	currentCtxMu.RUnlock()
+
+	rawCfgMu.Lock()
+	unsyncedStop(ctx)
+
 	currentCtxMu.Lock()
-	defer currentCtxMu.Unlock()
-	unsyncedStop(currentCtx)
 	currentCtx = Context{}
+	currentCtxMu.Unlock()
+
 	rawCfgJSON = nil
 	rawCfgIndex = nil
 	rawCfg[rawConfigKey] = nil
+	rawCfgMu.Unlock()
+
 	return nil
 }
 
-// unsyncedStop stops cfg from running, but has
-// no locking around cfg. It is a no-op if cfg is
-// nil. If any app returns an error when stopping,
+// unsyncedStop stops ctx from running, but has
+// no locking around ctx. It is a no-op if ctx has a
+// nil cfg. If any app returns an error when stopping,
 // it is logged and the function continues stopping
 // the next app. This function assumes all apps in
-// cfg were successfully started first.
+// ctx were successfully started first.
+//
+// A lock on rawCfgMu is required, even though this
+// function does not access rawCfg, that lock
+// synchronizes the stop/start of apps.
 func unsyncedStop(ctx Context) {
 	if ctx.cfg == nil {
 		return
@@ -969,14 +984,12 @@ type CtxKey string
 
 // This group of variables pertains to the current configuration.
 var (
-	// currentCtxMu protects everything in this var block.
-	currentCtxMu sync.RWMutex
-
 	// currentCtx is the root context for the currently-running
 	// configuration, which can be accessed through this value.
 	// If the Config contained in this value is not nil, then
 	// a config is currently active/running.
-	currentCtx Context
+	currentCtx   Context
+	currentCtxMu sync.RWMutex
 
 	// rawCfg is the current, generic-decoded configuration;
 	// we initialize it as a map with one field ("config")
@@ -994,6 +1007,10 @@ var (
 	// rawCfgIndex is the map of user-assigned ID to expanded
 	// path, for converting /id/ paths to /config/ paths.
 	rawCfgIndex map[string]string
+
+	// rawCfgMu protects all the rawCfg fields and also
+	// essentially synchronizes config changes/reloads.
+	rawCfgMu sync.RWMutex
 )
 
 // errSameConfig is returned if the new config is the same

caddyconfig/caddyfile/adapter.go

@@ -88,7 +88,7 @@ func FormattingDifference(filename string, body []byte) (caddyconfig.Warning, bo
 	return caddyconfig.Warning{
 		File:    filename,
 		Line:    line,
-		Message: "Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies",
+		Message: "Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies",
 	}, true
 }
 

caddyconfig/caddyfile/dispenser.go

@@ -101,12 +101,12 @@ func (d *Dispenser) nextOnSameLine() bool {
 		d.cursor++
 		return true
 	}
-	if d.cursor >= len(d.tokens) {
+	if d.cursor >= len(d.tokens)-1 {
 		return false
 	}
-	if d.cursor < len(d.tokens)-1 &&
-		d.tokens[d.cursor].File == d.tokens[d.cursor+1].File &&
-		d.tokens[d.cursor].Line+d.numLineBreaks(d.cursor) == d.tokens[d.cursor+1].Line {
+	curr := d.tokens[d.cursor]
+	next := d.tokens[d.cursor+1]
+	if !isNextOnNewLine(curr, next) {
 		d.cursor++
 		return true
 	}
@@ -122,12 +122,12 @@ func (d *Dispenser) NextLine() bool {
 		d.cursor++
 		return true
 	}
-	if d.cursor >= len(d.tokens) {
+	if d.cursor >= len(d.tokens)-1 {
 		return false
 	}
-	if d.cursor < len(d.tokens)-1 &&
-		(d.tokens[d.cursor].File != d.tokens[d.cursor+1].File ||
-			d.tokens[d.cursor].Line+d.numLineBreaks(d.cursor) < d.tokens[d.cursor+1].Line) {
+	curr := d.tokens[d.cursor]
+	next := d.tokens[d.cursor+1]
+	if isNextOnNewLine(curr, next) {
 		d.cursor++
 		return true
 	}
@@ -203,14 +203,17 @@ func (d *Dispenser) Val() string {
 }
 
 // ValRaw gets the raw text of the current token (including quotes).
+// If the token was a heredoc, then the delimiter is not included,
+// because that is not relevant to any unmarshaling logic at this time.
 // If there is no token loaded, it returns empty string.
 func (d *Dispenser) ValRaw() string {
 	if d.cursor < 0 || d.cursor >= len(d.tokens) {
 		return ""
 	}
 	quote := d.tokens[d.cursor].wasQuoted
-	if quote > 0 {
-		return string(quote) + d.tokens[d.cursor].Text + string(quote) // string literal
+	if quote > 0 && quote != '<' {
+		// string literal
+		return string(quote) + d.tokens[d.cursor].Text + string(quote)
 	}
 	return d.tokens[d.cursor].Text
 }
@@ -396,7 +399,7 @@ func (d *Dispenser) ArgErr() error {
 // SyntaxErr creates a generic syntax error which explains what was
 // found and what was expected.
 func (d *Dispenser) SyntaxErr(expected string) error {
-	msg := fmt.Sprintf("%s:%d - Syntax error: Unexpected token '%s', expecting '%s'", d.File(), d.Line(), d.Val(), expected)
+	msg := fmt.Sprintf("%s:%d - Syntax error: Unexpected token '%s', expecting '%s', import chain: ['%s']", d.File(), d.Line(), d.Val(), expected, strings.Join(d.Token().imports, "','"))
 	return errors.New(msg)
 }
 
@@ -418,7 +421,7 @@ func (d *Dispenser) Errf(format string, args ...any) error {
 
 // WrapErr takes an existing error and adds the Caddyfile file and line number.
 func (d *Dispenser) WrapErr(err error) error {
-	return fmt.Errorf("%s:%d - Error during parsing: %w", d.File(), d.Line(), err)
+	return fmt.Errorf("%s:%d - Error during parsing: %w, import chain: ['%s']", d.File(), d.Line(), err, strings.Join(d.Token().imports, "','"))
 }
 
 // Delete deletes the current token and returns the updated slice
@@ -438,14 +441,14 @@ func (d *Dispenser) Delete() []Token {
 	return d.tokens
 }
 
-// numLineBreaks counts how many line breaks are in the token
-// value given by the token index tknIdx. It returns 0 if the
-// token does not exist or there are no line breaks.
-func (d *Dispenser) numLineBreaks(tknIdx int) int {
-	if tknIdx < 0 || tknIdx >= len(d.tokens) {
-		return 0
+// DeleteN is the same as Delete, but can delete many tokens at once.
+// If there aren't N tokens available to delete, none are deleted.
+func (d *Dispenser) DeleteN(amount int) []Token {
+	if amount > 0 && d.cursor >= (amount-1) && d.cursor <= len(d.tokens)-1 {
+		d.tokens = append(d.tokens[:d.cursor-(amount-1)], d.tokens[d.cursor+1:]...)
+		d.cursor -= amount
 	}
-	return strings.Count(d.tokens[tknIdx].Text, "\n")
+	return d.tokens
 }
 
 // isNewLine determines whether the current token is on a different
@@ -461,25 +464,7 @@ func (d *Dispenser) isNewLine() bool {
 
 	prev := d.tokens[d.cursor-1]
 	curr := d.tokens[d.cursor]
-
-	// If the previous token is from a different file,
-	// we can assume it's from a different line
-	if prev.File != curr.File {
-		return true
-	}
-
-	// The previous token may contain line breaks if
-	// it was quoted and spanned multiple lines. e.g:
-	//
-	// dir "foo
-	//   bar
-	//   baz"
-	prevLineBreaks := d.numLineBreaks(d.cursor - 1)
-
-	// If the previous token (incl line breaks) ends
-	// on a line earlier than the current token,
-	// then the current token is on a new line
-	return prev.Line+prevLineBreaks < curr.Line
+	return isNextOnNewLine(prev, curr)
 }
 
 // isNextOnNewLine determines whether the current token is on a different
@@ -495,23 +480,5 @@ func (d *Dispenser) isNextOnNewLine() bool {
 
 	curr := d.tokens[d.cursor]
 	next := d.tokens[d.cursor+1]
-
-	// If the next token is from a different file,
-	// we can assume it's from a different line
-	if curr.File != next.File {
-		return true
-	}
-
-	// The current token may contain line breaks if
-	// it was quoted and spanned multiple lines. e.g:
-	//
-	// dir "foo
-	//   bar
-	//   baz"
-	currLineBreaks := d.numLineBreaks(d.cursor)
-
-	// If the current token (incl line breaks) ends
-	// on a line earlier than the next token,
-	// then the next token is on a new line
-	return curr.Line+currLineBreaks < next.Line
+	return isNextOnNewLine(curr, next)
 }

caddyconfig/caddyfile/importargs.go

@@ -0,0 +1,152 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"regexp"
+	"strconv"
+	"strings"
+
+	"github.com/caddyserver/caddy/v2"
+	"go.uber.org/zap"
+)
+
+// parseVariadic determines if the token is a variadic placeholder,
+// and if so, determines the index range (start/end) of args to use.
+// Returns a boolean signaling whether a variadic placeholder was found,
+// and the start and end indices.
+func parseVariadic(token Token, argCount int) (bool, int, int) {
+	if !strings.HasPrefix(token.Text, "{args[") {
+		return false, 0, 0
+	}
+	if !strings.HasSuffix(token.Text, "]}") {
+		return false, 0, 0
+	}
+
+	argRange := strings.TrimSuffix(strings.TrimPrefix(token.Text, "{args["), "]}")
+	if argRange == "" {
+		caddy.Log().Named("caddyfile").Warn(
+			"Placeholder "+token.Text+" cannot have an empty index",
+			zap.String("file", token.File+":"+strconv.Itoa(token.Line)), zap.Strings("import_chain", token.imports))
+		return false, 0, 0
+	}
+
+	start, end, found := strings.Cut(argRange, ":")
+
+	// If no ":" delimiter is found, this is not a variadic.
+	// The replacer will pick this up.
+	if !found {
+		return false, 0, 0
+	}
+
+	var (
+		startIndex = 0
+		endIndex   = argCount
+		err        error
+	)
+	if start != "" {
+		startIndex, err = strconv.Atoi(start)
+		if err != nil {
+			caddy.Log().Named("caddyfile").Warn(
+				"Variadic placeholder "+token.Text+" has an invalid start index",
+				zap.String("file", token.File+":"+strconv.Itoa(token.Line)), zap.Strings("import_chain", token.imports))
+			return false, 0, 0
+		}
+	}
+	if end != "" {
+		endIndex, err = strconv.Atoi(end)
+		if err != nil {
+			caddy.Log().Named("caddyfile").Warn(
+				"Variadic placeholder "+token.Text+" has an invalid end index",
+				zap.String("file", token.File+":"+strconv.Itoa(token.Line)), zap.Strings("import_chain", token.imports))
+			return false, 0, 0
+		}
+	}
+
+	// bound check
+	if startIndex < 0 || startIndex > endIndex || endIndex > argCount {
+		caddy.Log().Named("caddyfile").Warn(
+			"Variadic placeholder "+token.Text+" indices are out of bounds, only "+strconv.Itoa(argCount)+" argument(s) exist",
+			zap.String("file", token.File+":"+strconv.Itoa(token.Line)), zap.Strings("import_chain", token.imports))
+		return false, 0, 0
+	}
+	return true, startIndex, endIndex
+}
+
+// makeArgsReplacer prepares a Replacer which can replace
+// non-variadic args placeholders in imported tokens.
+func makeArgsReplacer(args []string) *caddy.Replacer {
+	repl := caddy.NewEmptyReplacer()
+	repl.Map(func(key string) (any, bool) {
+		// TODO: Remove the deprecated {args.*} placeholder
+		// support at some point in the future
+		if matches := argsRegexpIndexDeprecated.FindStringSubmatch(key); len(matches) > 0 {
+			// What's matched may be a substring of the key
+			if matches[0] != key {
+				return nil, false
+			}
+
+			value, err := strconv.Atoi(matches[1])
+			if err != nil {
+				caddy.Log().Named("caddyfile").Warn(
+					"Placeholder {args." + matches[1] + "} has an invalid index")
+				return nil, false
+			}
+			if value >= len(args) {
+				caddy.Log().Named("caddyfile").Warn(
+					"Placeholder {args." + matches[1] + "} index is out of bounds, only " + strconv.Itoa(len(args)) + " argument(s) exist")
+				return nil, false
+			}
+			caddy.Log().Named("caddyfile").Warn(
+				"Placeholder {args." + matches[1] + "} deprecated, use {args[" + matches[1] + "]} instead")
+			return args[value], true
+		}
+
+		// Handle args[*] form
+		if matches := argsRegexpIndex.FindStringSubmatch(key); len(matches) > 0 {
+			// What's matched may be a substring of the key
+			if matches[0] != key {
+				return nil, false
+			}
+
+			if strings.Contains(matches[1], ":") {
+				caddy.Log().Named("caddyfile").Warn(
+					"Variadic placeholder {args[" + matches[1] + "]} must be a token on its own")
+				return nil, false
+			}
+			value, err := strconv.Atoi(matches[1])
+			if err != nil {
+				caddy.Log().Named("caddyfile").Warn(
+					"Placeholder {args[" + matches[1] + "]} has an invalid index")
+				return nil, false
+			}
+			if value >= len(args) {
+				caddy.Log().Named("caddyfile").Warn(
+					"Placeholder {args[" + matches[1] + "]} index is out of bounds, only " + strconv.Itoa(len(args)) + " argument(s) exist")
+				return nil, false
+			}
+			return args[value], true
+		}
+
+		// Not an args placeholder, ignore
+		return nil, false
+	})
+	return repl
+}
+
+var (
+	argsRegexpIndexDeprecated = regexp.MustCompile(`args\.(.+)`)
+	argsRegexpIndex           = regexp.MustCompile(`args\[(.+)]`)
+)

caddyconfig/caddyfile/lexer.go

@@ -17,7 +17,10 @@ package caddyfile
 import (
 	"bufio"
 	"bytes"
+	"fmt"
 	"io"
+	"regexp"
+	"strings"
 	"unicode"
 )
 
@@ -35,15 +38,41 @@ type (
 
 	// Token represents a single parsable unit.
 	Token struct {
-		File        string
-		Line        int
-		Text        string
-		wasQuoted   rune // enclosing quote character, if any
-		inSnippet   bool
-		snippetName string
+		File          string
+		imports       []string
+		Line          int
+		Text          string
+		wasQuoted     rune // enclosing quote character, if any
+		heredocMarker string
+		snippetName   string
 	}
 )
 
+// Tokenize takes bytes as input and lexes it into
+// a list of tokens that can be parsed as a Caddyfile.
+// Also takes a filename to fill the token's File as
+// the source of the tokens, which is important to
+// determine relative paths for `import` directives.
+func Tokenize(input []byte, filename string) ([]Token, error) {
+	l := lexer{}
+	if err := l.load(bytes.NewReader(input)); err != nil {
+		return nil, err
+	}
+	var tokens []Token
+	for {
+		found, err := l.next()
+		if err != nil {
+			return nil, err
+		}
+		if !found {
+			break
+		}
+		l.token.File = filename
+		tokens = append(tokens, l.token)
+	}
+	return tokens, nil
+}
+
 // load prepares the lexer to scan an input for tokens.
 // It discards any leading byte order mark.
 func (l *lexer) load(input io.Reader) error {
@@ -75,28 +104,93 @@ func (l *lexer) load(input io.Reader) error {
 // may be escaped. The rest of the line is skipped
 // if a "#" character is read in. Returns true if
 // a token was loaded; false otherwise.
-func (l *lexer) next() bool {
+func (l *lexer) next() (bool, error) {
 	var val []rune
-	var comment, quoted, btQuoted, escaped bool
+	var comment, quoted, btQuoted, inHeredoc, heredocEscaped, escaped bool
+	var heredocMarker string
 
 	makeToken := func(quoted rune) bool {
 		l.token.Text = string(val)
 		l.token.wasQuoted = quoted
+		l.token.heredocMarker = heredocMarker
 		return true
 	}
 
 	for {
+		// Read a character in; if err then if we had
+		// read some characters, make a token. If we
+		// reached EOF, then no more tokens to read.
+		// If no EOF, then we had a problem.
 		ch, _, err := l.reader.ReadRune()
 		if err != nil {
 			if len(val) > 0 {
-				return makeToken(0)
+				if inHeredoc {
+					return false, fmt.Errorf("incomplete heredoc <<%s on line #%d, expected ending marker %s", heredocMarker, l.line+l.skippedLines, heredocMarker)
+				}
+
+				return makeToken(0), nil
 			}
 			if err == io.EOF {
-				return false
+				return false, nil
 			}
-			panic(err)
+			return false, err
 		}
 
+		// detect whether we have the start of a heredoc
+		if !inHeredoc && !heredocEscaped && len(val) > 1 && string(val[:2]) == "<<" {
+			if ch == '<' {
+				return false, fmt.Errorf("too many '<' for heredoc on line #%d; only use two, for example <<END", l.line)
+			}
+			if ch == '\r' {
+				continue
+			}
+			// after hitting a newline, we know that the heredoc marker
+			// is the characters after the two << and the newline.
+			// we reset the val because the heredoc is syntax we don't
+			// want to keep.
+			if ch == '\n' {
+				heredocMarker = string(val[2:])
+				if !heredocMarkerRegexp.Match([]byte(heredocMarker)) {
+					return false, fmt.Errorf("heredoc marker on line #%d must contain only alpha-numeric characters, dashes and underscores; got '%s'", l.line, heredocMarker)
+				}
+
+				inHeredoc = true
+				l.skippedLines++
+				val = nil
+				continue
+			}
+			val = append(val, ch)
+			continue
+		}
+
+		// if we're in a heredoc, all characters are read as-is
+		if inHeredoc {
+			val = append(val, ch)
+
+			if ch == '\n' {
+				l.skippedLines++
+			}
+
+			// check if we're done, i.e. that the last few characters are the marker
+			if len(val) > len(heredocMarker) && heredocMarker == string(val[len(val)-len(heredocMarker):]) {
+				// set the final value
+				val, err = l.finalizeHeredoc(val, heredocMarker)
+				if err != nil {
+					return false, err
+				}
+
+				// set the line counter, and make the token
+				l.line += l.skippedLines
+				l.skippedLines = 0
+				return makeToken('<'), nil
+			}
+
+			// stay in the heredoc until we find the ending marker
+			continue
+		}
+
+		// track whether we found an escape '\' for the next
+		// iteration to be contextually aware
 		if !escaped && !btQuoted && ch == '\\' {
 			escaped = true
 			continue
@@ -111,26 +205,29 @@ func (l *lexer) next() bool {
 				}
 				escaped = false
 			} else {
-				if quoted && ch == '"' {
-					return makeToken('"')
-				}
-				if btQuoted && ch == '`' {
-					return makeToken('`')
+				if (quoted && ch == '"') || (btQuoted && ch == '`') {
+					return makeToken(ch), nil
 				}
 			}
+			// allow quoted text to wrap continue on multiple lines
 			if ch == '\n' {
 				l.line += 1 + l.skippedLines
 				l.skippedLines = 0
 			}
+			// collect this character as part of the quoted token
 			val = append(val, ch)
 			continue
 		}
 
 		if unicode.IsSpace(ch) {
+			// ignore CR altogether, we only actually care about LF (\n)
 			if ch == '\r' {
 				continue
 			}
+			// end of the line
 			if ch == '\n' {
+				// newlines can be escaped to chain arguments
+				// onto multiple lines; else, increment the line count
 				if escaped {
 					l.skippedLines++
 					escaped = false
@@ -138,14 +235,18 @@ func (l *lexer) next() bool {
 					l.line += 1 + l.skippedLines
 					l.skippedLines = 0
 				}
+				// comments (#) are single-line only
 				comment = false
 			}
+			// any kind of space means we're at the end of this token
 			if len(val) > 0 {
-				return makeToken(0)
+				return makeToken(0), nil
 			}
 			continue
 		}
 
+		// comments must be at the start of a token,
+		// in other words, preceded by space or newline
 		if ch == '#' && len(val) == 0 {
 			comment = true
 		}
@@ -166,7 +267,12 @@ func (l *lexer) next() bool {
 		}
 
 		if escaped {
-			val = append(val, '\\')
+			// allow escaping the first < to skip the heredoc syntax
+			if ch == '<' {
+				heredocEscaped = true
+			} else {
+				val = append(val, '\\')
+			}
 			escaped = false
 		}
 
@@ -174,24 +280,86 @@ func (l *lexer) next() bool {
 	}
 }
 
-// Tokenize takes bytes as input and lexes it into
-// a list of tokens that can be parsed as a Caddyfile.
-// Also takes a filename to fill the token's File as
-// the source of the tokens, which is important to
-// determine relative paths for `import` directives.
-func Tokenize(input []byte, filename string) ([]Token, error) {
-	l := lexer{}
-	if err := l.load(bytes.NewReader(input)); err != nil {
-		return nil, err
+// finalizeHeredoc takes the runes read as the heredoc text and the marker,
+// and processes the text to strip leading whitespace, returning the final
+// value without the leading whitespace.
+func (l *lexer) finalizeHeredoc(val []rune, marker string) ([]rune, error) {
+	stringVal := string(val)
+
+	// find the last newline of the heredoc, which is where the contents end
+	lastNewline := strings.LastIndex(stringVal, "\n")
+
+	// collapse the content, then split into separate lines
+	lines := strings.Split(stringVal[:lastNewline+1], "\n")
+
+	// figure out how much whitespace we need to strip from the front of every line
+	// by getting the string that precedes the marker, on the last line
+	paddingToStrip := stringVal[lastNewline+1 : len(stringVal)-len(marker)]
+
+	// iterate over each line and strip the whitespace from the front
+	var out string
+	for lineNum, lineText := range lines[:len(lines)-1] {
+		// find an exact match for the padding
+		index := strings.Index(lineText, paddingToStrip)
+
+		// if the padding doesn't match exactly at the start then we can't safely strip
+		if index != 0 {
+			return nil, fmt.Errorf("mismatched leading whitespace in heredoc <<%s on line #%d [%s], expected whitespace [%s] to match the closing marker", marker, l.line+lineNum+1, lineText, paddingToStrip)
+		}
+
+		// strip, then append the line, with the newline, to the output.
+		// also removes all "\r" because Windows.
+		out += strings.ReplaceAll(lineText[len(paddingToStrip):]+"\n", "\r", "")
 	}
-	var tokens []Token
-	for l.next() {
-		l.token.File = filename
-		tokens = append(tokens, l.token)
+
+	// Remove the trailing newline from the loop
+	if len(out) > 0 && out[len(out)-1] == '\n' {
+		out = out[:len(out)-1]
 	}
-	return tokens, nil
+
+	// return the final value
+	return []rune(out), nil
 }
 
 func (t Token) Quoted() bool {
 	return t.wasQuoted > 0
 }
+
+// NumLineBreaks counts how many line breaks are in the token text.
+func (t Token) NumLineBreaks() int {
+	lineBreaks := strings.Count(t.Text, "\n")
+	if t.wasQuoted == '<' {
+		// heredocs have an extra linebreak because the opening
+		// delimiter is on its own line and is not included in the
+		// token Text itself, and the trailing newline is removed.
+		lineBreaks += 2
+	}
+	return lineBreaks
+}
+
+var heredocMarkerRegexp = regexp.MustCompile("^[A-Za-z0-9_-]+$")
+
+// isNextOnNewLine tests whether t2 is on a different line from t1
+func isNextOnNewLine(t1, t2 Token) bool {
+	// If the second token is from a different file,
+	// we can assume it's from a different line
+	if t1.File != t2.File {
+		return true
+	}
+
+	// If the second token is from a different import chain,
+	// we can assume it's from a different line
+	if len(t1.imports) != len(t2.imports) {
+		return true
+	}
+	for i, im := range t1.imports {
+		if im != t2.imports[i] {
+			return true
+		}
+	}
+
+	// If the first token (incl line breaks) ends
+	// on a line earlier than the next token,
+	// then the second token is on a new line
+	return t1.Line+t1.NumLineBreaks() < t2.Line
+}

caddyconfig/caddyfile/lexer_test.go

@@ -18,13 +18,13 @@ import (
 	"testing"
 )
 
-type lexerTestCase struct {
-	input    []byte
-	expected []Token
-}
-
 func TestLexer(t *testing.T) {
-	testCases := []lexerTestCase{
+	testCases := []struct {
+		input        []byte
+		expected     []Token
+		expectErr    bool
+		errorMessage string
+	}{
 		{
 			input: []byte(`host:123`),
 			expected: []Token{
@@ -249,12 +249,178 @@ func TestLexer(t *testing.T) {
 				{Line: 1, Text: `quotes`},
 			},
 		},
+		{
+			input: []byte(`heredoc <<EOF
+content
+EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "content"},
+				{Line: 3, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<VERY-LONG-MARKER
+content
+VERY-LONG-MARKER same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "content"},
+				{Line: 3, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<EOF
+extra-newline
+
+EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "extra-newline\n"},
+				{Line: 4, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<EOF
+		EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: ""},
+				{Line: 2, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<EOF
+	content
+	EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "content"},
+				{Line: 3, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`prev-line
+	heredoc <<EOF
+		multi
+		line
+		content
+	EOF same-line-arg
+	next-line
+	`),
+			expected: []Token{
+				{Line: 1, Text: `prev-line`},
+				{Line: 2, Text: `heredoc`},
+				{Line: 2, Text: "\tmulti\n\tline\n\tcontent"},
+				{Line: 6, Text: `same-line-arg`},
+				{Line: 7, Text: `next-line`},
+			},
+		},
+		{
+			input: []byte(`heredoc <EOF
+	content
+	EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: `<EOF`},
+				{Line: 2, Text: `content`},
+				{Line: 3, Text: `EOF`},
+				{Line: 3, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<s
+			�
+			s
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "�"},
+			},
+		},
+		{
+			input: []byte("\u000Aheredoc \u003C\u003C\u0073\u0073\u000A\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F\u000A\u0073\u0073\u000A\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F\u000A\u00BF\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F"),
+			expected: []Token{
+				{
+					Line: 2,
+					Text: "heredoc",
+				},
+				{
+					Line: 2,
+					Text: "\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F",
+				},
+				{
+					Line: 5,
+					Text: "\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F",
+				},
+				{
+					Line: 6,
+					Text: "\u00BF\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F",
+				},
+			},
+		},
+		{
+			input: []byte(`heredoc <<HERE SAME LINE
+	content
+	HERE same-line-arg
+	`),
+			expectErr:    true,
+			errorMessage: "heredoc marker on line #1 must contain only alpha-numeric characters, dashes and underscores; got 'HERE SAME LINE'",
+		},
+		{
+			input: []byte(`heredoc <<<EOF
+	content
+	EOF same-line-arg
+	`),
+			expectErr:    true,
+			errorMessage: "too many '<' for heredoc on line #1; only use two, for example <<END",
+		},
+		{
+			input: []byte(`heredoc <<EOF
+	content
+	`),
+			expectErr:    true,
+			errorMessage: "incomplete heredoc <<EOF on line #3, expected ending marker EOF",
+		},
+		{
+			input: []byte(`heredoc <<EOF
+	content
+		EOF
+	`),
+			expectErr:    true,
+			errorMessage: "mismatched leading whitespace in heredoc <<EOF on line #2 [\tcontent], expected whitespace [\t\t] to match the closing marker",
+		},
+		{
+			input: []byte(`heredoc <<EOF
+        content
+		EOF
+	`),
+			expectErr:    true,
+			errorMessage: "mismatched leading whitespace in heredoc <<EOF on line #2 [        content], expected whitespace [\t\t] to match the closing marker",
+		},
 	}
 
 	for i, testCase := range testCases {
 		actual, err := Tokenize(testCase.input, "")
+		if testCase.expectErr {
+			if err == nil {
+				t.Fatalf("expected error, got actual: %v", actual)
+				continue
+			}
+			if err.Error() != testCase.errorMessage {
+				t.Fatalf("expected error '%v', got: %v", testCase.errorMessage, err)
+			}
+			continue
+		}
+
 		if err != nil {
-			t.Errorf("%v", err)
+			t.Fatalf("%v", err)
 		}
 		lexerCompare(t, i, testCase.expected, actual)
 	}
@@ -262,17 +428,17 @@ func TestLexer(t *testing.T) {
 
 func lexerCompare(t *testing.T, n int, expected, actual []Token) {
 	if len(expected) != len(actual) {
-		t.Errorf("Test case %d: expected %d token(s) but got %d", n, len(expected), len(actual))
+		t.Fatalf("Test case %d: expected %d token(s) but got %d", n, len(expected), len(actual))
 	}
 
 	for i := 0; i < len(actual) && i < len(expected); i++ {
 		if actual[i].Line != expected[i].Line {
-			t.Errorf("Test case %d token %d ('%s'): expected line %d but was line %d",
+			t.Fatalf("Test case %d token %d ('%s'): expected line %d but was line %d",
 				n, i, expected[i].Text, expected[i].Line, actual[i].Line)
 			break
 		}
 		if actual[i].Text != expected[i].Text {
-			t.Errorf("Test case %d token %d: expected text '%s' but was '%s'",
+			t.Fatalf("Test case %d token %d: expected text '%s' but was '%s'",
 				n, i, expected[i].Text, actual[i].Text)
 			break
 		}

caddyconfig/caddyfile/parse.go

@@ -20,7 +20,6 @@ import (
 	"io"
 	"os"
 	"path/filepath"
-	"strconv"
 	"strings"
 
 	"github.com/caddyserver/caddy/v2"
@@ -149,7 +148,6 @@ func (p *parser) begin() error {
 	}
 
 	err := p.addresses()
-
 	if err != nil {
 		return err
 	}
@@ -160,6 +158,25 @@ func (p *parser) begin() error {
 		return nil
 	}
 
+	if ok, name := p.isNamedRoute(); ok {
+		// named routes only have one key, the route name
+		p.block.Keys = []string{name}
+		p.block.IsNamedRoute = true
+
+		// we just need a dummy leading token to ease parsing later
+		nameToken := p.Token()
+		nameToken.Text = name
+
+		// get all the tokens from the block, including the braces
+		tokens, err := p.blockTokens(true)
+		if err != nil {
+			return err
+		}
+		tokens = append([]Token{nameToken}, tokens...)
+		p.block.Segments = []Segment{tokens}
+		return nil
+	}
+
 	if ok, name := p.isSnippet(); ok {
 		if p.definedSnippets == nil {
 			p.definedSnippets = map[string][]Token{}
@@ -168,16 +185,15 @@ func (p *parser) begin() error {
 			return p.Errf("redeclaration of previously declared snippet %s", name)
 		}
 		// consume all tokens til matched close brace
-		tokens, err := p.snippetTokens()
+		tokens, err := p.blockTokens(false)
 		if err != nil {
 			return err
 		}
 		// Just as we need to track which file the token comes from, we need to
-		// keep track of which snippets do the tokens come from. This is helpful
-		// in tracking import cycles across files/snippets by namespacing them. Without
-		// this we end up with false-positives in cycle-detection.
+		// keep track of which snippet the token comes from. This is helpful
+		// in tracking import cycles across files/snippets by namespacing them.
+		// Without this, we end up with false-positives in cycle-detection.
 		for k, v := range tokens {
-			v.inSnippet = true
 			v.snippetName = name
 			tokens[k] = v
 		}
@@ -198,7 +214,7 @@ func (p *parser) addresses() error {
 
 		// special case: import directive replaces tokens during parse-time
 		if tkn == "import" && p.isNewLine() {
-			err := p.doImport()
+			err := p.doImport(0)
 			if err != nil {
 				return err
 			}
@@ -298,7 +314,7 @@ func (p *parser) directives() error {
 
 		// special case: import directive replaces tokens during parse-time
 		if p.Val() == "import" {
-			err := p.doImport()
+			err := p.doImport(1)
 			if err != nil {
 				return err
 			}
@@ -324,7 +340,7 @@ func (p *parser) directives() error {
 // is on the token before where the import directive was. In
 // other words, call Next() to access the first token that was
 // imported.
-func (p *parser) doImport() error {
+func (p *parser) doImport(nesting int) error {
 	// syntax checks
 	if !p.NextArg() {
 		return p.ArgErr()
@@ -337,11 +353,8 @@ func (p *parser) doImport() error {
 	// grab remaining args as placeholder replacements
 	args := p.RemainingArgs()
 
-	// add args to the replacer
-	repl := caddy.NewEmptyReplacer()
-	for index, arg := range args {
-		repl.Set("args."+strconv.Itoa(index), arg)
-	}
+	// set up a replacer for non-variadic args replacement
+	repl := makeArgsReplacer(args)
 
 	// splice out the import directive and its arguments
 	// (2 tokens, plus the length of args)
@@ -417,7 +430,7 @@ func (p *parser) doImport() error {
 	}
 
 	nodeName := p.File()
-	if p.Token().inSnippet {
+	if p.Token().snippetName != "" {
 		nodeName += fmt.Sprintf(":%s", p.Token().snippetName)
 	}
 	p.importGraph.addNode(nodeName)
@@ -428,13 +441,69 @@ func (p *parser) doImport() error {
 	}
 
 	// copy the tokens so we don't overwrite p.definedSnippets
-	tokensCopy := make([]Token, len(importedTokens))
-	copy(tokensCopy, importedTokens)
+	tokensCopy := make([]Token, 0, len(importedTokens))
+
+	var (
+		maybeSnippet   bool
+		maybeSnippetId bool
+		index          int
+	)
 
 	// run the argument replacer on the tokens
-	for index, token := range tokensCopy {
-		token.Text = repl.ReplaceKnown(token.Text, "")
-		tokensCopy[index] = token
+	// golang for range slice return a copy of value
+	// similarly, append also copy value
+	for i, token := range importedTokens {
+		// update the token's imports to refer to import directive filename, line number and snippet name if there is one
+		if token.snippetName != "" {
+			token.imports = append(token.imports, fmt.Sprintf("%s:%d (import %s)", p.File(), p.Line(), token.snippetName))
+		} else {
+			token.imports = append(token.imports, fmt.Sprintf("%s:%d (import)", p.File(), p.Line()))
+		}
+
+		// naive way of determine snippets, as snippets definition can only follow name + block
+		// format, won't check for nesting correctness or any other error, that's what parser does.
+		if !maybeSnippet && nesting == 0 {
+			// first of the line
+			if i == 0 || isNextOnNewLine(tokensCopy[i-1], token) {
+				index = 0
+			} else {
+				index++
+			}
+
+			if index == 0 && len(token.Text) >= 3 && strings.HasPrefix(token.Text, "(") && strings.HasSuffix(token.Text, ")") {
+				maybeSnippetId = true
+			}
+		}
+
+		switch token.Text {
+		case "{":
+			nesting++
+			if index == 1 && maybeSnippetId && nesting == 1 {
+				maybeSnippet = true
+				maybeSnippetId = false
+			}
+		case "}":
+			nesting--
+			if nesting == 0 && maybeSnippet {
+				maybeSnippet = false
+			}
+		}
+
+		if maybeSnippet {
+			tokensCopy = append(tokensCopy, token)
+			continue
+		}
+
+		foundVariadic, startIndex, endIndex := parseVariadic(token, len(args))
+		if foundVariadic {
+			for _, arg := range args[startIndex:endIndex] {
+				token.Text = arg
+				tokensCopy = append(tokensCopy, token)
+			}
+		} else {
+			token.Text = repl.ReplaceKnown(token.Text, "")
+			tokensCopy = append(tokensCopy, token)
+		}
 	}
 
 	// splice the imported tokens in the place of the import statement
@@ -509,6 +578,9 @@ func (p *parser) directive() error {
 			if !p.isNextOnNewLine() && p.Token().wasQuoted == 0 {
 				return p.Err("Unexpected next token after '{' on same line")
 			}
+			if p.isNewLine() {
+				return p.Err("Unexpected '{' on a new line; did you mean to place the '{' on the previous line?")
+			}
 		} else if p.Val() == "{}" {
 			if p.isNextOnNewLine() && p.Token().wasQuoted == 0 {
 				return p.Err("Unexpected '{}' at end of line")
@@ -521,7 +593,7 @@ func (p *parser) directive() error {
 		} else if p.Val() == "}" && p.nesting == 0 {
 			return p.Err("Unexpected '}' because no matching opening brace")
 		} else if p.Val() == "import" && p.isNewLine() {
-			if err := p.doImport(); err != nil {
+			if err := p.doImport(1); err != nil {
 				return err
 			}
 			p.cursor-- // cursor is advanced when we continue, so roll back one more
@@ -562,6 +634,15 @@ func (p *parser) closeCurlyBrace() error {
 	return nil
 }
 
+func (p *parser) isNamedRoute() (bool, string) {
+	keys := p.block.Keys
+	// A named route block is a single key with parens, prefixed with &.
+	if len(keys) == 1 && strings.HasPrefix(keys[0], "&(") && strings.HasSuffix(keys[0], ")") {
+		return true, strings.TrimSuffix(keys[0][2:], ")")
+	}
+	return false, ""
+}
+
 func (p *parser) isSnippet() (bool, string) {
 	keys := p.block.Keys
 	// A snippet block is a single key with parens. Nothing else qualifies.
@@ -572,18 +653,24 @@ func (p *parser) isSnippet() (bool, string) {
 }
 
 // read and store everything in a block for later replay.
-func (p *parser) snippetTokens() ([]Token, error) {
-	// snippet must have curlies.
+func (p *parser) blockTokens(retainCurlies bool) ([]Token, error) {
+	// block must have curlies.
 	err := p.openCurlyBrace()
 	if err != nil {
 		return nil, err
 	}
-	nesting := 1 // count our own nesting in snippets
+	nesting := 1 // count our own nesting
 	tokens := []Token{}
+	if retainCurlies {
+		tokens = append(tokens, p.Token())
+	}
 	for p.Next() {
 		if p.Val() == "}" {
 			nesting--
 			if nesting == 0 {
+				if retainCurlies {
+					tokens = append(tokens, p.Token())
+				}
 				break
 			}
 		}
@@ -603,9 +690,10 @@ func (p *parser) snippetTokens() ([]Token, error) {
 // head of the server block with tokens, which are
 // grouped by segments.
 type ServerBlock struct {
-	HasBraces bool
-	Keys      []string
-	Segments  []Segment
+	HasBraces    bool
+	Keys         []string
+	Segments     []Segment
+	IsNamedRoute bool
 }
 
 // DispenseDirective returns a dispenser that contains

caddyconfig/caddyfile/parse_test.go

@@ -21,6 +21,88 @@ import (
 	"testing"
 )
 
+func TestParseVariadic(t *testing.T) {
+	var args = make([]string, 10)
+	for i, tc := range []struct {
+		input  string
+		result bool
+	}{
+		{
+			input:  "",
+			result: false,
+		},
+		{
+			input:  "{args[1",
+			result: false,
+		},
+		{
+			input:  "1]}",
+			result: false,
+		},
+		{
+			input:  "{args[:]}aaaaa",
+			result: false,
+		},
+		{
+			input:  "aaaaa{args[:]}",
+			result: false,
+		},
+		{
+			input:  "{args.}",
+			result: false,
+		},
+		{
+			input:  "{args.1}",
+			result: false,
+		},
+		{
+			input:  "{args[]}",
+			result: false,
+		},
+		{
+			input:  "{args[:]}",
+			result: true,
+		},
+		{
+			input:  "{args[:]}",
+			result: true,
+		},
+		{
+			input:  "{args[0:]}",
+			result: true,
+		},
+		{
+			input:  "{args[:0]}",
+			result: true,
+		},
+		{
+			input:  "{args[-1:]}",
+			result: false,
+		},
+		{
+			input:  "{args[:11]}",
+			result: false,
+		},
+		{
+			input:  "{args[10:0]}",
+			result: false,
+		},
+		{
+			input:  "{args[0:10]}",
+			result: true,
+		},
+	} {
+		token := Token{
+			File: "test",
+			Line: 1,
+			Text: tc.input,
+		}
+		if v, _, _ := parseVariadic(token, len(args)); v != tc.result {
+			t.Errorf("Test %d error expectation failed Expected: %t, got %t", i, tc.result, v)
+		}
+	}
+}
+
 func TestAllTokens(t *testing.T) {
 	input := []byte("a b c\nd e")
 	expected := []string{"a", "b", "c", "d", "e"}
@@ -211,6 +293,14 @@ func TestParseOneAndImport(t *testing.T) {
 		// Unexpected next token after '{' on same line
 		{`localhost
 		  dir1 { a b }`, true, []string{"localhost"}, []int{}},
+
+		// Unexpected '{' on a new line
+		{`localhost
+		dir1
+		{
+			a b
+		}`, true, []string{"localhost"}, []int{}},
+
 		// Workaround with quotes
 		{`localhost
 		  dir1 "{" a b "}"`, false, []string{"localhost"}, []int{5}},
@@ -628,6 +718,36 @@ func TestEnvironmentReplacement(t *testing.T) {
 	}
 }
 
+func TestImportReplacementInJSONWithBrace(t *testing.T) {
+	for i, test := range []struct {
+		args   []string
+		input  string
+		expect string
+	}{
+		{
+			args:   []string{"123"},
+			input:  "{args[0]}",
+			expect: "123",
+		},
+		{
+			args:   []string{"123"},
+			input:  `{"key":"{args[0]}"}`,
+			expect: `{"key":"123"}`,
+		},
+		{
+			args:   []string{"123", "123"},
+			input:  `{"key":[{args[0]},{args[1]}]}`,
+			expect: `{"key":[123,123]}`,
+		},
+	} {
+		repl := makeArgsReplacer(test.args)
+		actual := repl.ReplaceKnown(test.input, "")
+		if actual != test.expect {
+			t.Errorf("Test %d: Expected: '%s' but got '%s'", i, test.expect, actual)
+		}
+	}
+}
+
 func TestSnippets(t *testing.T) {
 	p := testParser(`
 		(common) {

caddyconfig/caddyfile/testdata/import_args0.txt

@@ -1 +1 @@
-{args.0}
+{args[0]}

caddyconfig/caddyfile/testdata/import_args1.txt

@@ -1 +1 @@
-{args.0} {args.1}
+{args[0]} {args[1]}

caddyconfig/httpcaddyfile/builtins.go

@@ -48,6 +48,7 @@ func init() {
 	RegisterHandlerDirective("route", parseRoute)
 	RegisterHandlerDirective("handle", parseHandle)
 	RegisterDirective("handle_errors", parseHandleErrors)
+	RegisterHandlerDirective("invoke", parseInvoke)
 	RegisterDirective("log", parseLog)
 	RegisterHandlerDirective("skip_log", parseSkipLog)
 }
@@ -764,9 +765,31 @@ func parseHandleErrors(h Helper) ([]ConfigValue, error) {
 	}, nil
 }
 
+// parseInvoke parses the invoke directive.
+func parseInvoke(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	h.Next() // consume directive
+	if !h.NextArg() {
+		return nil, h.ArgErr()
+	}
+	for h.Next() || h.NextBlock(0) {
+		return nil, h.ArgErr()
+	}
+
+	// remember that we're invoking this name
+	// to populate the server with these named routes
+	if h.State[namedRouteKey] == nil {
+		h.State[namedRouteKey] = map[string]struct{}{}
+	}
+	h.State[namedRouteKey].(map[string]struct{})[h.Val()] = struct{}{}
+
+	// return the handler
+	return &caddyhttp.Invoke{Name: h.Val()}, nil
+}
+
 // parseLog parses the log directive. Syntax:
 //
-//	log {
+//	log <logger_name> {
+//	    hostnames <hostnames...>
 //	    output <writer_module> ...
 //	    format <encoder_module> ...
 //	    level  <level>
@@ -787,11 +810,13 @@ func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue
 	var configValues []ConfigValue
 	for h.Next() {
 		// Logic below expects that a name is always present when a
-		// global option is being parsed.
-		var globalLogName string
+		// global option is being parsed; or an optional override
+		// is supported for access logs.
+		var logName string
+
 		if parseAsGlobalOption {
 			if h.NextArg() {
-				globalLogName = h.Val()
+				logName = h.Val()
 
 				// Only a single argument is supported.
 				if h.NextArg() {
@@ -802,26 +827,47 @@ func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue
 				// reference the default logger. See the
 				// setupNewDefault function in the logging
 				// package for where this is configured.
-				globalLogName = caddy.DefaultLoggerName
+				logName = caddy.DefaultLoggerName
 			}
 
 			// Verify this name is unused.
-			_, used := globalLogNames[globalLogName]
+			_, used := globalLogNames[logName]
 			if used {
-				return nil, h.Err("duplicate global log option for: " + globalLogName)
+				return nil, h.Err("duplicate global log option for: " + logName)
 			}
-			globalLogNames[globalLogName] = struct{}{}
+			globalLogNames[logName] = struct{}{}
 		} else {
-			// No arguments are supported for the server block log directive
+			// An optional override of the logger name can be provided;
+			// otherwise a default will be used, like "log0", "log1", etc.
 			if h.NextArg() {
-				return nil, h.ArgErr()
+				logName = h.Val()
+
+				// Only a single argument is supported.
+				if h.NextArg() {
+					return nil, h.ArgErr()
+				}
 			}
 		}
 
 		cl := new(caddy.CustomLog)
 
+		// allow overriding the current site block's hostnames for this logger;
+		// this is useful for setting up loggers per subdomain in a site block
+		// with a wildcard domain
+		customHostnames := []string{}
+
 		for h.NextBlock(0) {
 			switch h.Val() {
+			case "hostnames":
+				if parseAsGlobalOption {
+					return nil, h.Err("hostnames is not allowed in the log global options")
+				}
+				args := h.RemainingArgs()
+				if len(args) == 0 {
+					return nil, h.ArgErr()
+				}
+				customHostnames = append(customHostnames, args...)
+
 			case "output":
 				if !h.NextArg() {
 					return nil, h.ArgErr()
@@ -880,18 +926,16 @@ func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue
 				}
 
 			case "include":
-				// This configuration is only allowed in the global options
 				if !parseAsGlobalOption {
-					return nil, h.ArgErr()
+					return nil, h.Err("include is not allowed in the log directive")
 				}
 				for h.NextArg() {
 					cl.Include = append(cl.Include, h.Val())
 				}
 
 			case "exclude":
-				// This configuration is only allowed in the global options
 				if !parseAsGlobalOption {
-					return nil, h.ArgErr()
+					return nil, h.Err("exclude is not allowed in the log directive")
 				}
 				for h.NextArg() {
 					cl.Exclude = append(cl.Exclude, h.Val())
@@ -903,24 +947,34 @@ func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue
 		}
 
 		var val namedCustomLog
+		val.hostnames = customHostnames
+
+		isEmptyConfig := reflect.DeepEqual(cl, new(caddy.CustomLog))
+
 		// Skip handling of empty logging configs
-		if !reflect.DeepEqual(cl, new(caddy.CustomLog)) {
-			if parseAsGlobalOption {
-				// Use indicated name for global log options
-				val.name = globalLogName
-				val.log = cl
-			} else {
+
+		if parseAsGlobalOption {
+			// Use indicated name for global log options
+			val.name = logName
+		} else {
+			if logName != "" {
+				val.name = logName
+			} else if !isEmptyConfig {
 				// Construct a log name for server log streams
 				logCounter, ok := h.State["logCounter"].(int)
 				if !ok {
 					logCounter = 0
 				}
 				val.name = fmt.Sprintf("log%d", logCounter)
-				cl.Include = []string{"http.log.access." + val.name}
-				val.log = cl
 				logCounter++
 				h.State["logCounter"] = logCounter
 			}
+			if val.name != "" {
+				cl.Include = []string{"http.log.access." + val.name}
+			}
+		}
+		if !isEmptyConfig {
+			val.log = cl
 		}
 		configValues = append(configValues, ConfigValue{
 			Class: "custom_log",

caddyconfig/httpcaddyfile/builtins_test.go

@@ -1,6 +1,7 @@
 package httpcaddyfile
 
 import (
+	"strings"
 	"testing"
 
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
@@ -51,12 +52,13 @@ func TestLogDirectiveSyntax(t *testing.T) {
 		},
 		{
 			input: `:8080 {
-				log invalid {
+				log name-override {
 					output file foo.log
 				}
 			}
 			`,
-			expectError: true,
+			output:      `{"logging":{"logs":{"default":{"exclude":["http.log.access.name-override"]},"name-override":{"writer":{"filename":"foo.log","output":"file"},"include":["http.log.access.name-override"]}}},"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{"default_logger_name":"name-override"}}}}}}`,
+			expectError: false,
 		},
 	} {
 
@@ -213,3 +215,139 @@ func TestRedirDirectiveSyntax(t *testing.T) {
 		}
 	}
 }
+
+func TestImportErrorLine(t *testing.T) {
+	for i, tc := range []struct {
+		input     string
+		errorFunc func(err error) bool
+	}{
+		{
+			input: `(t1) {
+					abort {args[:]}
+				}
+				:8080 {
+					import t1
+					import t1 true
+				}`,
+			errorFunc: func(err error) bool {
+				return err != nil && strings.Contains(err.Error(), "Caddyfile:6 (import t1)")
+			},
+		},
+		{
+			input: `(t1) {
+					abort {args[:]}
+				}
+				:8080 {
+					import t1 true
+				}`,
+			errorFunc: func(err error) bool {
+				return err != nil && strings.Contains(err.Error(), "Caddyfile:5 (import t1)")
+			},
+		},
+		{
+			input: `
+				import testdata/import_variadic_snippet.txt
+				:8080 {
+					import t1 true
+				}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+		{
+			input: `
+				import testdata/import_variadic_with_import.txt
+				:8080 {
+					import t1 true
+					import t2 true
+				}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+	} {
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if !tc.errorFunc(err) {
+			t.Errorf("Test %d error expectation failed, got %s", i, err)
+			continue
+		}
+	}
+}
+
+func TestNestedImport(t *testing.T) {
+	for i, tc := range []struct {
+		input     string
+		errorFunc func(err error) bool
+	}{
+		{
+			input: `(t1) {
+						respond {args[0]} {args[1]}
+					}
+					
+					(t2) {
+						import t1 {args[0]} 202
+					}
+					
+					:8080 {
+						handle {
+							import t2 "foobar"
+						}
+					}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+		{
+			input: `(t1) {
+						respond {args[:]}
+					}
+					
+					(t2) {
+						import t1 {args[0]} {args[1]}
+					}
+					
+					:8080 {
+						handle {
+							import t2 "foobar" 202
+						}
+					}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+		{
+			input: `(t1) {
+						respond {args[0]} {args[1]}
+					}
+					
+					(t2) {
+						import t1 {args[:]}
+					}
+					
+					:8080 {
+						handle {
+							import t2 "foobar" 202
+						}
+					}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+	} {
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if !tc.errorFunc(err) {
+			t.Errorf("Test %d error expectation failed, got %s", i, err)
+			continue
+		}
+	}
+}

caddyconfig/httpcaddyfile/directives.go

@@ -65,6 +65,7 @@ var directiveOrder = []string{
 	"templates",
 
 	// special routing & dispatching directives
+	"invoke",
 	"handle",
 	"handle_path",
 	"route",
@@ -172,6 +173,7 @@ func (h Helper) Caddyfiles() []string {
 	for file := range files {
 		filesSlice = append(filesSlice, file)
 	}
+	sort.Strings(filesSlice)
 	return filesSlice
 }
 
@@ -427,26 +429,16 @@ func sortRoutes(routes []ConfigValue) {
 			jPathLen = len(jPM[0])
 		}
 
-		// some directives involve setting values which can overwrite
-		// each other, so it makes most sense to reverse the order so
-		// that the lease specific matcher is first; everything else
-		// has most-specific matcher first
-		if iDir == "vars" {
+		sortByPath := func() bool {
 			// we can only confidently compare path lengths if both
 			// directives have a single path to match (issue #5037)
 			if iPathLen > 0 && jPathLen > 0 {
-				// sort least-specific (shortest) path first
-				return iPathLen < jPathLen
-			}
+				// if both paths are the same except for a trailing wildcard,
+				// sort by the shorter path first (which is more specific)
+				if strings.TrimSuffix(iPM[0], "*") == strings.TrimSuffix(jPM[0], "*") {
+					return iPathLen < jPathLen
+				}
 
-			// if both directives don't have a single path to compare,
-			// sort whichever one has no matcher first; if both have
-			// no matcher, sort equally (stable sort preserves order)
-			return len(iRoute.MatcherSetsRaw) == 0 && len(jRoute.MatcherSetsRaw) > 0
-		} else {
-			// we can only confidently compare path lengths if both
-			// directives have a single path to match (issue #5037)
-			if iPathLen > 0 && jPathLen > 0 {
 				// sort most-specific (longest) path first
 				return iPathLen > jPathLen
 			}
@@ -455,7 +447,18 @@ func sortRoutes(routes []ConfigValue) {
 			// sort whichever one has a matcher first; if both have
 			// a matcher, sort equally (stable sort preserves order)
 			return len(iRoute.MatcherSetsRaw) > 0 && len(jRoute.MatcherSetsRaw) == 0
+		}()
+
+		// some directives involve setting values which can overwrite
+		// each other, so it makes most sense to reverse the order so
+		// that the least-specific matcher is first, allowing the last
+		// matching one to win
+		if iDir == "vars" {
+			return !sortByPath
 		}
+
+		// everything else is most-specific matcher first
+		return sortByPath
 	})
 }
 

caddyconfig/httpcaddyfile/httptype.go

@@ -30,6 +30,7 @@ import (
 	"github.com/caddyserver/caddy/v2/modules/caddypki"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
 	"go.uber.org/zap"
+	"golang.org/x/exp/slices"
 )
 
 func init() {
@@ -52,8 +53,10 @@ type ServerType struct {
 }
 
 // Setup makes a config from the tokens.
-func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
-	options map[string]any) (*caddy.Config, []caddyconfig.Warning, error) {
+func (st ServerType) Setup(
+	inputServerBlocks []caddyfile.ServerBlock,
+	options map[string]any,
+) (*caddy.Config, []caddyconfig.Warning, error) {
 	var warnings []caddyconfig.Warning
 	gc := counter{new(int)}
 	state := make(map[string]any)
@@ -79,6 +82,11 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 		return nil, warnings, err
 	}
 
+	originalServerBlocks, err = st.extractNamedRoutes(originalServerBlocks, options, &warnings)
+	if err != nil {
+		return nil, warnings, err
+	}
+
 	// replace shorthand placeholders (which are convenient
 	// when writing a Caddyfile) with their actual placeholder
 	// identifiers or variable names
@@ -172,6 +180,18 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 				result.directive = dir
 				sb.pile[result.Class] = append(sb.pile[result.Class], result)
 			}
+
+			// specially handle named routes that were pulled out from
+			// the invoke directive, which could be nested anywhere within
+			// some subroutes in this directive; we add them to the pile
+			// for this server block
+			if state[namedRouteKey] != nil {
+				for name := range state[namedRouteKey].(map[string]struct{}) {
+					result := ConfigValue{Class: namedRouteKey, Value: name}
+					sb.pile[result.Class] = append(sb.pile[result.Class], result)
+				}
+				state[namedRouteKey] = nil
+			}
 		}
 	}
 
@@ -222,7 +242,7 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 		if ncl.name == caddy.DefaultLoggerName {
 			hasDefaultLog = true
 		}
-		if _, ok := options["debug"]; ok && ncl.log.Level == "" {
+		if _, ok := options["debug"]; ok && ncl.log != nil && ncl.log.Level == "" {
 			ncl.log.Level = zap.DebugLevel.CapitalString()
 		}
 		customLogs = append(customLogs, ncl)
@@ -241,7 +261,9 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 		if _, ok := options["debug"]; ok {
 			customLogs = append(customLogs, namedCustomLog{
 				name: caddy.DefaultLoggerName,
-				log:  &caddy.CustomLog{Level: zap.DebugLevel.CapitalString()},
+				log: &caddy.CustomLog{
+					BaseLog: caddy.BaseLog{Level: zap.DebugLevel.CapitalString()},
+				},
 			})
 		}
 	}
@@ -303,7 +325,21 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 				Logs: make(map[string]*caddy.CustomLog),
 			}
 		}
+
+		// Add the default log first if defined, so that it doesn't
+		// accidentally get re-created below due to the Exclude logic
 		for _, ncl := range customLogs {
+			if ncl.name == caddy.DefaultLoggerName && ncl.log != nil {
+				cfg.Logging.Logs[caddy.DefaultLoggerName] = ncl.log
+				break
+			}
+		}
+
+		// Add the rest of the custom logs
+		for _, ncl := range customLogs {
+			if ncl.log == nil || ncl.name == caddy.DefaultLoggerName {
+				continue
+			}
 			if ncl.name != "" {
 				cfg.Logging.Logs[ncl.name] = ncl.log
 			}
@@ -317,8 +353,16 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 					cfg.Logging.Logs[caddy.DefaultLoggerName] = defaultLog
 				}
 				defaultLog.Exclude = append(defaultLog.Exclude, ncl.log.Include...)
+
+				// avoid duplicates by sorting + compacting
+				sort.Strings(defaultLog.Exclude)
+				defaultLog.Exclude = slices.Compact[[]string, string](defaultLog.Exclude)
 			}
 		}
+		// we may have not actually added anything, so remove if empty
+		if len(cfg.Logging.Logs) == 0 {
+			cfg.Logging = nil
+		}
 	}
 
 	return cfg, warnings, nil
@@ -401,6 +445,77 @@ func (ServerType) evaluateGlobalOptionsBlock(serverBlocks []serverBlock, options
 	return serverBlocks[1:], nil
 }
 
+// extractNamedRoutes pulls out any named route server blocks
+// so they don't get parsed as sites, and stores them in options
+// for later.
+func (ServerType) extractNamedRoutes(
+	serverBlocks []serverBlock,
+	options map[string]any,
+	warnings *[]caddyconfig.Warning,
+) ([]serverBlock, error) {
+	namedRoutes := map[string]*caddyhttp.Route{}
+
+	gc := counter{new(int)}
+	state := make(map[string]any)
+
+	// copy the server blocks so we can
+	// splice out the named route ones
+	filtered := append([]serverBlock{}, serverBlocks...)
+	index := -1
+
+	for _, sb := range serverBlocks {
+		index++
+		if !sb.block.IsNamedRoute {
+			continue
+		}
+
+		// splice out this block, because we know it's not a real server
+		filtered = append(filtered[:index], filtered[index+1:]...)
+		index--
+
+		if len(sb.block.Segments) == 0 {
+			continue
+		}
+
+		// zip up all the segments since ParseSegmentAsSubroute
+		// was designed to take a directive+
+		wholeSegment := caddyfile.Segment{}
+		for _, segment := range sb.block.Segments {
+			wholeSegment = append(wholeSegment, segment...)
+		}
+
+		h := Helper{
+			Dispenser:    caddyfile.NewDispenser(wholeSegment),
+			options:      options,
+			warnings:     warnings,
+			matcherDefs:  nil,
+			parentBlock:  sb.block,
+			groupCounter: gc,
+			State:        state,
+		}
+
+		handler, err := ParseSegmentAsSubroute(h)
+		if err != nil {
+			return nil, err
+		}
+		subroute := handler.(*caddyhttp.Subroute)
+		route := caddyhttp.Route{}
+
+		if len(subroute.Routes) == 1 && len(subroute.Routes[0].MatcherSetsRaw) == 0 {
+			// if there's only one route with no matcher, then we can simplify
+			route.HandlersRaw = append(route.HandlersRaw, subroute.Routes[0].HandlersRaw[0])
+		} else {
+			// otherwise we need the whole subroute
+			route.HandlersRaw = []json.RawMessage{caddyconfig.JSONModuleObject(handler, "handler", subroute.CaddyModule().ID.Name(), h.warnings)}
+		}
+
+		namedRoutes[sb.block.Keys[0]] = &route
+	}
+	options["named_routes"] = namedRoutes
+
+	return filtered, nil
+}
+
 // serversFromPairings creates the servers for each pairing of addresses
 // to server blocks. Each pairing is essentially a server definition.
 func (st *ServerType) serversFromPairings(
@@ -411,6 +526,7 @@ func (st *ServerType) serversFromPairings(
 ) (map[string]*caddyhttp.Server, error) {
 	servers := make(map[string]*caddyhttp.Server)
 	defaultSNI := tryString(options["default_sni"], warnings)
+	fallbackSNI := tryString(options["fallback_sni"], warnings)
 
 	httpPort := strconv.Itoa(caddyhttp.DefaultHTTPPort)
 	if hp, ok := options["http_port"].(int); ok {
@@ -539,6 +655,24 @@ func (st *ServerType) serversFromPairings(
 			}
 		}
 
+		// add named routes to the server if 'invoke' was used inside of it
+		configuredNamedRoutes := options["named_routes"].(map[string]*caddyhttp.Route)
+		for _, sblock := range p.serverBlocks {
+			if len(sblock.pile[namedRouteKey]) == 0 {
+				continue
+			}
+			for _, value := range sblock.pile[namedRouteKey] {
+				if srv.NamedRoutes == nil {
+					srv.NamedRoutes = map[string]*caddyhttp.Route{}
+				}
+				name := value.Value.(string)
+				if configuredNamedRoutes[name] == nil {
+					return nil, fmt.Errorf("cannot invoke named route '%s', which was not defined", name)
+				}
+				srv.NamedRoutes[name] = configuredNamedRoutes[name]
+			}
+		}
+
 		// create a subroute for each site in the server block
 		for _, sblock := range p.serverBlocks {
 			matcherSetsEnc, err := st.compileEncodedMatcherSets(sblock)
@@ -568,6 +702,11 @@ func (st *ServerType) serversFromPairings(
 							cp.DefaultSNI = defaultSNI
 							break
 						}
+						if h == fallbackSNI {
+							hosts = append(hosts, "")
+							cp.FallbackSNI = fallbackSNI
+							break
+						}
 					}
 
 					if len(hosts) > 0 {
@@ -576,6 +715,7 @@ func (st *ServerType) serversFromPairings(
 						}
 					} else {
 						cp.DefaultSNI = defaultSNI
+						cp.FallbackSNI = fallbackSNI
 					}
 
 					// only append this policy if it actually changes something
@@ -653,12 +793,20 @@ func (st *ServerType) serversFromPairings(
 			sblockLogHosts := sblock.hostsFromKeys(true)
 			for _, cval := range sblock.pile["custom_log"] {
 				ncl := cval.Value.(namedCustomLog)
-				if sblock.hasHostCatchAllKey() {
+				if sblock.hasHostCatchAllKey() && len(ncl.hostnames) == 0 {
 					// all requests for hosts not able to be listed should use
 					// this log because it's a catch-all-hosts server block
 					srv.Logs.DefaultLoggerName = ncl.name
+				} else if len(ncl.hostnames) > 0 {
+					// if the logger overrides the hostnames, map that to the logger name
+					for _, h := range ncl.hostnames {
+						if srv.Logs.LoggerNames == nil {
+							srv.Logs.LoggerNames = make(map[string]string)
+						}
+						srv.Logs.LoggerNames[h] = ncl.name
+					}
 				} else {
-					// map each host to the user's desired logger name
+					// otherwise, map each host to the logger name
 					for _, h := range sblockLogHosts {
 						if srv.Logs.LoggerNames == nil {
 							srv.Logs.LoggerNames = make(map[string]string)
@@ -701,8 +849,8 @@ func (st *ServerType) serversFromPairings(
 		// policy missing for any HTTPS-enabled hosts, if so, add it... maybe?
 		if addressQualifiesForTLS &&
 			!hasCatchAllTLSConnPolicy &&
-			(len(srv.TLSConnPolicies) > 0 || !autoHTTPSWillAddConnPolicy || defaultSNI != "") {
-			srv.TLSConnPolicies = append(srv.TLSConnPolicies, &caddytls.ConnectionPolicy{DefaultSNI: defaultSNI})
+			(len(srv.TLSConnPolicies) > 0 || !autoHTTPSWillAddConnPolicy || defaultSNI != "" || fallbackSNI != "") {
+			srv.TLSConnPolicies = append(srv.TLSConnPolicies, &caddytls.ConnectionPolicy{DefaultSNI: defaultSNI, FallbackSNI: fallbackSNI})
 		}
 
 		// tidy things up a bit
@@ -974,7 +1122,7 @@ func buildSubroute(routes []ConfigValue, groupCounter counter, needsSorting bool
 	if needsSorting {
 		for _, val := range routes {
 			if !directiveIsOrdered(val.directive) {
-				return nil, fmt.Errorf("directive '%s' is not an ordered HTTP handler, so it cannot be used here", val.directive)
+				return nil, fmt.Errorf("directive '%s' is not an ordered HTTP handler, so it cannot be used here - try placing within a route block or using the order global option", val.directive)
 			}
 		}
 
@@ -1328,6 +1476,7 @@ func placeholderShorthands() []string {
 		"{tls_client_certificate_pem}", "{http.request.tls.client.certificate_pem}",
 		"{tls_client_certificate_der_base64}", "{http.request.tls.client.certificate_der_base64}",
 		"{upstream_hostport}", "{http.reverse_proxy.upstream.hostport}",
+		"{client_ip}", "{http.vars.client_ip}",
 	}
 }
 
@@ -1446,8 +1595,9 @@ func (c counter) nextGroup() string {
 }
 
 type namedCustomLog struct {
-	name string
-	log  *caddy.CustomLog
+	name      string
+	hostnames []string
+	log       *caddy.CustomLog
 }
 
 // sbAddrAssociation is a mapping from a list of
@@ -1459,6 +1609,7 @@ type sbAddrAssociation struct {
 }
 
 const matcherPrefix = "@"
+const namedRouteKey = "named_route"
 
 // Interface guard
 var _ caddyfile.ServerType = (*ServerType)(nil)

caddyconfig/httpcaddyfile/options.go

@@ -33,6 +33,7 @@ func init() {
 	RegisterGlobalOption("grace_period", parseOptDuration)
 	RegisterGlobalOption("shutdown_delay", parseOptDuration)
 	RegisterGlobalOption("default_sni", parseOptSingleString)
+	RegisterGlobalOption("fallback_sni", parseOptSingleString)
 	RegisterGlobalOption("order", parseOptOrder)
 	RegisterGlobalOption("storage", parseOptStorage)
 	RegisterGlobalOption("storage_clean_interval", parseOptDuration)

caddyconfig/httpcaddyfile/serveroptions.go

@@ -41,9 +41,11 @@ type serverOptions struct {
 	IdleTimeout          caddy.Duration
 	KeepAliveInterval    caddy.Duration
 	MaxHeaderBytes       int
+	EnableFullDuplex     bool
 	Protocols            []string
 	StrictSNIHost        *bool
 	TrustedProxiesRaw    json.RawMessage
+	ClientIPHeaders      []string
 	ShouldLogCredentials bool
 	Metrics              *caddyhttp.Metrics
 }
@@ -156,6 +158,12 @@ func unmarshalCaddyfileServerOptions(d *caddyfile.Dispenser) (any, error) {
 				}
 				serverOpts.MaxHeaderBytes = int(size)
 
+			case "enable_full_duplex":
+				if d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				serverOpts.EnableFullDuplex = true
+
 			case "log_credentials":
 				if d.NextArg() {
 					return nil, d.ArgErr()
@@ -208,6 +216,18 @@ func unmarshalCaddyfileServerOptions(d *caddyfile.Dispenser) (any, error) {
 				)
 				serverOpts.TrustedProxiesRaw = jsonSource
 
+			case "client_ip_headers":
+				headers := d.RemainingArgs()
+				for _, header := range headers {
+					if sliceContains(serverOpts.ClientIPHeaders, header) {
+						return nil, d.Errf("client IP header %s specified more than once", header)
+					}
+					serverOpts.ClientIPHeaders = append(serverOpts.ClientIPHeaders, header)
+				}
+				if nesting := d.Nesting(); d.NextBlock(nesting) {
+					return nil, d.ArgErr()
+				}
+
 			case "metrics":
 				if d.NextArg() {
 					return nil, d.ArgErr()
@@ -314,9 +334,11 @@ func applyServerOptions(
 		server.IdleTimeout = opts.IdleTimeout
 		server.KeepAliveInterval = opts.KeepAliveInterval
 		server.MaxHeaderBytes = opts.MaxHeaderBytes
+		server.EnableFullDuplex = opts.EnableFullDuplex
 		server.Protocols = opts.Protocols
 		server.StrictSNIHost = opts.StrictSNIHost
 		server.TrustedProxiesRaw = opts.TrustedProxiesRaw
+		server.ClientIPHeaders = opts.ClientIPHeaders
 		server.Metrics = opts.Metrics
 		if opts.ShouldLogCredentials {
 			if server.Logs == nil {

caddyconfig/httpcaddyfile/testdata/import_variadic.txt

@@ -0,0 +1,9 @@
+(t2) {
+    respond 200 {
+        body {args[:]}
+    }
+}
+
+:8082 {
+    import t2 false
+}

caddyconfig/httpcaddyfile/testdata/import_variadic_snippet.txt

@@ -0,0 +1,9 @@
+(t1) {
+    respond 200 {
+        body {args[:]}
+    }
+}
+
+:8081 {
+    import t1 false
+}

caddyconfig/httpcaddyfile/testdata/import_variadic_with_import.txt

@@ -0,0 +1,15 @@
+(t1) {
+    respond 200 {
+        body {args[:]}
+    }
+}
+
+:8081 {
+    import t1 false
+}
+
+import import_variadic.txt
+
+:8083 {
+    import t2 true
+}

caddyconfig/httpcaddyfile/tlsapp.go

@@ -206,8 +206,8 @@ func (st ServerType) buildTLSApp(
 			}
 
 			// associate our new automation policy with this server block's hosts
-			ap.Subjects = sblock.hostsFromKeysNotHTTP(httpPort)
-			sort.Strings(ap.Subjects) // solely for deterministic test results
+			ap.SubjectsRaw = sblock.hostsFromKeysNotHTTP(httpPort)
+			sort.Strings(ap.SubjectsRaw) // solely for deterministic test results
 
 			// if a combination of public and internal names were given
 			// for this same server block and no issuer was specified, we
@@ -217,7 +217,11 @@ func (st ServerType) buildTLSApp(
 			var ap2 *caddytls.AutomationPolicy
 			if len(ap.Issuers) == 0 {
 				var internal, external []string
-				for _, s := range ap.Subjects {
+				for _, s := range ap.SubjectsRaw {
+					// do not create Issuers for Tailscale domains; they will be given a Manager instead
+					if strings.HasSuffix(strings.ToLower(s), ".ts.net") {
+						continue
+					}
 					if !certmagic.SubjectQualifiesForCert(s) {
 						return nil, warnings, fmt.Errorf("subject does not qualify for certificate: '%s'", s)
 					}
@@ -235,10 +239,10 @@ func (st ServerType) buildTLSApp(
 					}
 				}
 				if len(external) > 0 && len(internal) > 0 {
-					ap.Subjects = external
+					ap.SubjectsRaw = external
 					apCopy := *ap
 					ap2 = &apCopy
-					ap2.Subjects = internal
+					ap2.SubjectsRaw = internal
 					ap2.IssuersRaw = []json.RawMessage{caddyconfig.JSONModuleObject(caddytls.InternalIssuer{}, "module", "internal", &warnings)}
 				}
 			}
@@ -339,14 +343,14 @@ func (st ServerType) buildTLSApp(
 		for h := range httpsHostsSharedWithHostlessKey {
 			al = append(al, h)
 			if !certmagic.SubjectQualifiesForPublicCert(h) {
-				internalAP.Subjects = append(internalAP.Subjects, h)
+				internalAP.SubjectsRaw = append(internalAP.SubjectsRaw, h)
 			}
 		}
 	}
 	if len(al) > 0 {
 		tlsApp.CertificatesRaw["automate"] = caddyconfig.JSON(al, &warnings)
 	}
-	if len(internalAP.Subjects) > 0 {
+	if len(internalAP.SubjectsRaw) > 0 {
 		if tlsApp.Automation == nil {
 			tlsApp.Automation = new(caddytls.AutomationConfig)
 		}
@@ -412,7 +416,7 @@ func (st ServerType) buildTLSApp(
 		// for convenience)
 		automationHostSet := make(map[string]struct{})
 		for _, ap := range tlsApp.Automation.Policies {
-			for _, s := range ap.Subjects {
+			for _, s := range ap.SubjectsRaw {
 				if _, ok := automationHostSet[s]; ok {
 					return nil, warnings, fmt.Errorf("hostname appears in more than one automation policy, making certificate management ambiguous: %s", s)
 				}
@@ -533,7 +537,7 @@ func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls
 		if automationPolicyIsSubset(aps[j], aps[i]) {
 			return false
 		}
-		return len(aps[i].Subjects) > len(aps[j].Subjects)
+		return len(aps[i].SubjectsRaw) > len(aps[j].SubjectsRaw)
 	})
 
 	emptyAPCount := 0
@@ -541,7 +545,7 @@ func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls
 	// compute the number of empty policies (disregarding subjects) - see #4128
 	emptyAP := new(caddytls.AutomationPolicy)
 	for i := 0; i < len(aps); i++ {
-		emptyAP.Subjects = aps[i].Subjects
+		emptyAP.SubjectsRaw = aps[i].SubjectsRaw
 		if reflect.DeepEqual(aps[i], emptyAP) {
 			emptyAPCount++
 			if !automationPolicyHasAllPublicNames(aps[i]) {
@@ -583,7 +587,7 @@ outer:
 				aps[i].KeyType == aps[j].KeyType &&
 				aps[i].OnDemand == aps[j].OnDemand &&
 				aps[i].RenewalWindowRatio == aps[j].RenewalWindowRatio {
-				if len(aps[i].Subjects) > 0 && len(aps[j].Subjects) == 0 {
+				if len(aps[i].SubjectsRaw) > 0 && len(aps[j].SubjectsRaw) == 0 {
 					// later policy (at j) has no subjects ("catch-all"), so we can
 					// remove the identical-but-more-specific policy that comes first
 					// AS LONG AS it is not shadowed by another policy before it; e.g.
@@ -598,9 +602,9 @@ outer:
 					}
 				} else {
 					// avoid repeated subjects
-					for _, subj := range aps[j].Subjects {
-						if !sliceContains(aps[i].Subjects, subj) {
-							aps[i].Subjects = append(aps[i].Subjects, subj)
+					for _, subj := range aps[j].SubjectsRaw {
+						if !sliceContains(aps[i].SubjectsRaw, subj) {
+							aps[i].SubjectsRaw = append(aps[i].SubjectsRaw, subj)
 						}
 					}
 					aps = append(aps[:j], aps[j+1:]...)
@@ -616,15 +620,15 @@ outer:
 // automationPolicyIsSubset returns true if a's subjects are a subset
 // of b's subjects.
 func automationPolicyIsSubset(a, b *caddytls.AutomationPolicy) bool {
-	if len(b.Subjects) == 0 {
+	if len(b.SubjectsRaw) == 0 {
 		return true
 	}
-	if len(a.Subjects) == 0 {
+	if len(a.SubjectsRaw) == 0 {
 		return false
 	}
-	for _, aSubj := range a.Subjects {
+	for _, aSubj := range a.SubjectsRaw {
 		var inSuperset bool
-		for _, bSubj := range b.Subjects {
+		for _, bSubj := range b.SubjectsRaw {
 			if certmagic.MatchWildcard(aSubj, bSubj) {
 				inSuperset = true
 				break
@@ -662,7 +666,7 @@ func subjectQualifiesForPublicCert(ap *caddytls.AutomationPolicy, subj string) b
 }
 
 func automationPolicyHasAllPublicNames(ap *caddytls.AutomationPolicy) bool {
-	for _, subj := range ap.Subjects {
+	for _, subj := range ap.SubjectsRaw {
 		if !subjectQualifiesForPublicCert(ap, subj) {
 			return false
 		}

caddyconfig/httpcaddyfile/tlsapp_test.go

@@ -47,8 +47,8 @@ func TestAutomationPolicyIsSubset(t *testing.T) {
 			expect: false,
 		},
 	} {
-		apA := &caddytls.AutomationPolicy{Subjects: test.a}
-		apB := &caddytls.AutomationPolicy{Subjects: test.b}
+		apA := &caddytls.AutomationPolicy{SubjectsRaw: test.a}
+		apB := &caddytls.AutomationPolicy{SubjectsRaw: test.b}
 		if actual := automationPolicyIsSubset(apA, apB); actual != test.expect {
 			t.Errorf("Test %d: Expected %t but got %t (A: %v  B: %v)", i, test.expect, actual, test.a, test.b)
 		}

caddyconfig/httploader.go

@@ -30,8 +30,14 @@ func init() {
 	caddy.RegisterModule(HTTPLoader{})
 }
 
-// HTTPLoader can load Caddy configs over HTTP(S). It can adapt the config
-// based on the Content-Type header of the HTTP response.
+// HTTPLoader can load Caddy configs over HTTP(S).
+//
+// If the response is not a JSON config, a config adapter must be specified
+// either in the loader config (`adapter`), or in the Content-Type HTTP header
+// returned in the HTTP response from the server. The Content-Type header is
+// read just like the admin API's `/load` endpoint. Uf you don't have control
+// over the HTTP server (but can still trust its response), you can override
+// the Content-Type header by setting the `adapter` property in this config.
 type HTTPLoader struct {
 	// The method for the request. Default: GET
 	Method string `json:"method,omitempty"`
@@ -45,6 +51,11 @@ type HTTPLoader struct {
 	// Maximum time allowed for a complete connection and request.
 	Timeout caddy.Duration `json:"timeout,omitempty"`
 
+	// The name of the config adapter to use, if any. Only needed
+	// if the HTTP response is not a JSON config and if the server's
+	// Content-Type header is missing or incorrect.
+	Adapter string `json:"adapter,omitempty"`
+
 	TLS *struct {
 		// Present this instance's managed remote identity credentials to the server.
 		UseServerIdentity bool `json:"use_server_identity,omitempty"`
@@ -108,7 +119,12 @@ func (hl HTTPLoader) LoadConfig(ctx caddy.Context) ([]byte, error) {
 		return nil, err
 	}
 
-	result, warnings, err := adaptByContentType(resp.Header.Get("Content-Type"), body)
+	// adapt the config based on either manually-configured adapter or server's response header
+	ct := resp.Header.Get("Content-Type")
+	if hl.Adapter != "" {
+		ct = "text/" + hl.Adapter
+	}
+	result, warnings, err := adaptByContentType(ct, body)
 	if err != nil {
 		return nil, err
 	}

caddytest/integration/caddyfile_adapt/global_server_options_single.txt

@@ -11,10 +11,13 @@
 			idle 30s
 		}
 		max_header_size 100MB
+		enable_full_duplex
 		log_credentials
 		protocols h1 h2 h2c h3
 		strict_sni_host
 		trusted_proxies static private_ranges
+		client_ip_headers Custom-Real-Client-IP X-Forwarded-For
+		client_ip_headers A-Third-One
 	}
 }
 
@@ -43,6 +46,7 @@ foo.com {
 					"write_timeout": 30000000000,
 					"idle_timeout": 30000000000,
 					"max_header_bytes": 100000000,
+					"enable_full_duplex": true,
 					"routes": [
 						{
 							"match": [
@@ -67,6 +71,11 @@ foo.com {
 						],
 						"source": "static"
 					},
+					"client_ip_headers": [
+						"Custom-Real-Client-IP",
+						"X-Forwarded-For",
+						"A-Third-One"
+					],
 					"logs": {
 						"should_log_credentials": true
 					},

caddytest/integration/caddyfile_adapt/header.txt

@@ -17,6 +17,8 @@
 		+Link "Foo"
 		+Link "Bar"
 	}
+	header >Set Defer
+	header >Replace Deferred Replacement
 }
 ----------
 {
@@ -136,6 +138,31 @@
 											]
 										}
 									}
+								},
+								{
+									"handler": "headers",
+									"response": {
+										"deferred": true,
+										"set": {
+											"Set": [
+												"Defer"
+											]
+										}
+									}
+								},
+								{
+									"handler": "headers",
+									"response": {
+										"deferred": true,
+										"replace": {
+											"Replace": [
+												{
+													"replace": "Replacement",
+													"search_regexp": "Deferred"
+												}
+											]
+										}
+									}
 								}
 							]
 						}

caddytest/integration/caddyfile_adapt/heredoc.txt

@@ -0,0 +1,50 @@
+example.com {
+    respond <<EOF
+    <html>
+      <head><title>Foo</title>
+      <body>Foo</body>
+    </html>
+    EOF 200
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "\u003chtml\u003e\n  \u003chead\u003e\u003ctitle\u003eFoo\u003c/title\u003e\n  \u003cbody\u003eFoo\u003c/body\u003e\n\u003c/html\u003e",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_snippet.txt

@@ -1,6 +1,6 @@
 (logging) {
 	log {
-		output file /var/log/caddy/{args.0}.access.log
+		output file /var/log/caddy/{args[0]}.access.log
 	}
 }
 

caddytest/integration/caddyfile_adapt/invoke_named_routes.txt

@@ -0,0 +1,154 @@
+&(first) {
+	@first path /first
+	vars @first first 1
+	respond "first"
+}
+
+&(second) {
+	respond "second"
+}
+
+:8881 {
+	invoke first
+	route {
+		invoke second
+	}
+}
+
+:8882 {
+	handle {
+		invoke second
+	}
+}
+
+:8883 {
+	respond "no invoke"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8881"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "invoke",
+									"name": "first"
+								},
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "invoke",
+													"name": "second"
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					],
+					"named_routes": {
+						"first": {
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"first": 1,
+													"handler": "vars"
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/first"
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"body": "first",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						},
+						"second": {
+							"handle": [
+								{
+									"body": "second",
+									"handler": "static_response"
+								}
+							]
+						}
+					}
+				},
+				"srv1": {
+					"listen": [
+						":8882"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "invoke",
+													"name": "second"
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					],
+					"named_routes": {
+						"second": {
+							"handle": [
+								{
+									"body": "second",
+									"handler": "static_response"
+								}
+							]
+						}
+					}
+				},
+				"srv2": {
+					"listen": [
+						":8883"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"body": "no invoke",
+									"handler": "static_response"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_override_hostname.txt

@@ -0,0 +1,71 @@
+*.example.com {
+	log {
+		hostnames foo.example.com bar.example.com
+		output file /foo-bar.txt
+	}
+	log {
+		hostnames baz.example.com
+		output file /baz.txt
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0",
+					"http.log.access.log1"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "/foo-bar.txt",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			},
+			"log1": {
+				"writer": {
+					"filename": "/baz.txt",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log1"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"*.example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"logs": {
+						"logger_names": {
+							"bar.example.com": "log0",
+							"baz.example.com": "log1",
+							"foo.example.com": "log0"
+						}
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_override_name_multiaccess.txt

@@ -0,0 +1,86 @@
+{
+	log access-console {
+		include http.log.access.foo
+		output file access-localhost.log
+		format console
+	}
+
+	log access-json {
+		include http.log.access.foo
+		output file access-localhost.json
+		format json
+	}
+}
+
+http://localhost:8881 {
+	log foo
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"access-console": {
+				"writer": {
+					"filename": "access-localhost.log",
+					"output": "file"
+				},
+				"encoder": {
+					"format": "console"
+				},
+				"include": [
+					"http.log.access.foo"
+				]
+			},
+			"access-json": {
+				"writer": {
+					"filename": "access-localhost.json",
+					"output": "file"
+				},
+				"encoder": {
+					"format": "json"
+				},
+				"include": [
+					"http.log.access.foo"
+				]
+			},
+			"default": {
+				"exclude": [
+					"http.log.access.foo"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8881"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"localhost"
+						]
+					},
+					"logs": {
+						"logger_names": {
+							"localhost:8881": "foo"
+						}
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_override_name_multiaccess_debug.txt

@@ -0,0 +1,91 @@
+{
+	debug
+
+	log access-console {
+		include http.log.access.foo
+		output file access-localhost.log
+		format console
+	}
+
+	log access-json {
+		include http.log.access.foo
+		output file access-localhost.json
+		format json
+	}
+}
+
+http://localhost:8881 {
+	log foo
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"access-console": {
+				"writer": {
+					"filename": "access-localhost.log",
+					"output": "file"
+				},
+				"encoder": {
+					"format": "console"
+				},
+				"level": "DEBUG",
+				"include": [
+					"http.log.access.foo"
+				]
+			},
+			"access-json": {
+				"writer": {
+					"filename": "access-localhost.json",
+					"output": "file"
+				},
+				"encoder": {
+					"format": "json"
+				},
+				"level": "DEBUG",
+				"include": [
+					"http.log.access.foo"
+				]
+			},
+			"default": {
+				"level": "DEBUG",
+				"exclude": [
+					"http.log.access.foo"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8881"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"localhost"
+						]
+					},
+					"logs": {
+						"logger_names": {
+							"localhost:8881": "foo"
+						}
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/map_and_vars_with_raw_types.txt

@@ -100,16 +100,16 @@ vars {
 													],
 													"source": "{http.request.host}"
 												},
-												{
-													"foo": "bar",
-													"handler": "vars"
-												},
 												{
 													"abc": true,
 													"def": 1,
 													"ghi": 2.3,
 													"handler": "vars",
 													"jkl": "mn op"
+												},
+												{
+													"foo": "bar",
+													"handler": "vars"
 												}
 											]
 										}

caddytest/integration/caddyfile_adapt/matcher_syntax.txt

@@ -43,6 +43,9 @@
 
 	@matcher11 remote_ip private_ranges
 	respond @matcher11 "remote_ip matcher with private ranges"
+
+	@matcher12 client_ip private_ranges
+	respond @matcher12 "client_ip matcher with private ranges"
 }
 ----------
 {
@@ -250,6 +253,28 @@
 									"handler": "static_response"
 								}
 							]
+						},
+						{
+							"match": [
+								{
+									"client_ip": {
+										"ranges": [
+											"192.168.0.0/16",
+											"172.16.0.0/12",
+											"10.0.0.0/8",
+											"127.0.0.1/8",
+											"fd00::/8",
+											"::1"
+										]
+									}
+								}
+							],
+							"handle": [
+								{
+									"body": "client_ip matcher with private ranges",
+									"handler": "static_response"
+								}
+							]
 						}
 					]
 				}

caddytest/integration/caddyfile_adapt/replaceable_upstream.txt

@@ -0,0 +1,100 @@
+*.sandbox.localhost {
+	@sandboxPort {
+		header_regexp first_label Host ^([0-9]{3})\.sandbox\.
+	}
+	handle @sandboxPort {
+		reverse_proxy {re.first_label.1}
+	}
+	handle {
+		redir {scheme}://application.localhost
+	}
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"*.sandbox.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "reverse_proxy",
+																	"upstreams": [
+																		{
+																			"dial": "{http.regexp.first_label.1}"
+																		}
+																	]
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"header_regexp": {
+														"Host": {
+															"name": "first_label",
+															"pattern": "^([0-9]{3})\\.sandbox\\."
+														}
+													}
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "static_response",
+																	"headers": {
+																		"Location": [
+																			"{http.request.scheme}://application.localhost"
+																		]
+																	},
+																	"status_code": 302
+																}
+															]
+														}
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/replaceable_upstream_partial_port.txt

@@ -0,0 +1,100 @@
+*.sandbox.localhost {
+	@sandboxPort {
+		header_regexp port Host ^([0-9]{3})\.sandbox\.
+	}
+	handle @sandboxPort {
+		reverse_proxy app:6{re.port.1}
+	}
+	handle {
+		redir {scheme}://application.localhost
+	}
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"*.sandbox.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "reverse_proxy",
+																	"upstreams": [
+																		{
+																			"dial": "app:6{http.regexp.port.1}"
+																		}
+																	]
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"header_regexp": {
+														"Host": {
+															"name": "port",
+															"pattern": "^([0-9]{3})\\.sandbox\\."
+														}
+													}
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "static_response",
+																	"headers": {
+																		"Location": [
+																			"{http.request.scheme}://application.localhost"
+																		]
+																	},
+																	"status_code": 302
+																}
+															]
+														}
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/replaceable_upstream_port.txt

@@ -0,0 +1,100 @@
+*.sandbox.localhost {
+	@sandboxPort {
+		header_regexp port Host ^([0-9]{3})\.sandbox\.
+	}
+	handle @sandboxPort {
+		reverse_proxy app:{re.port.1}
+	}
+	handle {
+		redir {scheme}://application.localhost
+	}
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"*.sandbox.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "reverse_proxy",
+																	"upstreams": [
+																		{
+																			"dial": "app:{http.regexp.port.1}"
+																		}
+																	]
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"header_regexp": {
+														"Host": {
+															"name": "port",
+															"pattern": "^([0-9]{3})\\.sandbox\\."
+														}
+													}
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "static_response",
+																	"headers": {
+																		"Location": [
+																			"{http.request.scheme}://application.localhost"
+																		]
+																	},
+																	"status_code": 302
+																}
+															]
+														}
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_dynamic_upstreams.txt

@@ -11,6 +11,7 @@
 			resolvers 8.8.8.8 8.8.4.4
 			dial_timeout 2s
 			dial_fallback_delay 300ms
+			versions ipv6
 		}
 	}
 }
@@ -66,7 +67,10 @@
 												"8.8.4.4"
 											]
 										},
-										"source": "a"
+										"source": "a",
+										"versions": {
+											"ipv6": true
+										}
 									},
 									"handler": "reverse_proxy"
 								}
@@ -113,4 +117,4 @@
 			}
 		}
 	}
-}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_load_balance_wrr.txt

@@ -0,0 +1,71 @@
+:8884
+
+reverse_proxy 127.0.0.1:65535 127.0.0.1:35535 {
+	lb_policy weighted_round_robin 10 1
+	lb_retries 5
+	lb_try_duration 10s
+	lb_try_interval 500ms
+	lb_retry_match {
+		path /foo*
+		method POST
+	}
+	lb_retry_match path /bar*
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"load_balancing": {
+										"retries": 5,
+										"retry_match": [
+											{
+												"method": [
+													"POST"
+												],
+												"path": [
+													"/foo*"
+												]
+											},
+											{
+												"path": [
+													"/bar*"
+												]
+											}
+										],
+										"selection_policy": {
+											"policy": "weighted_round_robin",
+											"weights": [
+												10,
+												1
+											]
+										},
+										"try_duration": 10000000000,
+										"try_interval": 500000000
+									},
+									"upstreams": [
+										{
+											"dial": "127.0.0.1:65535"
+										},
+										{
+											"dial": "127.0.0.1:35535"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_options.txt

@@ -6,7 +6,7 @@ https://example.com {
 		method GET
 		rewrite /rewritten?uri={uri}
 
-		buffer_requests
+		request_buffers 4KB
 
 		transport http {
 			read_buffer 10MB
@@ -54,7 +54,6 @@ https://example.com {
 										{
 											"handle": [
 												{
-													"buffer_requests": true,
 													"handler": "reverse_proxy",
 													"headers": {
 														"request": {
@@ -68,6 +67,7 @@ https://example.com {
 															}
 														}
 													},
+													"request_buffers": 4000,
 													"rewrite": {
 														"method": "GET",
 														"uri": "/rewritten?uri={http.request.uri}"

caddytest/integration/caddyfile_adapt/reverse_proxy_port_range.txt

@@ -0,0 +1,67 @@
+:8884 {
+	# Port range
+	reverse_proxy localhost:8001-8002
+
+	# Port range with placeholder
+	reverse_proxy {host}:8001-8002
+
+	# Port range with scheme
+	reverse_proxy https://localhost:8001-8002
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"upstreams": [
+										{
+											"dial": "localhost:8001"
+										},
+										{
+											"dial": "localhost:8002"
+										}
+									]
+								},
+								{
+									"handler": "reverse_proxy",
+									"upstreams": [
+										{
+											"dial": "{http.request.host}:8001"
+										},
+										{
+											"dial": "{http.request.host}:8002"
+										}
+									]
+								},
+								{
+									"handler": "reverse_proxy",
+									"transport": {
+										"protocol": "http",
+										"tls": {}
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:8001"
+										},
+										{
+											"dial": "localhost:8002"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/sort_directives_within_handle.txt

@@ -1,12 +1,15 @@
 *.example.com {
 	@foo host foo.example.com
 	handle @foo {
-		handle_path /strip* {
+		handle_path /strip {
 			respond "this should be first"
 		}
-		handle {
+		handle_path /strip* {
 			respond "this should be second"
 		}
+		handle {
+			respond "this should be last"
+		}
 	}
 	handle {
 		respond "this should be last"
@@ -35,13 +38,13 @@
 									"handler": "subroute",
 									"routes": [
 										{
-											"group": "group5",
+											"group": "group6",
 											"handle": [
 												{
 													"handler": "subroute",
 													"routes": [
 														{
-															"group": "group2",
+															"group": "group3",
 															"handle": [
 																{
 																	"handler": "subroute",
@@ -68,17 +71,25 @@
 															"match": [
 																{
 																	"path": [
-																		"/strip*"
+																		"/strip"
 																	]
 																}
 															]
 														},
 														{
-															"group": "group2",
+															"group": "group3",
 															"handle": [
 																{
 																	"handler": "subroute",
 																	"routes": [
+																		{
+																			"handle": [
+																				{
+																					"handler": "rewrite",
+																					"strip_path_prefix": "/strip"
+																				}
+																			]
+																		},
 																		{
 																			"handle": [
 																				{
@@ -89,6 +100,31 @@
 																		}
 																	]
 																}
+															],
+															"match": [
+																{
+																	"path": [
+																		"/strip*"
+																	]
+																}
+															]
+														},
+														{
+															"group": "group3",
+															"handle": [
+																{
+																	"handler": "subroute",
+																	"routes": [
+																		{
+																			"handle": [
+																				{
+																					"body": "this should be last",
+																					"handler": "static_response"
+																				}
+																			]
+																		}
+																	]
+																}
 															]
 														}
 													]
@@ -103,7 +139,7 @@
 											]
 										},
 										{
-											"group": "group5",
+											"group": "group6",
 											"handle": [
 												{
 													"handler": "subroute",

caddytest/integration/caddyfile_adapt/sort_vars_in_reverse.txt

@@ -1,7 +1,8 @@
 :80
 
 vars /foobar foo last
-vars /foo foo middle
+vars /foo foo middle-last
+vars /foo* foo middle-first
 vars * foo first
 ----------
 {
@@ -21,6 +22,21 @@ vars * foo first
 								}
 							]
 						},
+						{
+							"match": [
+								{
+									"path": [
+										"/foo*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"foo": "middle-first",
+									"handler": "vars"
+								}
+							]
+						},
 						{
 							"match": [
 								{
@@ -31,7 +47,7 @@ vars * foo first
 							],
 							"handle": [
 								{
-									"foo": "middle",
+									"foo": "middle-last",
 									"handler": "vars"
 								}
 							]

caddytest/integration/reverseproxy_test.go

@@ -22,80 +22,38 @@ func TestSRVReverseProxy(t *testing.T) {
 		},
 		"apps": {
 			"pki": {
-				"certificate_authorities" : {
-					"local" : {
-					"install_trust": false
+				"certificate_authorities": {
+					"local": {
+						"install_trust": false
 					}
 				}
 			},
-		  "http": {
-			"grace_period": 1,
-			"servers": {
-			  "srv0": {
-				"listen": [
-				  ":8080"
-				],
-				"routes": [
-				  {
-					"handle": [
-					  {
-						"handler": "reverse_proxy",
-						"upstreams": [
-						  {
-							"lookup_srv": "srv.host.service.consul"
-						  }
+			"http": {
+				"grace_period": 1,
+				"servers": {
+					"srv0": {
+						"listen": [
+							":18080"
+						],
+						"routes": [
+							{
+								"handle": [
+									{
+										"handler": "reverse_proxy",
+										"dynamic_upstreams": {
+											"source": "srv",
+											"name": "srv.host.service.consul"
+										}
+									}
+								]
+							}
 						]
-					  }
-					]
-				  }
-				]
-			  }
-			}
-		  }
-		}
-	  }
-  `, "json")
-}
-
-func TestSRVWithDial(t *testing.T) {
-	caddytest.AssertLoadError(t, `
-	{
-		"apps": {
-			"pki": {
-				"certificate_authorities" : {
-				  "local" : {
-					"install_trust": false
-				  }
+					}
 				}
-			  },
-		  "http": {
-			"grace_period": 1,
-			"servers": {
-			  "srv0": {
-				"listen": [
-				  ":8080"
-				],
-				"routes": [
-				  {
-					"handle": [
-					  {
-						"handler": "reverse_proxy",
-						"upstreams": [
-						  {
-							"dial": "tcp/address.to.upstream:80",
-							"lookup_srv": "srv.host.service.consul"
-						  }
-						]
-					  }
-					]
-				  }
-				]
-			  }
 			}
-		  }
 		}
-	  }
-	`, "json", `upstream: specifying dial address is incompatible with lookup_srv: 0: {\"dial\": \"tcp/address.to.upstream:80\", \"lookup_srv\": \"srv.host.service.consul\"}`)
+	}
+	`, "json")
 }
 
 func TestDialWithPlaceholderUnix(t *testing.T) {
@@ -138,41 +96,41 @@ func TestDialWithPlaceholderUnix(t *testing.T) {
 		},
 		"apps": {
 			"pki": {
-				"certificate_authorities" : {
-				  "local" : {
-					"install_trust": false
-				  }
+				"certificate_authorities": {
+					"local": {
+						"install_trust": false
+					}
 				}
-			  },
-		  "http": {
-			"grace_period": 1,
-			"servers": {
-			  "srv0": {
-				"listen": [
-				  ":8080"
-				],
-				"routes": [
-				  {
-					"handle": [
-					  {
-						"handler": "reverse_proxy",
-						"upstreams": [
-						  {
-							"dial": "unix/{http.request.header.X-Caddy-Upstream-Dial}"
-						  }
+			},
+			"http": {
+				"grace_period": 1,
+				"servers": {
+					"srv0": {
+						"listen": [
+							":18080"
+						],
+						"routes": [
+							{
+								"handle": [
+									{
+										"handler": "reverse_proxy",
+										"upstreams": [
+											{
+												"dial": "unix/{http.request.header.X-Caddy-Upstream-Dial}"
+											}
+										]
+									}
+								]
+							}
 						]
-					  }
-					]
-				  }
-				]
-			  }
-			}
-		  }
+					}
+				}
+		  	}
 		}
-	  }
+	}
 	`, "json")
 
-	req, err := http.NewRequest(http.MethodGet, "http://localhost:8080", nil)
+	req, err := http.NewRequest(http.MethodGet, "http://localhost:18080", nil)
 	if err != nil {
 		t.Fail()
 		return
@@ -190,18 +148,18 @@ func TestReverseProxyWithPlaceholderDialAddress(t *testing.T) {
 		},
 		"apps": {
 			"pki": {
-				"certificate_authorities" : {
-				  "local" : {
-					"install_trust": false
-				  }
+				"certificate_authorities": {
+					"local": {
+						"install_trust": false
+					}
 				}
-			  },
+			},
 			"http": {
 				"grace_period": 1,
 				"servers": {
 					"srv0": {
 						"listen": [
-							":8080"
+							":18080"
 						],
 						"routes": [
 							{
@@ -264,14 +222,14 @@ func TestReverseProxyWithPlaceholderDialAddress(t *testing.T) {
 			}
 		}
 	}
-  	`, "json")
+	`, "json")
 
 	req, err := http.NewRequest(http.MethodGet, "http://localhost:9080", nil)
 	if err != nil {
 		t.Fail()
 		return
 	}
-	req.Header.Set("X-Caddy-Upstream-Dial", "localhost:8080")
+	req.Header.Set("X-Caddy-Upstream-Dial", "localhost:18080")
 	tester.AssertResponse(req, 200, "Hello, World!")
 }
 
@@ -284,18 +242,18 @@ func TestReverseProxyWithPlaceholderTCPDialAddress(t *testing.T) {
 		},
 		"apps": {
 			"pki": {
-				"certificate_authorities" : {
-				  "local" : {
-					"install_trust": false
-				  }
+				"certificate_authorities": {
+					"local": {
+						"install_trust": false
+					}
 				}
-			  },
+			},
 			"http": {
 				"grace_period": 1,
 				"servers": {
 					"srv0": {
 						"listen": [
-							":8080"
+							":18080"
 						],
 						"routes": [
 							{
@@ -340,7 +298,7 @@ func TestReverseProxyWithPlaceholderTCPDialAddress(t *testing.T) {
 										"handler": "reverse_proxy",
 										"upstreams": [
 											{
-												"dial": "tcp/{http.request.header.X-Caddy-Upstream-Dial}:8080"
+												"dial": "tcp/{http.request.header.X-Caddy-Upstream-Dial}:18080"
 											}
 										]
 									}
@@ -358,7 +316,7 @@ func TestReverseProxyWithPlaceholderTCPDialAddress(t *testing.T) {
 			}
 		}
 	}
-  	`, "json")
+	`, "json")
 
 	req, err := http.NewRequest(http.MethodGet, "http://localhost:9080", nil)
 	if err != nil {
@@ -369,51 +327,6 @@ func TestReverseProxyWithPlaceholderTCPDialAddress(t *testing.T) {
 	tester.AssertResponse(req, 200, "Hello, World!")
 }
 
-func TestSRVWithActiveHealthcheck(t *testing.T) {
-	caddytest.AssertLoadError(t, `
-	{
-		"apps": {
-			"pki": {
-				"certificate_authorities" : {
-				  "local" : {
-					"install_trust": false
-				  }
-				}
-			  },
-		  "http": {
-			"grace_period": 1,
-			"servers": {
-			  "srv0": {
-				"listen": [
-				  ":8080"
-				],
-				"routes": [
-				  {
-					"handle": [
-					  {
-						"handler": "reverse_proxy",
-						"health_checks": {
-							"active": {
-								"path": "/ok"
-							}
-						},
-						"upstreams": [
-						  {
-							"lookup_srv": "srv.host.service.consul"
-						  }
-						]
-					  }
-					]
-				  }
-				]
-			  }
-			}
-		  }
-		}
-	  }
-	`, "json", `upstream: lookup_srv is incompatible with active health checks: 0: {\"dial\": \"\", \"lookup_srv\": \"srv.host.service.consul\"}`)
-}
-
 func TestReverseProxyHealthCheck(t *testing.T) {
 	tester := caddytest.NewTester(t)
 	tester.InitServer(`
@@ -440,7 +353,7 @@ func TestReverseProxyHealthCheck(t *testing.T) {
 			health_timeout 100ms
 		}
 	}
-  `, "caddyfile")
+	`, "caddyfile")
 
 	time.Sleep(100 * time.Millisecond) // TODO: for some reason this test seems particularly flaky, getting 503 when it should be 200, unless we wait
 	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")

caddytest/integration/stream_test.go

@@ -176,9 +176,7 @@ func testH2ToH2CStreamServeH2C(t *testing.T) *http.Server {
 
 		w.Header().Set("Cache-Control", "no-store")
 		w.WriteHeader(200)
-		if f, ok := w.(http.Flusher); ok {
-			f.Flush()
-		}
+		http.NewResponseController(w).Flush()
 
 		buf := make([]byte, 4*1024)
 

caddytest/integration/testdata/import_respond.txt

@@ -1 +1 @@
-respond "'I am {args.0}', hears {args.1}"
+respond "'I am {args[0]}', hears {args[1]}"

cmd/cobra.go

@@ -109,12 +109,21 @@ func caddyCmdToCobra(caddyCmd Command) *cobra.Command {
 		Use:   caddyCmd.Name,
 		Short: caddyCmd.Short,
 		Long:  caddyCmd.Long,
-		RunE: func(cmd *cobra.Command, _ []string) error {
-			fls := cmd.Flags()
-			_, err := caddyCmd.Func(Flags{fls})
-			return err
-		},
 	}
-	cmd.Flags().AddGoFlagSet(caddyCmd.Flags)
+	if caddyCmd.CobraFunc != nil {
+		caddyCmd.CobraFunc(cmd)
+	} else {
+		cmd.RunE = WrapCommandFuncForCobra(caddyCmd.Func)
+		cmd.Flags().AddGoFlagSet(caddyCmd.Flags)
+	}
 	return cmd
 }
+
+// WrapCommandFuncForCobra wraps a Caddy CommandFunc for use
+// in a cobra command's RunE field.
+func WrapCommandFuncForCobra(f CommandFunc) func(cmd *cobra.Command, _ []string) error {
+	return func(cmd *cobra.Command, _ []string) error {
+		_, err := f(Flags{cmd.Flags()})
+		return err
+	}
+}

cmd/commandfuncs.go

@@ -35,6 +35,7 @@ import (
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	"github.com/caddyserver/caddy/v2/internal"
 	"go.uber.org/zap"
 )
 
@@ -208,6 +209,16 @@ func cmdRun(fl Flags) (int, error) {
 		}
 	}
 
+	// create pidfile now, in case loading config takes a while (issue #5477)
+	if runCmdPidfileFlag != "" {
+		err := caddy.PIDFile(runCmdPidfileFlag)
+		if err != nil {
+			caddy.Log().Error("unable to write PID file",
+				zap.String("pidfile", runCmdPidfileFlag),
+				zap.Error(err))
+		}
+	}
+
 	// run the initial config
 	err = caddy.Load(config, true)
 	if err != nil {
@@ -242,16 +253,6 @@ func cmdRun(fl Flags) (int, error) {
 		go watchConfigFile(configFile, runCmdConfigAdapterFlag)
 	}
 
-	// create pidfile
-	if runCmdPidfileFlag != "" {
-		err := caddy.PIDFile(runCmdPidfileFlag)
-		if err != nil {
-			caddy.Log().Error("unable to write PID file",
-				zap.String("pidfile", runCmdPidfileFlag),
-				zap.Error(err))
-		}
-	}
-
 	// warn if the environment does not provide enough information about the disk
 	hasXDG := os.Getenv("XDG_DATA_HOME") != "" &&
 		os.Getenv("XDG_CONFIG_HOME") != "" &&
@@ -490,7 +491,7 @@ func cmdAdaptConfig(fl Flags) (int, error) {
 	// validate output if requested
 	if adaptCmdValidateFlag {
 		var cfg *caddy.Config
-		err = json.Unmarshal(adaptedConfig, &cfg)
+		err = caddy.StrictUnmarshalJSON(adaptedConfig, &cfg)
 		if err != nil {
 			return caddy.ExitCodeFailedStartup, fmt.Errorf("decoding config: %v", err)
 		}
@@ -523,7 +524,7 @@ func cmdValidateConfig(fl Flags) (int, error) {
 	input = caddy.RemoveMetaFields(input)
 
 	var cfg *caddy.Config
-	err = json.Unmarshal(input, &cfg)
+	err = caddy.StrictUnmarshalJSON(input, &cfg)
 	if err != nil {
 		return caddy.ExitCodeFailedStartup, fmt.Errorf("decoding config: %v", err)
 	}
@@ -589,7 +590,10 @@ func cmdFmt(fl Flags) (int, error) {
 	}
 
 	if warning, diff := caddyfile.FormattingDifference(formatCmdConfigFile, input); diff {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("%s:%d: Caddyfile input is not formatted", warning.File, warning.Line)
+		return caddy.ExitCodeFailedStartup, fmt.Errorf(`%s:%d: Caddyfile input is not formatted; Tip: use '--overwrite' to update your Caddyfile in-place instead of previewing it. Consult '--help' for more options`,
+			warning.File,
+			warning.Line,
+		)
 	}
 
 	return caddy.ExitCodeSuccess, nil
@@ -607,7 +611,17 @@ func AdminAPIRequest(adminAddr, method, uri string, headers http.Header, body io
 	}
 	origin := "http://" + parsedAddr.JoinHostPort(0)
 	if parsedAddr.IsUnixNetwork() {
-		origin = "http://unixsocket" // hack so that http.NewRequest() is happy
+		origin = "http://127.0.0.1" // bogus host is a hack so that http.NewRequest() is happy
+
+		// the unix address at this point might still contain the optional
+		// unix socket permissions, which are part of the address/host.
+		// those need to be removed first, as they aren't part of the
+		// resulting unix file path
+		addr, _, err := internal.SplitUnixSocketPermissionsBits(parsedAddr.Host)
+		if err != nil {
+			return nil, err
+		}
+		parsedAddr.Host = addr
 	}
 
 	// form the request
@@ -616,20 +630,24 @@ func AdminAPIRequest(adminAddr, method, uri string, headers http.Header, body io
 		return nil, fmt.Errorf("making request: %v", err)
 	}
 	if parsedAddr.IsUnixNetwork() {
-		// When listening on a unix socket, the admin endpoint doesn't
-		// accept any Host header because there is no host:port for
-		// a unix socket's address. The server's host check is fairly
-		// strict for security reasons, so we don't allow just any
-		// Host header. For unix sockets, the Host header must be
-		// empty. Unfortunately, Go makes it impossible to make HTTP
-		// requests with an empty Host header... except with this one
-		// weird trick. (Hopefully they don't fix it. It's already
-		// hard enough to use HTTP over unix sockets.)
+		// We used to conform to RFC 2616 Section 14.26 which requires
+		// an empty host header when there is no host, as is the case
+		// with unix sockets. However, Go required a Host value so we
+		// used a hack of a space character as the host (it would see
+		// the Host was non-empty, then trim the space later). As of
+		// Go 1.20.6 (July 2023), this hack no longer works. See:
+		// https://github.com/golang/go/issues/60374
+		// See also the discussion here:
+		// https://github.com/golang/go/issues/61431
 		//
-		// An equivalent curl command would be something like:
-		// $ curl --unix-socket caddy.sock http:/:$REQUEST_URI
-		req.URL.Host = " "
-		req.Host = ""
+		// After that, we now require a Host value of either 127.0.0.1
+		// or ::1 if one is set. Above I choose to use 127.0.0.1. Even
+		// though the value should be completely irrelevant (it could be
+		// "srldkjfsd"), if for some reason the Host *is* used, at least
+		// we can have some reasonable assurance it will stay on the local
+		// machine and that browsers, if they ever allow access to unix
+		// sockets, can still enforce CORS, ensuring it is still coming
+		// from the local machine.
 	} else {
 		req.Header.Set("Origin", origin)
 	}

cmd/commands.go

@@ -34,12 +34,6 @@ type Command struct {
 	// Required.
 	Name string
 
-	// Func is a function that executes a subcommand using
-	// the parsed flags. It returns an exit code and any
-	// associated error.
-	// Required.
-	Func CommandFunc
-
 	// Usage is a brief message describing the syntax of
 	// the subcommand's flags and args. Use [] to indicate
 	// optional parameters and <> to enclose literal values
@@ -60,7 +54,21 @@ type Command struct {
 	Long string
 
 	// Flags is the flagset for command.
+	// This is ignored if CobraFunc is set.
 	Flags *flag.FlagSet
+
+	// Func is a function that executes a subcommand using
+	// the parsed flags. It returns an exit code and any
+	// associated error.
+	// Required if CobraFunc is not set.
+	Func CommandFunc
+
+	// CobraFunc allows further configuration of the command
+	// via cobra's APIs. If this is set, then Func and Flags
+	// are ignored, with the assumption that they are set in
+	// this function. A caddycmd.WrapCommandFuncForCobra helper
+	// exists to simplify porting CommandFunc to Cobra's RunE.
+	CobraFunc func(*cobra.Command)
 }
 
 // CommandFunc is a command's function. It runs the
@@ -79,7 +87,6 @@ var commands = make(map[string]Command)
 func init() {
 	RegisterCommand(Command{
 		Name:  "start",
-		Func:  cmdStart,
 		Usage: "[--config <path> [--adapter <name>]] [--envfile <path>] [--watch] [--pidfile <file>]",
 		Short: "Starts the Caddy process in the background and then returns",
 		Long: `
@@ -91,22 +98,21 @@ the KEY=VALUE format will be loaded into the Caddy process.
 
 On Windows, the spawned child process will remain attached to the terminal, so
 closing the window will forcefully stop Caddy; to avoid forgetting this, try
-using 'caddy run' instead to keep it in the foreground.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("start", flag.ExitOnError)
-			fs.String("config", "", "Configuration file")
-			fs.String("envfile", "", "Environment file to load")
-			fs.String("adapter", "", "Name of config adapter to apply")
-			fs.String("pidfile", "", "Path of file to which to write process ID")
-			fs.Bool("watch", false, "Reload changed config file automatically")
-			return fs
-		}(),
+using 'caddy run' instead to keep it in the foreground.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("config", "c", "", "Configuration file")
+			cmd.Flags().StringP("adapter", "a", "", "Name of config adapter to apply")
+			cmd.Flags().StringP("envfile", "", "", "Environment file to load")
+			cmd.Flags().BoolP("watch", "w", false, "Reload changed config file automatically")
+			cmd.Flags().StringP("pidfile", "", "", "Path of file to which to write process ID")
+			cmd.RunE = WrapCommandFuncForCobra(cmdStart)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "run",
-		Func:  cmdRun,
-		Usage: "[--config <path> [--adapter <name>]] [--envfile <path>] [--environ] [--resume] [--watch] [--pidfile <fil>]",
+		Usage: "[--config <path> [--adapter <name>]] [--envfile <path>] [--environ] [--resume] [--watch] [--pidfile <file>]",
 		Short: `Starts the Caddy process and blocks indefinitely`,
 		Long: `
 Starts the Caddy process, optionally bootstrapped with an initial config file,
@@ -138,44 +144,42 @@ save file. It is not an error if --resume is used and no autosave file exists.
 
 If --watch is specified, the config file will be loaded automatically after
 changes. ⚠️ This can make unintentional config changes easier; only use this
-option in a local development environment.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("run", flag.ExitOnError)
-			fs.String("config", "", "Configuration file")
-			fs.String("adapter", "", "Name of config adapter to apply")
-			fs.String("envfile", "", "Environment file to load")
-			fs.Bool("environ", false, "Print environment")
-			fs.Bool("resume", false, "Use saved config, if any (and prefer over --config file)")
-			fs.Bool("watch", false, "Watch config file for changes and reload it automatically")
-			fs.String("pidfile", "", "Path of file to which to write process ID")
-			fs.String("pingback", "", "Echo confirmation bytes to this address on success")
-			return fs
-		}(),
+option in a local development environment.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("config", "c", "", "Configuration file")
+			cmd.Flags().StringP("adapter", "a", "", "Name of config adapter to apply")
+			cmd.Flags().StringP("envfile", "", "", "Environment file to load")
+			cmd.Flags().BoolP("environ", "e", false, "Print environment")
+			cmd.Flags().BoolP("resume", "r", false, "Use saved config, if any (and prefer over --config file)")
+			cmd.Flags().BoolP("watch", "w", false, "Watch config file for changes and reload it automatically")
+			cmd.Flags().StringP("pidfile", "", "", "Path of file to which to write process ID")
+			cmd.Flags().StringP("pingback", "", "", "Echo confirmation bytes to this address on success")
+			cmd.RunE = WrapCommandFuncForCobra(cmdRun)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "stop",
-		Func:  cmdStop,
-		Usage: "[--address <interface>] [--config <path> [--adapter <name>]]",
+		Usage: "[--config <path> [--adapter <name>]] [--address <interface>]",
 		Short: "Gracefully stops a started Caddy process",
 		Long: `
 Stops the background Caddy process as gracefully as possible.
 
 It requires that the admin API is enabled and accessible, since it will
 use the API's /stop endpoint. The address of this request can be customized
-using the --address flag, or from the given --config, if not the default.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("stop", flag.ExitOnError)
-			fs.String("address", "", "The address to use to reach the admin API endpoint, if not the default")
-			fs.String("config", "", "Configuration file to use to parse the admin address, if --address is not used")
-			fs.String("adapter", "", "Name of config adapter to apply (when --config is used)")
-			return fs
-		}(),
+using the --address flag, or from the given --config, if not the default.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("config", "c", "", "Configuration file to use to parse the admin address, if --address is not used")
+			cmd.Flags().StringP("adapter", "a", "", "Name of config adapter to apply (when --config is used)")
+			cmd.Flags().StringP("address", "", "", "The address to use to reach the admin API endpoint, if not the default")
+			cmd.RunE = WrapCommandFuncForCobra(cmdStop)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "reload",
-		Func:  cmdReload,
 		Usage: "--config <path> [--adapter <name>] [--address <interface>]",
 		Short: "Changes the config of the running Caddy instance",
 		Long: `
@@ -185,20 +189,19 @@ workflows revolving around config files.
 
 Since the admin endpoint is configurable, the endpoint configuration is loaded
 from the --address flag if specified; otherwise it is loaded from the given
-config file; otherwise the default is assumed.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("reload", flag.ExitOnError)
-			fs.String("config", "", "Configuration file (required)")
-			fs.String("adapter", "", "Name of config adapter to apply")
-			fs.String("address", "", "Address of the administration listener, if different from config")
-			fs.Bool("force", false, "Force config reload, even if it is the same")
-			return fs
-		}(),
+config file; otherwise the default is assumed.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("config", "c", "", "Configuration file (required)")
+			cmd.Flags().StringP("adapter", "a", "", "Name of config adapter to apply")
+			cmd.Flags().StringP("address", "", "", "Address of the administration listener, if different from config")
+			cmd.Flags().BoolP("force", "f", false, "Force config reload, even if it is the same")
+			cmd.RunE = WrapCommandFuncForCobra(cmdReload)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "version",
-		Func:  cmdVersion,
 		Short: "Prints the version",
 		Long: `
 Prints the version of this Caddy binary.
@@ -213,31 +216,29 @@ detailed version information is printed as given by Go modules.
 For more details about the full version string, see the Go module
 documentation: https://go.dev/doc/modules/version-numbers
 `,
+		Func: cmdVersion,
 	})
 
 	RegisterCommand(Command{
 		Name:  "list-modules",
-		Func:  cmdListModules,
-		Usage: "[--packages] [--versions]",
+		Usage: "[--packages] [--versions] [--skip-standard]",
 		Short: "Lists the installed Caddy modules",
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("list-modules", flag.ExitOnError)
-			fs.Bool("packages", false, "Print package paths")
-			fs.Bool("versions", false, "Print version information")
-			fs.Bool("skip-standard", false, "Skip printing standard modules")
-			return fs
-		}(),
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().BoolP("packages", "", false, "Print package paths")
+			cmd.Flags().BoolP("versions", "", false, "Print version information")
+			cmd.Flags().BoolP("skip-standard", "s", false, "Skip printing standard modules")
+			cmd.RunE = WrapCommandFuncForCobra(cmdListModules)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "build-info",
-		Func:  cmdBuildInfo,
 		Short: "Prints information about this build",
+		Func:  cmdBuildInfo,
 	})
 
 	RegisterCommand(Command{
 		Name:  "environ",
-		Func:  cmdEnviron,
 		Short: "Prints the environment",
 		Long: `
 Prints the environment as seen by this Caddy process.
@@ -257,11 +258,11 @@ by adding the "--environ" flag.
 
 Environments may contain sensitive data.
 `,
+		Func: cmdEnviron,
 	})
 
 	RegisterCommand(Command{
 		Name:  "adapt",
-		Func:  cmdAdaptConfig,
 		Usage: "--config <path> [--adapter <name>] [--pretty] [--validate]",
 		Short: "Adapts a configuration to Caddy's native JSON",
 		Long: `
@@ -273,20 +274,19 @@ for human readability.
 
 If --validate is used, the adapted config will be checked for validity.
 If the config is invalid, an error will be printed to stderr and a non-
-zero exit status will be returned.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("adapt", flag.ExitOnError)
-			fs.String("config", "", "Configuration file to adapt (required)")
-			fs.String("adapter", "caddyfile", "Name of config adapter")
-			fs.Bool("pretty", false, "Format the output for human readability")
-			fs.Bool("validate", false, "Validate the output")
-			return fs
-		}(),
+zero exit status will be returned.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("config", "c", "", "Configuration file to adapt (required)")
+			cmd.Flags().StringP("adapter", "a", "caddyfile", "Name of config adapter")
+			cmd.Flags().BoolP("pretty", "p", false, "Format the output for human readability")
+			cmd.Flags().BoolP("validate", "", false, "Validate the output")
+			cmd.RunE = WrapCommandFuncForCobra(cmdAdaptConfig)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "validate",
-		Func:  cmdValidateConfig,
 		Usage: "--config <path> [--adapter <name>] [--envfile <path>]",
 		Short: "Tests whether a configuration file is valid",
 		Long: `
@@ -295,20 +295,69 @@ This reveals any errors with the configuration through the loading and
 provisioning stages.
 
 If --envfile is specified, an environment file with environment variables in
-the KEY=VALUE format will be loaded into the Caddy process.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("validate", flag.ExitOnError)
-			fs.String("config", "", "Input configuration file")
-			fs.String("adapter", "", "Name of config adapter")
-			fs.String("envfile", "", "Environment file to load")
-			return fs
-		}(),
+the KEY=VALUE format will be loaded into the Caddy process.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("config", "c", "", "Input configuration file")
+			cmd.Flags().StringP("adapter", "a", "", "Name of config adapter")
+			cmd.Flags().StringP("envfile", "", "", "Environment file to load")
+			cmd.RunE = WrapCommandFuncForCobra(cmdValidateConfig)
+		},
+	})
+
+	RegisterCommand(Command{
+		Name:  "storage",
+		Short: "Commands for working with Caddy's storage (EXPERIMENTAL)",
+		Long: `
+Allows exporting and importing Caddy's storage contents. The two commands can be
+combined in a pipeline to transfer directly from one storage to another:
+
+$ caddy storage export --config Caddyfile.old --output - |
+> caddy storage import --config Caddyfile.new --input -
+
+The - argument refers to stdout and stdin, respectively.
+
+NOTE: When importing to or exporting from file_system storage (the default), the command
+should be run as the user that owns the associated root path.
+
+EXPERIMENTAL: May be changed or removed.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			exportCmd := &cobra.Command{
+				Use:   "export --config <path> --output <path>",
+				Short: "Exports storage assets as a tarball",
+				Long: `
+The contents of the configured storage module (TLS certificates, etc)
+are exported via a tarball.
+
+--output is required, - can be given for stdout.
+`,
+				RunE: WrapCommandFuncForCobra(cmdExportStorage),
+			}
+			exportCmd.Flags().StringP("config", "c", "", "Input configuration file (required)")
+			exportCmd.Flags().StringP("output", "o", "", "Output path")
+			cmd.AddCommand(exportCmd)
+
+			importCmd := &cobra.Command{
+				Use:   "import --config <path> --input <path>",
+				Short: "Imports storage assets from a tarball.",
+				Long: `
+Imports storage assets to the configured storage module. The import file must be
+a tar archive.
+
+--input is required, - can be given for stdin.
+`,
+				RunE: WrapCommandFuncForCobra(cmdImportStorage),
+			}
+			importCmd.Flags().StringP("config", "c", "", "Configuration file to load (required)")
+			importCmd.Flags().StringP("input", "i", "", "Tar of assets to load (required)")
+			cmd.AddCommand(importCmd)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "fmt",
-		Func:  cmdFmt,
-		Usage: "[--overwrite] [<path>]",
+		Usage: "[--overwrite] [--diff] [<path>]",
 		Short: "Formats a Caddyfile",
 		Long: `
 Formats the Caddyfile by adding proper indentation and spaces to improve
@@ -324,32 +373,30 @@ is not a valid patch format.
 
 If you wish you use stdin instead of a regular file, use - as the path.
 When reading from stdin, the --overwrite flag has no effect: the result
-is always printed to stdout.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("fmt", flag.ExitOnError)
-			fs.Bool("overwrite", false, "Overwrite the input file with the results")
-			fs.Bool("diff", false, "Print the differences between the input file and the formatted output")
-			return fs
-		}(),
+is always printed to stdout.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().BoolP("overwrite", "w", false, "Overwrite the input file with the results")
+			cmd.Flags().BoolP("diff", "d", false, "Print the differences between the input file and the formatted output")
+			cmd.RunE = WrapCommandFuncForCobra(cmdFmt)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "upgrade",
-		Func:  cmdUpgrade,
 		Short: "Upgrade Caddy (EXPERIMENTAL)",
 		Long: `
 Downloads an updated Caddy binary with the same modules/plugins at the
-latest versions. EXPERIMENTAL: May be changed or removed.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("upgrade", flag.ExitOnError)
-			fs.Bool("keep-backup", false, "Keep the backed up binary, instead of deleting it")
-			return fs
-		}(),
+latest versions. EXPERIMENTAL: May be changed or removed.
+`,
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().BoolP("keep-backup", "k", false, "Keep the backed up binary, instead of deleting it")
+			cmd.RunE = WrapCommandFuncForCobra(cmdUpgrade)
+		},
 	})
 
 	RegisterCommand(Command{
 		Name:  "add-package",
-		Func:  cmdAddPackage,
 		Usage: "<packages...>",
 		Short: "Adds Caddy packages (EXPERIMENTAL)",
 		Long: `
@@ -357,11 +404,10 @@ Downloads an updated Caddy binary with the specified packages (module/plugin)
 added. Retains existing packages. Returns an error if the any of packages are 
 already included. EXPERIMENTAL: May be changed or removed.
 `,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("add-package", flag.ExitOnError)
-			fs.Bool("keep-backup", false, "Keep the backed up binary, instead of deleting it")
-			return fs
-		}(),
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().BoolP("keep-backup", "k", false, "Keep the backed up binary, instead of deleting it")
+			cmd.RunE = WrapCommandFuncForCobra(cmdAddPackage)
+		},
 	})
 
 	RegisterCommand(Command{
@@ -374,31 +420,14 @@ Downloads an updated Caddy binaries without the specified packages (module/plugi
 Returns an error if any of the packages are not included. 
 EXPERIMENTAL: May be changed or removed.
 `,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("remove-package", flag.ExitOnError)
-			fs.Bool("keep-backup", false, "Keep the backed up binary, instead of deleting it")
-			return fs
-		}(),
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().BoolP("keep-backup", "k", false, "Keep the backed up binary, instead of deleting it")
+			cmd.RunE = WrapCommandFuncForCobra(cmdRemovePackage)
+		},
 	})
 
 	RegisterCommand(Command{
-		Name: "manpage",
-		Func: func(fl Flags) (int, error) {
-			dir := strings.TrimSpace(fl.String("directory"))
-			if dir == "" {
-				return caddy.ExitCodeFailedQuit, fmt.Errorf("designated output directory and specified section are required")
-			}
-			if err := os.MkdirAll(dir, 0755); err != nil {
-				return caddy.ExitCodeFailedQuit, err
-			}
-			if err := doc.GenManTree(rootCmd, &doc.GenManHeader{
-				Title:   "Caddy",
-				Section: "8", // https://en.wikipedia.org/wiki/Man_page#Manual_sections
-			}, dir); err != nil {
-				return caddy.ExitCodeFailedQuit, err
-			}
-			return caddy.ExitCodeSuccess, nil
-		},
+		Name:  "manpage",
 		Usage: "--directory <path>",
 		Short: "Generates the manual pages for Caddy commands",
 		Long: `
@@ -408,11 +437,25 @@ tagged into section 8 (System Administration).
 The manual page files are generated into the directory specified by the
 argument of --directory. If the directory does not exist, it will be created.
 `,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("manpage", flag.ExitOnError)
-			fs.String("directory", "", "The output directory where the manpages are generated")
-			return fs
-		}(),
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("directory", "o", "", "The output directory where the manpages are generated")
+			cmd.RunE = WrapCommandFuncForCobra(func(fl Flags) (int, error) {
+				dir := strings.TrimSpace(fl.String("directory"))
+				if dir == "" {
+					return caddy.ExitCodeFailedQuit, fmt.Errorf("designated output directory and specified section are required")
+				}
+				if err := os.MkdirAll(dir, 0755); err != nil {
+					return caddy.ExitCodeFailedQuit, err
+				}
+				if err := doc.GenManTree(rootCmd, &doc.GenManHeader{
+					Title:   "Caddy",
+					Section: "8", // https://en.wikipedia.org/wiki/Man_page#Manual_sections
+				}, dir); err != nil {
+					return caddy.ExitCodeFailedQuit, err
+				}
+				return caddy.ExitCodeSuccess, nil
+			})
+		},
 	})
 
 	// source: https://github.com/spf13/cobra/blob/main/shell_completions.md
@@ -496,7 +539,7 @@ func RegisterCommand(cmd Command) {
 	if cmd.Name == "" {
 		panic("command name is required")
 	}
-	if cmd.Func == nil {
+	if cmd.Func == nil && cmd.CobraFunc == nil {
 		panic("command function missing")
 	}
 	if cmd.Short == "" {

cmd/main.go

@@ -89,6 +89,10 @@ func handlePingbackConn(conn net.Conn, expect []byte) error {
 // and returns the resulting JSON config bytes along with
 // the name of the loaded config file (if any).
 func LoadConfig(configFile, adapterName string) ([]byte, string, error) {
+	return loadConfigWithLogger(caddy.Log(), configFile, adapterName)
+}
+
+func loadConfigWithLogger(logger *zap.Logger, configFile, adapterName string) ([]byte, string, error) {
 	// specifying an adapter without a config file is ambiguous
 	if adapterName != "" && configFile == "" {
 		return nil, "", fmt.Errorf("cannot adapt config without config file (use --config)")
@@ -107,9 +111,11 @@ func LoadConfig(configFile, adapterName string) ([]byte, string, error) {
 		if err != nil {
 			return nil, "", fmt.Errorf("reading config file: %v", err)
 		}
-		caddy.Log().Info("using provided configuration",
-			zap.String("config_file", configFile),
-			zap.String("config_adapter", adapterName))
+		if logger != nil {
+			logger.Info("using provided configuration",
+				zap.String("config_file", configFile),
+				zap.String("config_adapter", adapterName))
+		}
 	} else if adapterName == "" {
 		// as a special case when no config file or adapter
 		// is specified, see if the Caddyfile adapter is
@@ -126,7 +132,9 @@ func LoadConfig(configFile, adapterName string) ([]byte, string, error) {
 			} else {
 				// success reading default Caddyfile
 				configFile = "Caddyfile"
-				caddy.Log().Info("using adjacent Caddyfile")
+				if logger != nil {
+					logger.Info("using adjacent Caddyfile")
+				}
 			}
 		}
 	}
@@ -161,7 +169,9 @@ func LoadConfig(configFile, adapterName string) ([]byte, string, error) {
 			if warn.Directive != "" {
 				msg = fmt.Sprintf("%s: %s", warn.Directive, warn.Message)
 			}
-			caddy.Log().Warn(msg, zap.String("adapter", adapterName), zap.String("file", warn.File), zap.Int("line", warn.Line))
+			if logger != nil {
+				logger.Warn(msg, zap.String("adapter", adapterName), zap.String("file", warn.File), zap.Int("line", warn.Line))
+			}
 		}
 		config = adaptedConfig
 	}
@@ -174,6 +184,8 @@ func LoadConfig(configFile, adapterName string) ([]byte, string, error) {
 // blocks indefinitely; it only quits if the poller has errors for
 // long enough time. The filename passed in must be the actual
 // config file used, not one to be discovered.
+// Each second the config files is loaded and parsed into an object
+// and is compared to the last config object that was loaded
 func watchConfigFile(filename, adapterName string) {
 	defer func() {
 		if err := recover(); err != nil {
@@ -189,64 +201,36 @@ func watchConfigFile(filename, adapterName string) {
 			With(zap.String("config_file", filename))
 	}
 
-	// get the initial timestamp on the config file
-	info, err := os.Stat(filename)
+	// get current config
+	lastCfg, _, err := loadConfigWithLogger(nil, filename, adapterName)
 	if err != nil {
-		logger().Error("cannot watch config file", zap.Error(err))
+		logger().Error("unable to load latest config", zap.Error(err))
 		return
 	}
-	lastModified := info.ModTime()
 
 	logger().Info("watching config file for changes")
 
-	// if the file disappears or something, we can
-	// stop polling if the error lasts long enough
-	var lastErr time.Time
-	finalError := func(err error) bool {
-		if lastErr.IsZero() {
-			lastErr = time.Now()
-			return false
-		}
-		if time.Since(lastErr) > 30*time.Second {
-			logger().Error("giving up watching config file; too many errors",
-				zap.Error(err))
-			return true
-		}
-		return false
-	}
-
 	// begin poller
 	//nolint:staticcheck
 	for range time.Tick(1 * time.Second) {
-		// get the file info
-		info, err := os.Stat(filename)
-		if err != nil {
-			if finalError(err) {
-				return
-			}
-			continue
-		}
-		lastErr = time.Time{} // no error, so clear any memory of one
-
-		// if it hasn't changed, nothing to do
-		if !info.ModTime().After(lastModified) {
-			continue
-		}
-
-		logger().Info("config file changed; reloading")
-
-		// remember this timestamp
-		lastModified = info.ModTime()
-
-		// load the contents of the file
-		config, _, err := LoadConfig(filename, adapterName)
+		// get current config
+		newCfg, _, err := loadConfigWithLogger(nil, filename, adapterName)
 		if err != nil {
 			logger().Error("unable to load latest config", zap.Error(err))
-			continue
+			return
 		}
 
+		// if it hasn't changed, nothing to do
+		if bytes.Equal(lastCfg, newCfg) {
+			continue
+		}
+		logger().Info("config file changed; reloading")
+
+		// remember the current config
+		lastCfg = newCfg
+
 		// apply the updated config
-		err = caddy.Load(config, false)
+		err = caddy.Load(lastCfg, false)
 		if err != nil {
 			logger().Error("applying latest config", zap.Error(err))
 			continue
@@ -374,18 +358,19 @@ func parseEnvFile(envInput io.Reader) (map[string]string, error) {
 		}
 
 		// quoted value: support newlines
-		if strings.HasPrefix(val, `"`) {
-			for !(strings.HasSuffix(line, `"`) && !strings.HasSuffix(line, `\"`)) {
-				val = strings.ReplaceAll(val, `\"`, `"`)
+		if strings.HasPrefix(val, `"`) || strings.HasPrefix(val, "'") {
+			quote := string(val[0])
+			for !(strings.HasSuffix(line, quote) && !strings.HasSuffix(line, `\`+quote)) {
+				val = strings.ReplaceAll(val, `\`+quote, quote)
 				if !scanner.Scan() {
 					break
 				}
 				lineNumber++
-				line = strings.ReplaceAll(scanner.Text(), `\"`, `"`)
+				line = strings.ReplaceAll(scanner.Text(), `\`+quote, quote)
 				val += "\n" + line
 			}
-			val = strings.TrimPrefix(val, `"`)
-			val = strings.TrimSuffix(val, `"`)
+			val = strings.TrimPrefix(val, quote)
+			val = strings.TrimSuffix(val, quote)
 		}
 
 		envMap[key] = val

cmd/storagefuncs.go

@@ -0,0 +1,220 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddycmd
+
+import (
+	"archive/tar"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+
+	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/certmagic"
+)
+
+type storVal struct {
+	StorageRaw json.RawMessage `json:"storage,omitempty" caddy:"namespace=caddy.storage inline_key=module"`
+}
+
+// determineStorage returns the top-level storage module from the given config.
+// It may return nil even if no error.
+func determineStorage(configFile string, configAdapter string) (*storVal, error) {
+	cfg, _, err := LoadConfig(configFile, configAdapter)
+	if err != nil {
+		return nil, err
+	}
+
+	// storage defaults to FileStorage if not explicitly
+	// defined in the config, so the config can be valid
+	// json but unmarshaling will fail.
+	if !json.Valid(cfg) {
+		return nil, &json.SyntaxError{}
+	}
+	var tmpStruct storVal
+	err = json.Unmarshal(cfg, &tmpStruct)
+	if err != nil {
+		// default case, ignore the error
+		var jsonError *json.SyntaxError
+		if errors.As(err, &jsonError) {
+			return nil, nil
+		}
+		return nil, err
+	}
+
+	return &tmpStruct, nil
+}
+
+func cmdImportStorage(fl Flags) (int, error) {
+	importStorageCmdConfigFlag := fl.String("config")
+	importStorageCmdImportFile := fl.String("input")
+
+	if importStorageCmdConfigFlag == "" {
+		return caddy.ExitCodeFailedStartup, errors.New("--config is required")
+	}
+	if importStorageCmdImportFile == "" {
+		return caddy.ExitCodeFailedStartup, errors.New("--input is required")
+	}
+
+	// extract storage from config if possible
+	storageCfg, err := determineStorage(importStorageCmdConfigFlag, "")
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	// load specified storage or fallback to default
+	var stor certmagic.Storage
+	ctx, cancel := caddy.NewContext(caddy.Context{Context: context.Background()})
+	defer cancel()
+	if storageCfg != nil && storageCfg.StorageRaw != nil {
+		val, err := ctx.LoadModule(storageCfg, "StorageRaw")
+		if err != nil {
+			return caddy.ExitCodeFailedStartup, err
+		}
+		stor, err = val.(caddy.StorageConverter).CertMagicStorage()
+		if err != nil {
+			return caddy.ExitCodeFailedStartup, err
+		}
+	} else {
+		stor = caddy.DefaultStorage
+	}
+
+	// setup input
+	var f *os.File
+	if importStorageCmdImportFile == "-" {
+		f = os.Stdin
+	} else {
+		f, err = os.Open(importStorageCmdImportFile)
+		if err != nil {
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("opening input file: %v", err)
+		}
+		defer f.Close()
+	}
+
+	// store each archive element
+	tr := tar.NewReader(f)
+	for {
+		hdr, err := tr.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			return caddy.ExitCodeFailedQuit, fmt.Errorf("reading archive: %v", err)
+		}
+
+		b, err := io.ReadAll(tr)
+		if err != nil {
+			return caddy.ExitCodeFailedQuit, fmt.Errorf("reading archive: %v", err)
+		}
+
+		err = stor.Store(ctx, hdr.Name, b)
+		if err != nil {
+			return caddy.ExitCodeFailedQuit, fmt.Errorf("reading archive: %v", err)
+		}
+	}
+
+	fmt.Println("Successfully imported storage")
+	return caddy.ExitCodeSuccess, nil
+}
+
+func cmdExportStorage(fl Flags) (int, error) {
+	exportStorageCmdConfigFlag := fl.String("config")
+	exportStorageCmdOutputFlag := fl.String("output")
+
+	if exportStorageCmdConfigFlag == "" {
+		return caddy.ExitCodeFailedStartup, errors.New("--config is required")
+	}
+	if exportStorageCmdOutputFlag == "" {
+		return caddy.ExitCodeFailedStartup, errors.New("--output is required")
+	}
+
+	// extract storage from config if possible
+	storageCfg, err := determineStorage(exportStorageCmdConfigFlag, "")
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	// load specified storage or fallback to default
+	var stor certmagic.Storage
+	ctx, cancel := caddy.NewContext(caddy.Context{Context: context.Background()})
+	defer cancel()
+	if storageCfg != nil && storageCfg.StorageRaw != nil {
+		val, err := ctx.LoadModule(storageCfg, "StorageRaw")
+		if err != nil {
+			return caddy.ExitCodeFailedStartup, err
+		}
+		stor, err = val.(caddy.StorageConverter).CertMagicStorage()
+		if err != nil {
+			return caddy.ExitCodeFailedStartup, err
+		}
+	} else {
+		stor = caddy.DefaultStorage
+	}
+
+	// enumerate all keys
+	keys, err := stor.List(ctx, "", true)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	// setup output
+	var f *os.File
+	if exportStorageCmdOutputFlag == "-" {
+		f = os.Stdout
+	} else {
+		f, err = os.Create(exportStorageCmdOutputFlag)
+		if err != nil {
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("opening output file: %v", err)
+		}
+		defer f.Close()
+	}
+
+	// `IsTerminal: true` keys hold the values we
+	// care about, write them out
+	tw := tar.NewWriter(f)
+	for _, k := range keys {
+		info, err := stor.Stat(ctx, k)
+		if err != nil {
+			return caddy.ExitCodeFailedQuit, err
+		}
+
+		if info.IsTerminal {
+			v, err := stor.Load(ctx, k)
+			if err != nil {
+				return caddy.ExitCodeFailedQuit, err
+			}
+
+			hdr := &tar.Header{
+				Name: k,
+				Mode: 0600,
+				Size: int64(len(v)),
+			}
+
+			if err = tw.WriteHeader(hdr); err != nil {
+				return caddy.ExitCodeFailedQuit, fmt.Errorf("writing archive: %v", err)
+			}
+			if _, err = tw.Write(v); err != nil {
+				return caddy.ExitCodeFailedQuit, fmt.Errorf("writing archive: %v", err)
+			}
+		}
+	}
+	if err = tw.Close(); err != nil {
+		return caddy.ExitCodeFailedQuit, fmt.Errorf("writing archive: %v", err)
+	}
+
+	return caddy.ExitCodeSuccess, nil
+}

context.go

@@ -326,7 +326,7 @@ func (ctx Context) LoadModuleByID(id string, rawMsg json.RawMessage) (any, error
 
 	// fill in its config only if there is a config to fill in
 	if len(rawMsg) > 0 {
-		err := strictUnmarshalJSON(rawMsg, &val)
+		err := StrictUnmarshalJSON(rawMsg, &val)
 		if err != nil {
 			return nil, fmt.Errorf("decoding module config: %s: %v", modInfo, err)
 		}
@@ -410,6 +410,11 @@ func (ctx Context) loadModuleInline(moduleNameKey, moduleScope string, raw json.
 // called during the Provision/Validate phase to reference a
 // module's own host app (since the parent app module is still
 // in the process of being provisioned, it is not yet ready).
+//
+// We return any type instead of the App type because it is NOT
+// intended for the caller of this method to be the one to start
+// or stop App modules. The caller is expected to assert to the
+// concrete type.
 func (ctx Context) App(name string) (any, error) {
 	if app, ok := ctx.cfg.apps[name]; ok {
 		return app, nil
@@ -426,15 +431,23 @@ func (ctx Context) App(name string) (any, error) {
 	return modVal, nil
 }
 
-// AppIsConfigured returns whether an app named name has been
-// configured. Can be called before calling App() to avoid
-// instantiating an empty app when that's not desirable.
-func (ctx Context) AppIsConfigured(name string) bool {
-	if _, ok := ctx.cfg.apps[name]; ok {
-		return true
+// AppIfConfigured returns an app by its name if it has been
+// configured. Can be called instead of App() to avoid
+// instantiating an empty app when that's not desirable. If
+// the app has not been loaded, nil is returned.
+//
+// We return any type instead of the App type because it is not
+// intended for the caller of this method to be the one to start
+// or stop App modules. The caller is expected to assert to the
+// concrete type.
+func (ctx Context) AppIfConfigured(name string) any {
+	if ctx.cfg == nil {
+		// this can happen if the currently-active context
+		// is being accessed, but no config has successfully
+		// been loaded yet
+		return nil
 	}
-	appRaw := ctx.cfg.AppsRaw[name]
-	return appRaw != nil
+	return ctx.cfg.apps[name]
 }
 
 // Storage returns the configured Caddy storage implementation.
@@ -475,6 +488,9 @@ func (ctx Context) Logger(module ...Module) *zap.Logger {
 	if len(module) > 0 {
 		mod = module[0]
 	}
+	if mod == nil {
+		return Log()
+	}
 	return ctx.cfg.Logging.Logger(mod)
 }
 

go.mod

@@ -1,59 +1,73 @@
 module github.com/caddyserver/caddy/v2
 
-go 1.18
+go 1.20
 
 require (
-	github.com/BurntSushi/toml v1.2.1
+	github.com/BurntSushi/toml v1.3.2
 	github.com/Masterminds/sprig/v3 v3.2.3
-	github.com/alecthomas/chroma/v2 v2.5.0
+	github.com/alecthomas/chroma/v2 v2.7.0
 	github.com/aryann/difflib v0.0.0-20210328193216-ff5ff6dc229b
-	github.com/caddyserver/certmagic v0.17.2
+	github.com/caddyserver/certmagic v0.19.1
 	github.com/dustin/go-humanize v1.0.1
 	github.com/go-chi/chi v4.1.2+incompatible
-	github.com/google/cel-go v0.13.0
+	github.com/google/cel-go v0.15.1
 	github.com/google/uuid v1.3.0
-	github.com/klauspost/compress v1.15.15
-	github.com/klauspost/cpuid/v2 v2.2.3
-	github.com/mholt/acmez v1.1.0
+	github.com/klauspost/compress v1.16.7
+	github.com/klauspost/cpuid/v2 v2.2.5
+	github.com/mastercactapus/proxyprotocol v0.0.4
+	github.com/mholt/acmez v1.2.0
 	github.com/prometheus/client_golang v1.14.0
-	github.com/quic-go/quic-go v0.32.0
-	github.com/smallstep/certificates v0.23.2
-	github.com/smallstep/nosql v0.5.0
+	github.com/quic-go/quic-go v0.37.3
+	github.com/smallstep/certificates v0.24.3-rc.5
+	github.com/smallstep/nosql v0.6.0
 	github.com/smallstep/truststore v0.12.1
-	github.com/spf13/cobra v1.6.1
+	github.com/spf13/cobra v1.7.0
 	github.com/spf13/pflag v1.0.5
-	github.com/tailscale/tscert v0.0.0-20230124224810-c6dc1f4049b2
-	github.com/yuin/goldmark v1.5.4
-	github.com/yuin/goldmark-highlighting/v2 v2.0.0-20220924101305-151362477c87
-	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.39.0
-	go.opentelemetry.io/otel v1.13.0
-	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.4.0
-	go.opentelemetry.io/otel/sdk v1.13.0
-	go.uber.org/zap v1.24.0
-	golang.org/x/crypto v0.5.0
-	golang.org/x/net v0.7.0
-	golang.org/x/sync v0.1.0
-	golang.org/x/term v0.5.0
-	google.golang.org/genproto v0.0.0-20230202175211-008b39050e57
+	github.com/stretchr/testify v1.8.4
+	github.com/tailscale/tscert v0.0.0-20230509043813-4e9cb4f2b4ad
+	github.com/yuin/goldmark v1.5.5
+	github.com/yuin/goldmark-highlighting/v2 v2.0.0-20230729083705-37449abec8cc
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.42.0
+	go.opentelemetry.io/contrib/propagators/autoprop v0.42.0
+	go.opentelemetry.io/otel v1.16.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.16.0
+	go.opentelemetry.io/otel/sdk v1.16.0
+	go.uber.org/zap v1.25.0
+	golang.org/x/crypto v0.11.0
+	golang.org/x/exp v0.0.0-20230310171629-522b1b587ee0
+	golang.org/x/net v0.12.0
+	golang.org/x/sync v0.3.0
+	golang.org/x/term v0.10.0
+	google.golang.org/genproto/googleapis/api v0.0.0-20230706204954-ccb25ca9f130
 	gopkg.in/natefinch/lumberjack.v2 v2.2.1
 	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
 	github.com/Microsoft/go-winio v0.6.0 // indirect
+	github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.0.0-20230305170008-8188dc5388df // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/fxamacker/cbor/v2 v2.4.0 // indirect
-	github.com/golang/glog v1.0.0 // indirect
+	github.com/golang/glog v1.1.0 // indirect
 	github.com/golang/mock v1.6.0 // indirect
-	github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 // indirect
-	github.com/onsi/ginkgo/v2 v2.2.0 // indirect
+	github.com/google/certificate-transparency-go v1.1.4 // indirect
+	github.com/google/go-tpm v0.3.3 // indirect
+	github.com/google/go-tspi v0.3.0 // indirect
+	github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0 // indirect
+	github.com/onsi/ginkgo/v2 v2.9.5 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/quic-go/qpack v0.4.0 // indirect
-	github.com/quic-go/qtls-go1-18 v0.2.0 // indirect
-	github.com/quic-go/qtls-go1-19 v0.2.0 // indirect
-	github.com/quic-go/qtls-go1-20 v0.1.0 // indirect
-	github.com/rogpeppe/go-internal v1.9.0 // indirect
+	github.com/quic-go/qtls-go1-20 v0.3.1 // indirect
+	github.com/smallstep/go-attestation v0.4.4-0.20230509120429-e17291421738 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
-	golang.org/x/exp v0.0.0-20221205204356-47842c84f3db // indirect
+	github.com/zeebo/blake3 v0.2.3 // indirect
+	go.opentelemetry.io/contrib/propagators/aws v1.17.0 // indirect
+	go.opentelemetry.io/contrib/propagators/b3 v1.17.0 // indirect
+	go.opentelemetry.io/contrib/propagators/jaeger v1.17.0 // indirect
+	go.opentelemetry.io/contrib/propagators/ot v1.17.0 // indirect
+	google.golang.org/genproto v0.0.0-20230706204954-ccb25ca9f130 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98 // indirect
 )
 
 require (
@@ -61,12 +75,11 @@ require (
 	github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 // indirect
 	github.com/Masterminds/goutils v1.1.1 // indirect
 	github.com/Masterminds/semver/v3 v3.2.0 // indirect
-	github.com/antlr/antlr4/runtime/Go/antlr v1.4.10 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/cenkalti/backoff/v4 v4.1.2 // indirect
+	github.com/cenkalti/backoff/v4 v4.2.1 // indirect
 	github.com/cespare/xxhash v1.1.0 // indirect
-	github.com/cespare/xxhash/v2 v2.1.2 // indirect
-	github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e // indirect
+	github.com/cespare/xxhash/v2 v2.2.0 // indirect
+	github.com/chzyer/readline v1.5.1 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
 	github.com/dgraph-io/badger v1.6.2 // indirect
 	github.com/dgraph-io/badger/v2 v2.2007.4 // indirect
@@ -76,32 +89,31 @@ require (
 	github.com/felixge/httpsnoop v1.0.3 // indirect
 	github.com/go-kit/kit v0.10.0 // indirect
 	github.com/go-logfmt/logfmt v0.5.1 // indirect
-	github.com/go-logr/logr v1.2.3 // indirect
+	github.com/go-logr/logr v1.2.4 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
-	github.com/go-sql-driver/mysql v1.6.0 // indirect
-	github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect
-	github.com/golang/protobuf v1.5.2 // indirect
+	github.com/go-sql-driver/mysql v1.7.0 // indirect
+	github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect
+	github.com/golang/protobuf v1.5.3 // indirect
 	github.com/golang/snappy v0.0.4 // indirect
-	github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
 	github.com/huandu/xstrings v1.3.3 // indirect
 	github.com/imdario/mergo v0.3.12 // indirect
-	github.com/inconshreveable/mousetrap v1.0.1 // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jackc/chunkreader/v2 v2.0.1 // indirect
-	github.com/jackc/pgconn v1.13.0 // indirect
+	github.com/jackc/pgconn v1.14.0 // indirect
 	github.com/jackc/pgio v1.0.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
-	github.com/jackc/pgproto3/v2 v2.3.1 // indirect
-	github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
-	github.com/jackc/pgtype v1.12.0 // indirect
-	github.com/jackc/pgx/v4 v4.17.2 // indirect
+	github.com/jackc/pgproto3/v2 v2.3.2 // indirect
+	github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
+	github.com/jackc/pgtype v1.14.0 // indirect
+	github.com/jackc/pgx/v4 v4.18.0 // indirect
 	github.com/libdns/libdns v0.2.1 // indirect
 	github.com/manifoldco/promptui v0.9.0 // indirect
 	github.com/mattn/go-colorable v0.1.8 // indirect
-	github.com/mattn/go-isatty v0.0.13 // indirect
+	github.com/mattn/go-isatty v0.0.16 // indirect
 	github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
 	github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
 	github.com/micromdm/scep/v2 v2.1.0 // indirect
-	github.com/miekg/dns v1.1.50 // indirect
+	github.com/miekg/dns v1.1.55 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/go-ps v1.0.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
@@ -109,33 +121,32 @@ require (
 	github.com/prometheus/client_model v0.3.0 // indirect
 	github.com/prometheus/common v0.37.0 // indirect
 	github.com/prometheus/procfs v0.8.0 // indirect
-	github.com/rs/xid v1.4.0 // indirect
+	github.com/rs/xid v1.5.0 // indirect
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
 	github.com/shopspring/decimal v1.2.0 // indirect
 	github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
-	github.com/sirupsen/logrus v1.9.0 // indirect
+	github.com/sirupsen/logrus v1.9.3 // indirect
 	github.com/slackhq/nebula v1.6.1 // indirect
 	github.com/spf13/cast v1.4.1 // indirect
 	github.com/stoewer/go-strcase v1.2.0 // indirect
-	github.com/urfave/cli v1.22.12 // indirect
-	go.etcd.io/bbolt v1.3.6 // indirect
+	github.com/urfave/cli v1.22.14 // indirect
+	go.etcd.io/bbolt v1.3.7 // indirect
 	go.mozilla.org/pkcs7 v0.0.0-20210826202110-33d05740a352 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.4.0 // indirect
-	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.4.0 // indirect
-	go.opentelemetry.io/otel/metric v0.36.0 // indirect
-	go.opentelemetry.io/otel/trace v1.13.0 // indirect
-	go.opentelemetry.io/proto/otlp v0.12.0 // indirect
-	go.step.sm/cli-utils v0.7.5 // indirect
-	go.step.sm/crypto v0.23.2
-	go.step.sm/linkedca v0.19.0 // indirect
-	go.uber.org/atomic v1.9.0 // indirect
-	go.uber.org/multierr v1.6.0 // indirect
-	golang.org/x/mod v0.6.0 // indirect
-	golang.org/x/sys v0.5.0
-	golang.org/x/text v0.7.0 // indirect
-	golang.org/x/tools v0.2.0 // indirect
-	google.golang.org/grpc v1.52.3 // indirect
-	google.golang.org/protobuf v1.28.1 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.16.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.16.0 // indirect
+	go.opentelemetry.io/otel/metric v1.16.0 // indirect
+	go.opentelemetry.io/otel/trace v1.16.0
+	go.opentelemetry.io/proto/otlp v0.19.0 // indirect
+	go.step.sm/cli-utils v0.8.0 // indirect
+	go.step.sm/crypto v0.33.0
+	go.step.sm/linkedca v0.20.0 // indirect
+	go.uber.org/multierr v1.11.0 // indirect
+	golang.org/x/mod v0.11.0 // indirect
+	golang.org/x/sys v0.10.0
+	golang.org/x/text v0.11.0 // indirect
+	golang.org/x/tools v0.10.0 // indirect
+	google.golang.org/grpc v1.56.2 // indirect
+	google.golang.org/protobuf v1.31.0 // indirect
 	gopkg.in/square/go-jose.v2 v2.6.0 // indirect
 	howett.net/plist v1.0.0 // indirect
 )

internal/sockets.go

@@ -0,0 +1,56 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package internal
+
+import (
+	"fmt"
+	"io/fs"
+	"strconv"
+	"strings"
+)
+
+// SplitUnixSocketPermissionsBits takes a unix socket address in the
+// unusual "path|bits" format (e.g. /run/caddy.sock|0222) and tries
+// to split it into socket path (host) and permissions bits (port).
+// Colons (":") can't be used as separator, as socket paths on Windows
+// may include a drive letter (e.g. `unix/c:\absolute\path.sock`).
+// Permission bits will default to 0200 if none are specified.
+// Throws an error, if the first carrying bit does not
+// include write perms (e.g. `0422` or `022`).
+// Symbolic permission representation (e.g. `u=w,g=w,o=w`)
+// is not supported and will throw an error for now!
+func SplitUnixSocketPermissionsBits(addr string) (path string, fileMode fs.FileMode, err error) {
+	addrSplit := strings.SplitN(addr, "|", 2)
+
+	if len(addrSplit) == 2 {
+		// parse octal permission bit string as uint32
+		fileModeUInt64, err := strconv.ParseUint(addrSplit[1], 8, 32)
+		if err != nil {
+			return "", 0, fmt.Errorf("could not parse octal permission bits in %s: %v", addr, err)
+		}
+		fileMode = fs.FileMode(fileModeUInt64)
+
+		// FileMode.String() returns a string like `-rwxr-xr--` for `u=rwx,g=rx,o=r` (`0754`)
+		if string(fileMode.String()[2]) != "w" {
+			return "", 0, fmt.Errorf("owner of the socket requires '-w-' (write, octal: '2') permissions at least; got '%s' in %s", fileMode.String()[1:4], addr)
+		}
+
+		return addrSplit[0], fileMode, nil
+	}
+
+	// default to 0200 (symbolic: `u=w,g=,o=`)
+	// if no permission bits are specified
+	return addr, 0200, nil
+}

listen.go

@@ -12,10 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// TODO: Go 1.19 introduced the "unix" build tag. We have to support Go 1.18 until Go 1.20 is released.
-// When Go 1.19 is our minimum, change this build tag to simply "!unix".
-// (see similar change needed in listen_unix.go)
-//go:build !(aix || android || darwin || dragonfly || freebsd || hurd || illumos || ios || linux || netbsd || openbsd || solaris)
+//go:build !unix
 
 package caddy
 

listen_unix.go

@@ -12,10 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// TODO: Go 1.19 introduced the "unix" build tag. We have to support Go 1.18 until Go 1.20 is released.
-// When Go 1.19 is our minimum, remove this build tag, since "_unix" in the filename will do this.
-// (see also change needed in listen.go)
-//go:build aix || android || darwin || dragonfly || freebsd || hurd || illumos || ios || linux || netbsd || openbsd || solaris
+// Even though the filename ends in _unix.go, we still have to specify the
+// build constraint here, because the filename convention only works for
+// literal GOOS values, and "unix" is a shortcut unique to build tags.
+//go:build unix
 
 package caddy
 
@@ -98,7 +98,28 @@ func listenTCPOrUnix(ctx context.Context, lnKey string, network, address string,
 		}
 		return reusePort(network, address, c)
 	}
-	return config.Listen(ctx, network, address)
+
+	// even though SO_REUSEPORT lets us bind the socket multiple times,
+	// we still put it in the listenerPool so we can count how many
+	// configs are using this socket; necessary to ensure we can know
+	// whether to enforce shutdown delays, for example (see #5393).
+	ln, err := config.Listen(ctx, network, address)
+	if err == nil {
+		listenerPool.LoadOrStore(lnKey, nil)
+	}
+
+	// if new listener is a unix socket, make sure we can reuse it later
+	// (we do our own "unlink on close" -- not required, but more tidy)
+	one := int32(1)
+	if unix, ok := ln.(*net.UnixListener); ok {
+		unix.SetUnlinkOnClose(false)
+		ln = &unixListener{unix, lnKey, &one}
+		unixSockets[lnKey] = ln.(*unixListener)
+	}
+
+	// lightly wrap the listener so that when it is closed,
+	// we can decrement the usage pool counter
+	return deleteListener{ln, lnKey}, err
 }
 
 // reusePort sets SO_REUSEPORT. Ineffective for unix sockets.
@@ -107,7 +128,7 @@ func reusePort(network, address string, conn syscall.RawConn) error {
 		return nil
 	}
 	return conn.Control(func(descriptor uintptr) {
-		if err := unix.SetsockoptInt(int(descriptor), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1); err != nil {
+		if err := unix.SetsockoptInt(int(descriptor), unix.SOL_SOCKET, unixSOREUSEPORT, 1); err != nil {
 			Log().Error("setting SO_REUSEPORT",
 				zap.String("network", network),
 				zap.String("address", address),
@@ -116,3 +137,37 @@ func reusePort(network, address string, conn syscall.RawConn) error {
 		}
 	})
 }
+
+type unixListener struct {
+	*net.UnixListener
+	mapKey string
+	count  *int32 // accessed atomically
+}
+
+func (uln *unixListener) Close() error {
+	newCount := atomic.AddInt32(uln.count, -1)
+	if newCount == 0 {
+		defer func() {
+			addr := uln.Addr().String()
+			unixSocketsMu.Lock()
+			delete(unixSockets, uln.mapKey)
+			unixSocketsMu.Unlock()
+			_ = syscall.Unlink(addr)
+		}()
+	}
+	return uln.UnixListener.Close()
+}
+
+// deleteListener is a type that simply deletes itself
+// from the listenerPool when it closes. It is used
+// solely for the purpose of reference counting (i.e.
+// counting how many configs are using a given socket).
+type deleteListener struct {
+	net.Listener
+	lnKey string
+}
+
+func (dl deleteListener) Close() error {
+	_, _ = listenerPool.Delete(dl.lnKey)
+	return dl.Listener.Close()
+}

listen_unix_setopt.go

@@ -0,0 +1,7 @@
+//go:build unix && !freebsd
+
+package caddy
+
+import "golang.org/x/sys/unix"
+
+const unixSOREUSEPORT = unix.SO_REUSEPORT

listen_unix_setopt_freebsd.go

@@ -0,0 +1,7 @@
+//go:build freebsd
+
+package caddy
+
+import "golang.org/x/sys/unix"
+
+const unixSOREUSEPORT = unix.SO_REUSEPORT_LB

listeners.go

@@ -20,6 +20,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"io/fs"
 	"net"
 	"net/netip"
 	"os"
@@ -30,6 +31,7 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/caddyserver/caddy/v2/internal"
 	"github.com/quic-go/quic-go"
 	"github.com/quic-go/quic-go/http3"
 	"go.uber.org/zap"
@@ -148,11 +150,32 @@ func (na NetworkAddress) Listen(ctx context.Context, portOffset uint, config net
 func (na NetworkAddress) listen(ctx context.Context, portOffset uint, config net.ListenConfig) (any, error) {
 	var ln any
 	var err error
+	var address string
+	var unixFileMode fs.FileMode
+	var isAbtractUnixSocket bool
 
-	address := na.JoinHostPort(portOffset)
+	// split unix socket addr early so lnKey
+	// is independent of permissions bits
+	if na.IsUnixNetwork() {
+		var err error
+		address, unixFileMode, err = internal.SplitUnixSocketPermissionsBits(na.Host)
+		if err != nil {
+			return nil, err
+		}
+		isAbtractUnixSocket = strings.HasPrefix(address, "@")
+	} else {
+		address = na.JoinHostPort(portOffset)
+	}
 
-	// if this is a unix socket, see if we already have it open
+	// if this is a unix socket, see if we already have it open,
+	// force socket permissions on it and return early
 	if socket, err := reuseUnixSocket(na.Network, address); socket != nil || err != nil {
+		if !isAbtractUnixSocket {
+			if err := os.Chmod(address, unixFileMode); err != nil {
+				return nil, fmt.Errorf("unable to set permissions (%s) on %s: %v", unixFileMode, address, err)
+			}
+
+		}
 		return socket, err
 	}
 
@@ -174,7 +197,8 @@ func (na NetworkAddress) listen(ctx context.Context, portOffset uint, config net
 		if err != nil {
 			return nil, err
 		}
-		ln = &fakeClosePacketConn{sharedPacketConn: sharedPc.(*sharedPacketConn)}
+		spc := sharedPc.(*sharedPacketConn)
+		ln = &fakeClosePacketConn{spc: spc, UDPConn: spc.PacketConn.(*net.UDPConn)}
 	}
 	if strings.HasPrefix(na.Network, "ip") {
 		ln, err = config.ListenPacket(ctx, na.Network, address)
@@ -186,17 +210,19 @@ func (na NetworkAddress) listen(ctx context.Context, portOffset uint, config net
 		return nil, fmt.Errorf("unsupported network type: %s", na.Network)
 	}
 
-	// if new listener is a unix socket, make sure we can reuse it later
-	// (we do our own "unlink on close" -- not required, but more tidy)
-	one := int32(1)
-	switch unix := ln.(type) {
-	case *net.UnixListener:
-		unix.SetUnlinkOnClose(false)
-		ln = &unixListener{unix, lnKey, &one}
-		unixSockets[lnKey] = ln.(*unixListener)
-	case *net.UnixConn:
+	// TODO: Not 100% sure this is necessary, but we do this for net.UnixListener in listen_unix.go, so...
+	if unix, ok := ln.(*net.UnixConn); ok {
+		one := int32(1)
 		ln = &unixConn{unix, address, lnKey, &one}
-		unixSockets[lnKey] = ln.(*unixConn)
+		unixSockets[lnKey] = unix
+	}
+
+	if IsUnixNetwork(na.Network) {
+		if !isAbtractUnixSocket {
+			if err := os.Chmod(address, unixFileMode); err != nil {
+				return nil, fmt.Errorf("unable to set permissions (%s) on %s: %v", unixFileMode, address, err)
+			}
+		}
 	}
 
 	return ln, nil
@@ -303,22 +329,32 @@ func IsUnixNetwork(netw string) bool {
 // Network addresses are distinct from URLs and do not
 // use URL syntax.
 func ParseNetworkAddress(addr string) (NetworkAddress, error) {
+	return ParseNetworkAddressWithDefaults(addr, "tcp", 0)
+}
+
+// ParseNetworkAddressWithDefaults is like ParseNetworkAddress but allows
+// the default network and port to be specified.
+func ParseNetworkAddressWithDefaults(addr, defaultNetwork string, defaultPort uint) (NetworkAddress, error) {
 	var host, port string
 	network, host, port, err := SplitNetworkAddress(addr)
 	if err != nil {
 		return NetworkAddress{}, err
 	}
 	if network == "" {
-		network = "tcp"
+		network = defaultNetwork
 	}
 	if IsUnixNetwork(network) {
+		_, _, err := internal.SplitUnixSocketPermissionsBits(host)
 		return NetworkAddress{
 			Network: network,
 			Host:    host,
-		}, nil
+		}, err
 	}
 	var start, end uint64
-	if port != "" {
+	if port == "" {
+		start = uint64(defaultPort)
+		end = uint64(defaultPort)
+	} else {
 		before, after, found := strings.Cut(port, "-")
 		if !found {
 			after = before
@@ -436,11 +472,16 @@ func ListenPacket(network, addr string) (net.PacketConn, error) {
 // NOTE: This API is EXPERIMENTAL and may be changed or removed.
 //
 // TODO: See if we can find a more elegant solution closer to the new NetworkAddress.Listen API.
-func ListenQUIC(ln net.PacketConn, tlsConf *tls.Config, activeRequests *int64) (quic.EarlyListener, error) {
+func ListenQUIC(ln net.PacketConn, tlsConf *tls.Config, activeRequests *int64) (http3.QUICEarlyListener, error) {
 	lnKey := listenerKey("quic+"+ln.LocalAddr().Network(), ln.LocalAddr().String())
 
 	sharedEarlyListener, _, err := listenerPool.LoadOrNew(lnKey, func() (Destructor, error) {
-		earlyLn, err := quic.ListenEarly(ln, http3.ConfigureTLSConfig(tlsConf), &quic.Config{
+		sqtc := newSharedQUICTLSConfig(tlsConf)
+		// http3.ConfigureTLSConfig only uses this field and tls App sets this field as well
+		//nolint:gosec
+		quicTlsConfig := &tls.Config{GetConfigForClient: sqtc.getConfigForClient}
+		earlyLn, err := quic.ListenEarly(ln, http3.ConfigureTLSConfig(quicTlsConfig), &quic.Config{
+			Allow0RTT: true,
 			RequireAddressValidation: func(clientAddr net.Addr) bool {
 				var highLoad bool
 				if activeRequests != nil {
@@ -452,12 +493,16 @@ func ListenQUIC(ln net.PacketConn, tlsConf *tls.Config, activeRequests *int64) (
 		if err != nil {
 			return nil, err
 		}
-		return &sharedQuicListener{EarlyListener: earlyLn, key: lnKey}, nil
+		return &sharedQuicListener{EarlyListener: earlyLn, sqtc: sqtc, key: lnKey}, nil
 	})
 	if err != nil {
 		return nil, err
 	}
 
+	sql := sharedEarlyListener.(*sharedQuicListener)
+	// add current tls.Config to sqtc, so GetConfigForClient will always return the latest tls.Config in case of context cancellation
+	ctx, cancel := sql.sqtc.addTLSConfig(tlsConf)
+
 	// TODO: to serve QUIC over a unix socket, currently we need to hold onto
 	// the underlying net.PacketConn (which we wrap as unixConn to keep count
 	// of closes) because closing the quic.EarlyListener doesn't actually close
@@ -469,9 +514,8 @@ func ListenQUIC(ln net.PacketConn, tlsConf *tls.Config, activeRequests *int64) (
 		unix = uc
 	}
 
-	ctx, cancel := context.WithCancel(context.Background())
 	return &fakeCloseQuicListener{
-		sharedQuicListener: sharedEarlyListener.(*sharedQuicListener),
+		sharedQuicListener: sql,
 		uc:                 unix,
 		context:            ctx,
 		contextCancel:      cancel,
@@ -484,10 +528,77 @@ func ListenerUsage(network, addr string) int {
 	return count
 }
 
+// contextAndCancelFunc groups context and its cancelFunc
+type contextAndCancelFunc struct {
+	context.Context
+	context.CancelFunc
+}
+
+// sharedQUICTLSConfig manages GetConfigForClient
+// see issue: https://github.com/caddyserver/caddy/pull/4849
+type sharedQUICTLSConfig struct {
+	rmu           sync.RWMutex
+	tlsConfs      map[*tls.Config]contextAndCancelFunc
+	activeTlsConf *tls.Config
+}
+
+// newSharedQUICTLSConfig creates a new sharedQUICTLSConfig
+func newSharedQUICTLSConfig(tlsConfig *tls.Config) *sharedQUICTLSConfig {
+	sqtc := &sharedQUICTLSConfig{
+		tlsConfs:      make(map[*tls.Config]contextAndCancelFunc),
+		activeTlsConf: tlsConfig,
+	}
+	sqtc.addTLSConfig(tlsConfig)
+	return sqtc
+}
+
+// getConfigForClient is used as tls.Config's GetConfigForClient field
+func (sqtc *sharedQUICTLSConfig) getConfigForClient(ch *tls.ClientHelloInfo) (*tls.Config, error) {
+	sqtc.rmu.RLock()
+	defer sqtc.rmu.RUnlock()
+	return sqtc.activeTlsConf.GetConfigForClient(ch)
+}
+
+// addTLSConfig adds tls.Config to the map if not present and returns the corresponding context and its cancelFunc
+// so that when cancelled, the active tls.Config will change
+func (sqtc *sharedQUICTLSConfig) addTLSConfig(tlsConfig *tls.Config) (context.Context, context.CancelFunc) {
+	sqtc.rmu.Lock()
+	defer sqtc.rmu.Unlock()
+
+	if cacc, ok := sqtc.tlsConfs[tlsConfig]; ok {
+		return cacc.Context, cacc.CancelFunc
+	}
+
+	ctx, cancel := context.WithCancel(context.Background())
+	wrappedCancel := func() {
+		cancel()
+
+		sqtc.rmu.Lock()
+		defer sqtc.rmu.Unlock()
+
+		delete(sqtc.tlsConfs, tlsConfig)
+		if sqtc.activeTlsConf == tlsConfig {
+			// select another tls.Config, if there is none,
+			// related sharedQuicListener will be destroyed anyway
+			for tc := range sqtc.tlsConfs {
+				sqtc.activeTlsConf = tc
+				break
+			}
+		}
+	}
+	sqtc.tlsConfs[tlsConfig] = contextAndCancelFunc{ctx, wrappedCancel}
+	// there should be at most 2 tls.Configs
+	if len(sqtc.tlsConfs) > 2 {
+		Log().Warn("quic listener tls configs are more than 2", zap.Int("number of configs", len(sqtc.tlsConfs)))
+	}
+	return ctx, wrappedCancel
+}
+
 // sharedQuicListener is like sharedListener, but for quic.EarlyListeners.
 type sharedQuicListener struct {
-	quic.EarlyListener
-	key string
+	*quic.EarlyListener
+	sqtc *sharedQUICTLSConfig
+	key  string
 }
 
 // Destruct closes the underlying QUIC listener.
@@ -525,37 +636,30 @@ func fakeClosedErr(l interface{ Addr() net.Addr }) error {
 // socket is actually left open.
 var errFakeClosed = fmt.Errorf("listener 'closed' 😉")
 
-// fakeClosePacketConn is like fakeCloseListener, but for PacketConns.
+// fakeClosePacketConn is like fakeCloseListener, but for PacketConns,
+// or more specifically, *net.UDPConn
 type fakeClosePacketConn struct {
-	closed            int32 // accessed atomically; belongs to this struct only
-	*sharedPacketConn       // embedded, so we also become a net.PacketConn
+	closed       int32             // accessed atomically; belongs to this struct only
+	spc          *sharedPacketConn // its key is used in Close
+	*net.UDPConn                   // embedded, so we also become a net.PacketConn and enable several other optimizations done by quic-go
 }
 
+// interface guard for extra optimizations
+// needed by QUIC implementation: https://github.com/caddyserver/caddy/issues/3998, https://github.com/caddyserver/caddy/issues/5605
+var _ quic.OOBCapablePacketConn = (*fakeClosePacketConn)(nil)
+
+// https://pkg.go.dev/golang.org/x/net/ipv4#NewPacketConn is used by quic-go and requires a net.PacketConn type assertable to a net.Conn,
+// but doesn't actually use these methods, the only methods needed are `ReadMsgUDP` and `SyscallConn`.
+var _ net.Conn = (*fakeClosePacketConn)(nil)
+
+// Close won't close the underlying socket unless there is no more reference, then listenerPool will close it.
 func (fcpc *fakeClosePacketConn) Close() error {
 	if atomic.CompareAndSwapInt32(&fcpc.closed, 0, 1) {
-		_, _ = listenerPool.Delete(fcpc.sharedPacketConn.key)
+		_, _ = listenerPool.Delete(fcpc.spc.key)
 	}
 	return nil
 }
 
-// Supports QUIC implementation: https://github.com/caddyserver/caddy/issues/3998
-func (fcpc fakeClosePacketConn) SetReadBuffer(bytes int) error {
-	if conn, ok := fcpc.PacketConn.(interface{ SetReadBuffer(int) error }); ok {
-		return conn.SetReadBuffer(bytes)
-	}
-	return fmt.Errorf("SetReadBuffer() not implemented for %T", fcpc.PacketConn)
-}
-
-// Supports QUIC implementation: https://github.com/caddyserver/caddy/issues/3998
-func (fcpc fakeClosePacketConn) SyscallConn() (syscall.RawConn, error) {
-	if conn, ok := fcpc.PacketConn.(interface {
-		SyscallConn() (syscall.RawConn, error)
-	}); ok {
-		return conn.SyscallConn()
-	}
-	return nil, fmt.Errorf("SyscallConn() not implemented for %T", fcpc.PacketConn)
-}
-
 type fakeCloseQuicListener struct {
 	closed              int32     // accessed atomically; belongs to this struct only
 	*sharedQuicListener           // embedded, so we also become a quic.EarlyListener
@@ -616,26 +720,6 @@ func RegisterNetwork(network string, getListener ListenerFunc) {
 	networkTypes[network] = getListener
 }
 
-type unixListener struct {
-	*net.UnixListener
-	mapKey string
-	count  *int32 // accessed atomically
-}
-
-func (uln *unixListener) Close() error {
-	newCount := atomic.AddInt32(uln.count, -1)
-	if newCount == 0 {
-		defer func() {
-			addr := uln.Addr().String()
-			unixSocketsMu.Lock()
-			delete(unixSockets, uln.mapKey)
-			unixSocketsMu.Unlock()
-			_ = syscall.Unlink(addr)
-		}()
-	}
-	return uln.UnixListener.Close()
-}
-
 type unixConn struct {
 	*net.UnixConn
 	filename string

listeners_test.go

@@ -17,6 +17,8 @@ package caddy
 import (
 	"reflect"
 	"testing"
+
+	"github.com/caddyserver/caddy/v2/internal"
 )
 
 func TestSplitNetworkAddress(t *testing.T) {
@@ -175,47 +177,57 @@ func TestJoinNetworkAddress(t *testing.T) {
 
 func TestParseNetworkAddress(t *testing.T) {
 	for i, tc := range []struct {
-		input      string
-		expectAddr NetworkAddress
-		expectErr  bool
+		input          string
+		defaultNetwork string
+		defaultPort    uint
+		expectAddr     NetworkAddress
+		expectErr      bool
 	}{
 		{
 			input:     "",
 			expectErr: true,
 		},
 		{
-			input: ":",
+			input:          ":",
+			defaultNetwork: "udp",
 			expectAddr: NetworkAddress{
-				Network: "tcp",
+				Network: "udp",
 			},
 		},
 		{
-			input: "[::]",
+			input:          "[::]",
+			defaultNetwork: "udp",
+			defaultPort:    53,
 			expectAddr: NetworkAddress{
-				Network: "tcp",
-				Host:    "::",
+				Network:   "udp",
+				Host:      "::",
+				StartPort: 53,
+				EndPort:   53,
 			},
 		},
 		{
-			input: ":1234",
+			input:          ":1234",
+			defaultNetwork: "udp",
 			expectAddr: NetworkAddress{
-				Network:   "tcp",
+				Network:   "udp",
 				Host:      "",
 				StartPort: 1234,
 				EndPort:   1234,
 			},
 		},
 		{
-			input: "tcp/:1234",
+			input:          "udp/:1234",
+			defaultNetwork: "udp",
 			expectAddr: NetworkAddress{
-				Network:   "tcp",
+				Network:   "udp",
 				Host:      "",
 				StartPort: 1234,
 				EndPort:   1234,
 			},
 		},
 		{
-			input: "tcp6/:1234",
+			input:          "tcp6/:1234",
+			defaultNetwork: "tcp",
 			expectAddr: NetworkAddress{
 				Network:   "tcp6",
 				Host:      "",
@@ -224,7 +236,8 @@ func TestParseNetworkAddress(t *testing.T) {
 			},
 		},
 		{
-			input: "tcp4/localhost:1234",
+			input:          "tcp4/localhost:1234",
+			defaultNetwork: "tcp",
 			expectAddr: NetworkAddress{
 				Network:   "tcp4",
 				Host:      "localhost",
@@ -233,14 +246,16 @@ func TestParseNetworkAddress(t *testing.T) {
 			},
 		},
 		{
-			input: "unix//foo/bar",
+			input:          "unix//foo/bar",
+			defaultNetwork: "tcp",
 			expectAddr: NetworkAddress{
 				Network: "unix",
 				Host:    "/foo/bar",
 			},
 		},
 		{
-			input: "localhost:1234-1234",
+			input:          "localhost:1234-1234",
+			defaultNetwork: "tcp",
 			expectAddr: NetworkAddress{
 				Network:   "tcp",
 				Host:      "localhost",
@@ -249,11 +264,13 @@ func TestParseNetworkAddress(t *testing.T) {
 			},
 		},
 		{
-			input:     "localhost:2-1",
-			expectErr: true,
+			input:          "localhost:2-1",
+			defaultNetwork: "tcp",
+			expectErr:      true,
 		},
 		{
-			input: "localhost:0",
+			input:          "localhost:0",
+			defaultNetwork: "tcp",
 			expectAddr: NetworkAddress{
 				Network:   "tcp",
 				Host:      "localhost",
@@ -262,11 +279,138 @@ func TestParseNetworkAddress(t *testing.T) {
 			},
 		},
 		{
-			input:     "localhost:1-999999999999",
-			expectErr: true,
+			input:          "localhost:1-999999999999",
+			defaultNetwork: "tcp",
+			expectErr:      true,
 		},
 	} {
-		actualAddr, err := ParseNetworkAddress(tc.input)
+		actualAddr, err := ParseNetworkAddressWithDefaults(tc.input, tc.defaultNetwork, tc.defaultPort)
+		if tc.expectErr && err == nil {
+			t.Errorf("Test %d: Expected error but got: %v", i, err)
+		}
+		if !tc.expectErr && err != nil {
+			t.Errorf("Test %d: Expected no error but got: %v", i, err)
+		}
+
+		if actualAddr.Network != tc.expectAddr.Network {
+			t.Errorf("Test %d: Expected network '%v' but got '%v'", i, tc.expectAddr, actualAddr)
+		}
+		if !reflect.DeepEqual(tc.expectAddr, actualAddr) {
+			t.Errorf("Test %d: Expected addresses %v but got %v", i, tc.expectAddr, actualAddr)
+		}
+	}
+}
+
+func TestParseNetworkAddressWithDefaults(t *testing.T) {
+	for i, tc := range []struct {
+		input          string
+		defaultNetwork string
+		defaultPort    uint
+		expectAddr     NetworkAddress
+		expectErr      bool
+	}{
+		{
+			input:     "",
+			expectErr: true,
+		},
+		{
+			input:          ":",
+			defaultNetwork: "udp",
+			expectAddr: NetworkAddress{
+				Network: "udp",
+			},
+		},
+		{
+			input:          "[::]",
+			defaultNetwork: "udp",
+			defaultPort:    53,
+			expectAddr: NetworkAddress{
+				Network:   "udp",
+				Host:      "::",
+				StartPort: 53,
+				EndPort:   53,
+			},
+		},
+		{
+			input:          ":1234",
+			defaultNetwork: "udp",
+			expectAddr: NetworkAddress{
+				Network:   "udp",
+				Host:      "",
+				StartPort: 1234,
+				EndPort:   1234,
+			},
+		},
+		{
+			input:          "udp/:1234",
+			defaultNetwork: "udp",
+			expectAddr: NetworkAddress{
+				Network:   "udp",
+				Host:      "",
+				StartPort: 1234,
+				EndPort:   1234,
+			},
+		},
+		{
+			input:          "tcp6/:1234",
+			defaultNetwork: "tcp",
+			expectAddr: NetworkAddress{
+				Network:   "tcp6",
+				Host:      "",
+				StartPort: 1234,
+				EndPort:   1234,
+			},
+		},
+		{
+			input:          "tcp4/localhost:1234",
+			defaultNetwork: "tcp",
+			expectAddr: NetworkAddress{
+				Network:   "tcp4",
+				Host:      "localhost",
+				StartPort: 1234,
+				EndPort:   1234,
+			},
+		},
+		{
+			input:          "unix//foo/bar",
+			defaultNetwork: "tcp",
+			expectAddr: NetworkAddress{
+				Network: "unix",
+				Host:    "/foo/bar",
+			},
+		},
+		{
+			input:          "localhost:1234-1234",
+			defaultNetwork: "tcp",
+			expectAddr: NetworkAddress{
+				Network:   "tcp",
+				Host:      "localhost",
+				StartPort: 1234,
+				EndPort:   1234,
+			},
+		},
+		{
+			input:          "localhost:2-1",
+			defaultNetwork: "tcp",
+			expectErr:      true,
+		},
+		{
+			input:          "localhost:0",
+			defaultNetwork: "tcp",
+			expectAddr: NetworkAddress{
+				Network:   "tcp",
+				Host:      "localhost",
+				StartPort: 0,
+				EndPort:   0,
+			},
+		},
+		{
+			input:          "localhost:1-999999999999",
+			defaultNetwork: "tcp",
+			expectErr:      true,
+		},
+	} {
+		actualAddr, err := ParseNetworkAddressWithDefaults(tc.input, tc.defaultNetwork, tc.defaultPort)
 		if tc.expectErr && err == nil {
 			t.Errorf("Test %d: Expected error but got: %v", i, err)
 		}
@@ -413,3 +557,98 @@ func TestExpand(t *testing.T) {
 		}
 	}
 }
+
+func TestSplitUnixSocketPermissionsBits(t *testing.T) {
+	for i, tc := range []struct {
+		input          string
+		expectNetwork  string
+		expectPath     string
+		expectFileMode string
+		expectErr      bool
+	}{
+		{
+			input:          "./foo.socket",
+			expectPath:     "./foo.socket",
+			expectFileMode: "--w-------",
+		},
+		{
+			input:          `.\relative\path.socket`,
+			expectPath:     `.\relative\path.socket`,
+			expectFileMode: "--w-------",
+		},
+		{
+			// literal colon in resulting address
+			// and defaulting to 0200 bits
+			input:          "./foo.socket:0666",
+			expectPath:     "./foo.socket:0666",
+			expectFileMode: "--w-------",
+		},
+		{
+			input:          "./foo.socket|0220",
+			expectPath:     "./foo.socket",
+			expectFileMode: "--w--w----",
+		},
+		{
+			input:          "/var/run/foo|222",
+			expectPath:     "/var/run/foo",
+			expectFileMode: "--w--w--w-",
+		},
+		{
+			input:          "./foo.socket|0660",
+			expectPath:     "./foo.socket",
+			expectFileMode: "-rw-rw----",
+		},
+		{
+			input:          "./foo.socket|0666",
+			expectPath:     "./foo.socket",
+			expectFileMode: "-rw-rw-rw-",
+		},
+		{
+			input:          "/var/run/foo|666",
+			expectPath:     "/var/run/foo",
+			expectFileMode: "-rw-rw-rw-",
+		},
+		{
+			input:          `c:\absolute\path.socket|220`,
+			expectPath:     `c:\absolute\path.socket`,
+			expectFileMode: "--w--w----",
+		},
+		{
+			// symbolic permission representation is not supported for now
+			input:     "./foo.socket|u=rw,g=rw,o=rw",
+			expectErr: true,
+		},
+		{
+			// octal (base-8) permission representation has to be between
+			// `0` for no read, no write, no exec (`---`) and
+			// `7` for read (4), write (2), exec (1) (`rwx` => `4+2+1 = 7`)
+			input:     "./foo.socket|888",
+			expectErr: true,
+		},
+		{
+			// too many colons in address
+			input:     "./foo.socket|123456|0660",
+			expectErr: true,
+		},
+		{
+			// owner is missing write perms
+			input:     "./foo.socket|0522",
+			expectErr: true,
+		},
+	} {
+		actualPath, actualFileMode, err := internal.SplitUnixSocketPermissionsBits(tc.input)
+		if tc.expectErr && err == nil {
+			t.Errorf("Test %d: Expected error but got: %v", i, err)
+		}
+		if !tc.expectErr && err != nil {
+			t.Errorf("Test %d: Expected no error but got: %v", i, err)
+		}
+		if actualPath != tc.expectPath {
+			t.Errorf("Test %d: Expected path '%s' but got '%s'", i, tc.expectPath, actualPath)
+		}
+		// fileMode.Perm().String() parses 0 to "----------"
+		if !tc.expectErr && actualFileMode.Perm().String() != tc.expectFileMode {
+			t.Errorf("Test %d: Expected perms '%s' but got '%s'", i, tc.expectFileMode, actualFileMode.Perm().String())
+		}
+	}
+}

logging.go

@@ -62,7 +62,7 @@ type Logging struct {
 	// in dependencies that are not designed specifically for use
 	// in Caddy. Because it is global and unstructured, the sink
 	// lacks most advanced features and customizations.
-	Sink *StandardLibLog `json:"sink,omitempty"`
+	Sink *SinkLog `json:"sink,omitempty"`
 
 	// Logs are your logs, keyed by an arbitrary name of your
 	// choosing. The default log can be customized by defining
@@ -259,55 +259,11 @@ func (wdest writerDestructor) Destruct() error {
 	return wdest.Close()
 }
 
-// StandardLibLog configures the default Go standard library
-// global logger in the log package. This is necessary because
-// module dependencies which are not built specifically for
-// Caddy will use the standard logger. This is also known as
-// the "sink" logger.
-type StandardLibLog struct {
+// BaseLog contains the common logging parameters for logging.
+type BaseLog struct {
 	// The module that writes out log entries for the sink.
 	WriterRaw json.RawMessage `json:"writer,omitempty" caddy:"namespace=caddy.logging.writers inline_key=output"`
 
-	writer io.WriteCloser
-}
-
-func (sll *StandardLibLog) provision(ctx Context, logging *Logging) error {
-	if sll.WriterRaw != nil {
-		mod, err := ctx.LoadModule(sll, "WriterRaw")
-		if err != nil {
-			return fmt.Errorf("loading sink log writer module: %v", err)
-		}
-		wo := mod.(WriterOpener)
-
-		var isNew bool
-		sll.writer, isNew, err = logging.openWriter(wo)
-		if err != nil {
-			return fmt.Errorf("opening sink log writer %#v: %v", mod, err)
-		}
-
-		if isNew {
-			log.Printf("[INFO] Redirecting sink to: %s", wo)
-			log.SetOutput(sll.writer)
-			log.Printf("[INFO] Redirected sink to here (%s)", wo)
-		}
-	}
-
-	return nil
-}
-
-// CustomLog represents a custom logger configuration.
-//
-// By default, a log will emit all log entries. Some entries
-// will be skipped if sampling is enabled. Further, the Include
-// and Exclude parameters define which loggers (by name) are
-// allowed or rejected from emitting in this log. If both Include
-// and Exclude are populated, their values must be mutually
-// exclusive, and longer namespaces have priority. If neither
-// are populated, all logs are emitted.
-type CustomLog struct {
-	// The writer defines where log entries are emitted.
-	WriterRaw json.RawMessage `json:"writer,omitempty" caddy:"namespace=caddy.logging.writers inline_key=output"`
-
 	// The encoder is how the log entries are formatted or encoded.
 	EncoderRaw json.RawMessage `json:"encoder,omitempty" caddy:"namespace=caddy.logging.encoders inline_key=format"`
 
@@ -321,16 +277,6 @@ type CustomLog struct {
 	// servers.
 	Sampling *LogSampling `json:"sampling,omitempty"`
 
-	// Include defines the names of loggers to emit in this
-	// log. For example, to include only logs emitted by the
-	// admin API, you would include "admin.api".
-	Include []string `json:"include,omitempty"`
-
-	// Exclude defines the names of loggers that should be
-	// skipped by this log. For example, to exclude only
-	// HTTP access logs, you would exclude "http.log.access".
-	Exclude []string `json:"exclude,omitempty"`
-
 	writerOpener WriterOpener
 	writer       io.WriteCloser
 	encoder      zapcore.Encoder
@@ -338,8 +284,23 @@ type CustomLog struct {
 	core         zapcore.Core
 }
 
-func (cl *CustomLog) provision(ctx Context, logging *Logging) error {
-	// Replace placeholder for log level
+func (cl *BaseLog) provisionCommon(ctx Context, logging *Logging) error {
+	if cl.WriterRaw != nil {
+		mod, err := ctx.LoadModule(cl, "WriterRaw")
+		if err != nil {
+			return fmt.Errorf("loading log writer module: %v", err)
+		}
+		cl.writerOpener = mod.(WriterOpener)
+	}
+	if cl.writerOpener == nil {
+		cl.writerOpener = StderrWriter{}
+	}
+	var err error
+	cl.writer, _, err = logging.openWriter(cl.writerOpener)
+	if err != nil {
+		return fmt.Errorf("opening log writer using %#v: %v", cl.writerOpener, err)
+	}
+
 	repl := NewReplacer()
 	level, err := repl.ReplaceOrErr(cl.Level, true, true)
 	if err != nil {
@@ -365,6 +326,101 @@ func (cl *CustomLog) provision(ctx Context, logging *Logging) error {
 		return fmt.Errorf("unrecognized log level: %s", cl.Level)
 	}
 
+	if cl.EncoderRaw != nil {
+		mod, err := ctx.LoadModule(cl, "EncoderRaw")
+		if err != nil {
+			return fmt.Errorf("loading log encoder module: %v", err)
+		}
+		cl.encoder = mod.(zapcore.Encoder)
+	}
+	if cl.encoder == nil {
+		// only allow colorized output if this log is going to stdout or stderr
+		var colorize bool
+		switch cl.writerOpener.(type) {
+		case StdoutWriter, StderrWriter,
+			*StdoutWriter, *StderrWriter:
+			colorize = true
+		}
+		cl.encoder = newDefaultProductionLogEncoder(colorize)
+	}
+	cl.buildCore()
+	return nil
+}
+
+func (cl *BaseLog) buildCore() {
+	// logs which only discard their output don't need
+	// to perform encoding or any other processing steps
+	// at all, so just shorcut to a nop core instead
+	if _, ok := cl.writerOpener.(*DiscardWriter); ok {
+		cl.core = zapcore.NewNopCore()
+		return
+	}
+	c := zapcore.NewCore(
+		cl.encoder,
+		zapcore.AddSync(cl.writer),
+		cl.levelEnabler,
+	)
+	if cl.Sampling != nil {
+		if cl.Sampling.Interval == 0 {
+			cl.Sampling.Interval = 1 * time.Second
+		}
+		if cl.Sampling.First == 0 {
+			cl.Sampling.First = 100
+		}
+		if cl.Sampling.Thereafter == 0 {
+			cl.Sampling.Thereafter = 100
+		}
+		c = zapcore.NewSamplerWithOptions(c, cl.Sampling.Interval,
+			cl.Sampling.First, cl.Sampling.Thereafter)
+	}
+	cl.core = c
+}
+
+// SinkLog configures the default Go standard library
+// global logger in the log package. This is necessary because
+// module dependencies which are not built specifically for
+// Caddy will use the standard logger. This is also known as
+// the "sink" logger.
+type SinkLog struct {
+	BaseLog
+}
+
+func (sll *SinkLog) provision(ctx Context, logging *Logging) error {
+	if err := sll.provisionCommon(ctx, logging); err != nil {
+		return err
+	}
+	ctx.cleanupFuncs = append(ctx.cleanupFuncs, zap.RedirectStdLog(zap.New(sll.core)))
+	return nil
+}
+
+// CustomLog represents a custom logger configuration.
+//
+// By default, a log will emit all log entries. Some entries
+// will be skipped if sampling is enabled. Further, the Include
+// and Exclude parameters define which loggers (by name) are
+// allowed or rejected from emitting in this log. If both Include
+// and Exclude are populated, their values must be mutually
+// exclusive, and longer namespaces have priority. If neither
+// are populated, all logs are emitted.
+type CustomLog struct {
+	BaseLog
+
+	// Include defines the names of loggers to emit in this
+	// log. For example, to include only logs emitted by the
+	// admin API, you would include "admin.api".
+	Include []string `json:"include,omitempty"`
+
+	// Exclude defines the names of loggers that should be
+	// skipped by this log. For example, to exclude only
+	// HTTP access logs, you would exclude "http.log.access".
+	Exclude []string `json:"exclude,omitempty"`
+}
+
+func (cl *CustomLog) provision(ctx Context, logging *Logging) error {
+	if err := cl.provisionCommon(ctx, logging); err != nil {
+		return err
+	}
+
 	// If both Include and Exclude lists are populated, then each item must
 	// be a superspace or subspace of an item in the other list, because
 	// populating both lists means that any given item is either a rule
@@ -394,75 +450,9 @@ func (cl *CustomLog) provision(ctx Context, logging *Logging) error {
 			return fmt.Errorf("when both include and exclude are populated, each element must be a superspace or subspace of one in the other list; check '%s' in include", allow)
 		}
 	}
-
-	if cl.WriterRaw != nil {
-		mod, err := ctx.LoadModule(cl, "WriterRaw")
-		if err != nil {
-			return fmt.Errorf("loading log writer module: %v", err)
-		}
-		cl.writerOpener = mod.(WriterOpener)
-	}
-	if cl.writerOpener == nil {
-		cl.writerOpener = StderrWriter{}
-	}
-
-	cl.writer, _, err = logging.openWriter(cl.writerOpener)
-	if err != nil {
-		return fmt.Errorf("opening log writer using %#v: %v", cl.writerOpener, err)
-	}
-
-	if cl.EncoderRaw != nil {
-		mod, err := ctx.LoadModule(cl, "EncoderRaw")
-		if err != nil {
-			return fmt.Errorf("loading log encoder module: %v", err)
-		}
-		cl.encoder = mod.(zapcore.Encoder)
-	}
-	if cl.encoder == nil {
-		// only allow colorized output if this log is going to stdout or stderr
-		var colorize bool
-		switch cl.writerOpener.(type) {
-		case StdoutWriter, StderrWriter,
-			*StdoutWriter, *StderrWriter:
-			colorize = true
-		}
-		cl.encoder = newDefaultProductionLogEncoder(colorize)
-	}
-
-	cl.buildCore()
-
 	return nil
 }
 
-func (cl *CustomLog) buildCore() {
-	// logs which only discard their output don't need
-	// to perform encoding or any other processing steps
-	// at all, so just shorcut to a nop core instead
-	if _, ok := cl.writerOpener.(*DiscardWriter); ok {
-		cl.core = zapcore.NewNopCore()
-		return
-	}
-	c := zapcore.NewCore(
-		cl.encoder,
-		zapcore.AddSync(cl.writer),
-		cl.levelEnabler,
-	)
-	if cl.Sampling != nil {
-		if cl.Sampling.Interval == 0 {
-			cl.Sampling.Interval = 1 * time.Second
-		}
-		if cl.Sampling.First == 0 {
-			cl.Sampling.First = 100
-		}
-		if cl.Sampling.Thereafter == 0 {
-			cl.Sampling.Thereafter = 100
-		}
-		c = zapcore.NewSamplerWithOptions(c, cl.Sampling.Interval,
-			cl.Sampling.First, cl.Sampling.Thereafter)
-	}
-	cl.core = c
-}
-
 func (cl *CustomLog) matchesModule(moduleID string) bool {
 	return cl.loggerAllowed(moduleID, true)
 }

metrics.go

@@ -66,3 +66,9 @@ func (d *delegator) WriteHeader(code int) {
 	d.status = code
 	d.ResponseWriter.WriteHeader(code)
 }
+
+// Unwrap returns the underlying ResponseWriter, necessary for
+// http.ResponseController to work correctly.
+func (d *delegator) Unwrap() http.ResponseWriter {
+	return d.ResponseWriter
+}

modules.go

@@ -333,11 +333,11 @@ func ParseStructTag(tag string) (map[string]string, error) {
 	return results, nil
 }
 
-// strictUnmarshalJSON is like json.Unmarshal but returns an error
+// StrictUnmarshalJSON is like json.Unmarshal but returns an error
 // if any of the fields are unrecognized. Useful when decoding
 // module configurations, where you want to be more sure they're
 // correct.
-func strictUnmarshalJSON(data []byte, v any) error {
+func StrictUnmarshalJSON(data []byte, v any) error {
 	dec := json.NewDecoder(bytes.NewReader(data))
 	dec.DisallowUnknownFields()
 	return dec.Decode(v)

modules/caddyhttp/app.go

@@ -20,7 +20,9 @@ import (
 	"fmt"
 	"net"
 	"net/http"
+	"runtime"
 	"strconv"
+	"strings"
 	"sync"
 	"time"
 
@@ -232,6 +234,11 @@ func (app *App) Provision(ctx caddy.Context) error {
 			srv.trustedProxies = val.(IPRangeSource)
 		}
 
+		// set the default client IP header to read from
+		if srv.ClientIPHeaders == nil {
+			srv.ClientIPHeaders = []string{"X-Forwarded-For"}
+		}
+
 		// process each listener address
 		for i := range srv.Listen {
 			lnOut, err := repl.ReplaceOrErr(srv.Listen[i], true, true)
@@ -288,11 +295,19 @@ func (app *App) Provision(ctx caddy.Context) error {
 		if srv.Errors != nil {
 			err := srv.Errors.Routes.Provision(ctx)
 			if err != nil {
-				return fmt.Errorf("server %s: setting up server error handling routes: %v", srvName, err)
+				return fmt.Errorf("server %s: setting up error handling routes: %v", srvName, err)
 			}
 			srv.errorHandlerChain = srv.Errors.Routes.Compile(errorEmptyHandler)
 		}
 
+		// provision the named routes (they get compiled at runtime)
+		for name, route := range srv.NamedRoutes {
+			err := route.Provision(ctx, srv.Metrics)
+			if err != nil {
+				return fmt.Errorf("server %s: setting up named route '%s' handlers: %v", name, srvName, err)
+			}
+		}
+
 		// prepare the TLS connection policies
 		err = srv.TLSConnPolicies.Provision(ctx)
 		if err != nil {
@@ -312,9 +327,15 @@ func (app *App) Provision(ctx caddy.Context) error {
 
 // Validate ensures the app's configuration is valid.
 func (app *App) Validate() error {
+	isGo120 := strings.Contains(runtime.Version(), "go1.20")
+
 	// each server must use distinct listener addresses
 	lnAddrs := make(map[string]string)
 	for srvName, srv := range app.Servers {
+		if isGo120 && srv.EnableFullDuplex {
+			app.logger.Warn("enable_full_duplex is not supported in Go 1.20, use a build made with Go 1.21 or later", zap.String("server", srvName))
+		}
+
 		for _, addr := range srv.Listen {
 			listenAddr, err := caddy.ParseNetworkAddress(addr)
 			if err != nil {
@@ -352,6 +373,14 @@ func (app *App) Start() error {
 			MaxHeaderBytes:    srv.MaxHeaderBytes,
 			Handler:           srv,
 			ErrorLog:          serverLogger,
+			ConnContext: func(ctx context.Context, c net.Conn) context.Context {
+				return context.WithValue(ctx, ConnCtxKey, c)
+			},
+		}
+		h2server := &http2.Server{
+			NewWriteScheduler: func() http2.WriteScheduler {
+				return http2.NewPriorityWriteScheduler(nil)
+			},
 		}
 
 		// disable HTTP/2, which we enabled by default during provisioning
@@ -373,6 +402,9 @@ func (app *App) Start() error {
 					}
 				}
 			}
+		} else {
+			//nolint:errcheck
+			http2.ConfigureServer(srv.server, h2server)
 		}
 
 		// this TLS config is used by the std lib to choose the actual TLS config for connections
@@ -382,9 +414,6 @@ func (app *App) Start() error {
 
 		// enable H2C if configured
 		if srv.protocol("h2c") {
-			h2server := &http2.Server{
-				IdleTimeout: time.Duration(srv.IdleTimeout),
-			}
 			srv.server.Handler = h2c.NewHandler(srv, h2server)
 		}
 
@@ -451,6 +480,17 @@ func (app *App) Start() error {
 					ln = srv.listenerWrappers[i].WrapListener(ln)
 				}
 
+				// handle http2 if use tls listener wrapper
+				if useTLS {
+					http2lnWrapper := &http2Listener{
+						Listener: ln,
+						server:   srv.server,
+						h2server: h2server,
+					}
+					srv.h2listeners = append(srv.h2listeners, http2lnWrapper)
+					ln = http2lnWrapper
+				}
+
 				// if binding to port 0, the OS chooses a port for us;
 				// but the user won't know the port unless we print it
 				if !listenAddr.IsUnixNetwork() && listenAddr.StartPort == 0 && listenAddr.EndPort == 0 {
@@ -517,7 +557,7 @@ func (app *App) Stop() error {
 
 	// honor scheduled/delayed shutdown time
 	if delay {
-		app.logger.Debug("shutdown scheduled",
+		app.logger.Info("shutdown scheduled",
 			zap.Duration("delay_duration", time.Duration(app.ShutdownDelay)),
 			zap.Time("time", scheduledTime))
 		time.Sleep(time.Duration(app.ShutdownDelay))
@@ -528,9 +568,9 @@ func (app *App) Stop() error {
 		var cancel context.CancelFunc
 		ctx, cancel = context.WithTimeout(ctx, time.Duration(app.GracePeriod))
 		defer cancel()
-		app.logger.Debug("servers shutting down; grace period initiated", zap.Duration("duration", time.Duration(app.GracePeriod)))
+		app.logger.Info("servers shutting down; grace period initiated", zap.Duration("duration", time.Duration(app.GracePeriod)))
 	} else {
-		app.logger.Debug("servers shutting down with eternal grace period")
+		app.logger.Info("servers shutting down with eternal grace period")
 	}
 
 	// goroutines aren't guaranteed to be scheduled right away,
@@ -562,6 +602,21 @@ func (app *App) Stop() error {
 			return
 		}
 
+		// First close h3server then close listeners unlike stdlib for several reasons:
+		// 1, udp has only a single socket, once closed, no more data can be read and
+		// written. In contrast, closing tcp listeners won't affect established connections.
+		// This have something to do with graceful shutdown when upstream implements it.
+		// 2, h3server will only close listeners it's registered (quic listeners). Closing
+		// listener first and these listeners maybe unregistered thus won't be closed. caddy
+		// distinguishes quic-listener and underlying datagram sockets.
+
+		// TODO: CloseGracefully, once implemented upstream (see https://github.com/quic-go/quic-go/issues/2103)
+		if err := server.h3server.Close(); err != nil {
+			app.logger.Error("HTTP/3 server shutdown",
+				zap.Error(err),
+				zap.Strings("addresses", server.Listen))
+		}
+
 		// TODO: we have to manually close our listeners because quic-go won't
 		// close listeners it didn't create along with the server itself...
 		// see https://github.com/quic-go/quic-go/issues/3560
@@ -572,20 +627,26 @@ func (app *App) Stop() error {
 					zap.String("address", el.LocalAddr().String()))
 			}
 		}
+	}
+	stopH2Listener := func(server *Server) {
+		defer finishedShutdown.Done()
+		startedShutdown.Done()
 
-		// TODO: CloseGracefully, once implemented upstream (see https://github.com/quic-go/quic-go/issues/2103)
-		if err := server.h3server.Close(); err != nil {
-			app.logger.Error("HTTP/3 server shutdown",
-				zap.Error(err),
-				zap.Strings("addresses", server.Listen))
+		for i, s := range server.h2listeners {
+			if err := s.Shutdown(ctx); err != nil {
+				app.logger.Error("http2 listener shutdown",
+					zap.Error(err),
+					zap.Int("index", i))
+			}
 		}
 	}
 
 	for _, server := range app.Servers {
-		startedShutdown.Add(2)
-		finishedShutdown.Add(2)
+		startedShutdown.Add(3)
+		finishedShutdown.Add(3)
 		go stopServer(server)
 		go stopH3Server(server)
+		go stopH2Listener(server)
 	}
 
 	// block until all the goroutines have been run by the scheduler;

modules/caddyhttp/autohttps.go

@@ -83,6 +83,8 @@ func (ahc AutoHTTPSConfig) Skipped(name string, skipSlice []string) bool {
 // even servers to the app, which still need to be set up with the
 // rest of them during provisioning.
 func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) error {
+	logger := app.logger.Named("auto_https")
+
 	// this map acts as a set to store the domain names
 	// for which we will manage certificates automatically
 	uniqueDomainsForCerts := make(map[string]struct{})
@@ -114,13 +116,13 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 			srv.AutoHTTPS = new(AutoHTTPSConfig)
 		}
 		if srv.AutoHTTPS.Disabled {
-			app.logger.Warn("automatic HTTPS is completely disabled for server", zap.String("server_name", srvName))
+			logger.Warn("automatic HTTPS is completely disabled for server", zap.String("server_name", srvName))
 			continue
 		}
 
 		// skip if all listeners use the HTTP port
 		if !srv.listenersUseAnyPortOtherThan(app.httpPort()) {
-			app.logger.Warn("server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server",
+			logger.Warn("server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server",
 				zap.String("server_name", srvName),
 				zap.Int("http_port", app.httpPort()),
 			)
@@ -134,7 +136,7 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 		// needing to specify one empty policy to enable it
 		if srv.TLSConnPolicies == nil &&
 			!srv.listenersUseAnyPortOtherThan(app.httpsPort()) {
-			app.logger.Info("server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS",
+			logger.Info("server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS",
 				zap.String("server_name", srvName),
 				zap.Int("https_port", app.httpsPort()),
 			)
@@ -186,22 +188,16 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 		// a deduplicated list of names for which to obtain certs
 		// (only if cert management not disabled for this server)
 		if srv.AutoHTTPS.DisableCerts {
-			app.logger.Warn("skipping automated certificate management for server because it is disabled", zap.String("server_name", srvName))
+			logger.Warn("skipping automated certificate management for server because it is disabled", zap.String("server_name", srvName))
 		} else {
 			for d := range serverDomainSet {
-				// the implicit Tailscale manager module will get its own certs at run-time
-				if isTailscaleDomain(d) {
-					continue
-				}
-
 				if certmagic.SubjectQualifiesForCert(d) &&
 					!srv.AutoHTTPS.Skipped(d, srv.AutoHTTPS.SkipCerts) {
 					// if a certificate for this name is already loaded,
 					// don't obtain another one for it, unless we are
 					// supposed to ignore loaded certificates
-					if !srv.AutoHTTPS.IgnoreLoadedCerts &&
-						len(app.tlsApp.AllMatchingCertificates(d)) > 0 {
-						app.logger.Info("skipping automatic certificate management because one or more matching certificates are already loaded",
+					if !srv.AutoHTTPS.IgnoreLoadedCerts && app.tlsApp.HasCertificateForSubject(d) {
+						logger.Info("skipping automatic certificate management because one or more matching certificates are already loaded",
 							zap.String("domain", d),
 							zap.String("server_name", srvName),
 						)
@@ -212,7 +208,7 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 					// can handle that, but as a courtesy, warn the user
 					if strings.Contains(d, "*") &&
 						strings.Count(strings.Trim(d, "."), ".") == 1 {
-						app.logger.Warn("most clients do not trust second-level wildcard certificates (*.tld)",
+						logger.Warn("most clients do not trust second-level wildcard certificates (*.tld)",
 							zap.String("domain", d))
 					}
 
@@ -228,11 +224,11 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 
 		// nothing left to do if auto redirects are disabled
 		if srv.AutoHTTPS.DisableRedir {
-			app.logger.Warn("automatic HTTP->HTTPS redirects are disabled", zap.String("server_name", srvName))
+			logger.Warn("automatic HTTP->HTTPS redirects are disabled", zap.String("server_name", srvName))
 			continue
 		}
 
-		app.logger.Info("enabling automatic HTTP->HTTPS redirects", zap.String("server_name", srvName))
+		logger.Info("enabling automatic HTTP->HTTPS redirects", zap.String("server_name", srvName))
 
 		// create HTTP->HTTPS redirects
 		for _, listenAddr := range srv.Listen {
@@ -272,12 +268,15 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 	// we now have a list of all the unique names for which we need certs;
 	// turn the set into a slice so that phase 2 can use it
 	app.allCertDomains = make([]string, 0, len(uniqueDomainsForCerts))
-	var internal []string
+	var internal, tailscale []string
 uniqueDomainsLoop:
 	for d := range uniqueDomainsForCerts {
-		// whether or not there is already an automation policy for this
-		// name, we should add it to the list to manage a cert for it
-		app.allCertDomains = append(app.allCertDomains, d)
+		if !isTailscaleDomain(d) {
+			// whether or not there is already an automation policy for this
+			// name, we should add it to the list to manage a cert for it,
+			// unless it's a Tailscale domain, because we don't manage those
+			app.allCertDomains = append(app.allCertDomains, d)
+		}
 
 		// some names we've found might already have automation policies
 		// explicitly specified for them; we should exclude those from
@@ -285,7 +284,7 @@ uniqueDomainsLoop:
 		// one automation policy would be confusing and an error
 		if app.tlsApp.Automation != nil {
 			for _, ap := range app.tlsApp.Automation.Policies {
-				for _, apHost := range ap.Subjects {
+				for _, apHost := range ap.Subjects() {
 					if apHost == d {
 						continue uniqueDomainsLoop
 					}
@@ -295,13 +294,15 @@ uniqueDomainsLoop:
 
 		// if no automation policy exists for the name yet, we
 		// will associate it with an implicit one
-		if !certmagic.SubjectQualifiesForPublicCert(d) {
+		if isTailscaleDomain(d) {
+			tailscale = append(tailscale, d)
+		} else if !certmagic.SubjectQualifiesForPublicCert(d) {
 			internal = append(internal, d)
 		}
 	}
 
 	// ensure there is an automation policy to handle these certs
-	err := app.createAutomationPolicies(ctx, internal)
+	err := app.createAutomationPolicies(ctx, internal, tailscale)
 	if err != nil {
 		return err
 	}
@@ -424,6 +425,10 @@ redirServersLoop:
 		}
 	}
 
+	logger.Debug("adjusted config",
+		zap.Reflect("tls", app.tlsApp),
+		zap.Reflect("http", app))
+
 	return nil
 }
 
@@ -466,7 +471,7 @@ func (app *App) makeRedirRoute(redirToPort uint, matcherSet MatcherSet) Route {
 // automation policy exists, it will be shallow-copied and used as the
 // base for the new ones (this is important for preserving behavior the
 // user intends to be "defaults").
-func (app *App) createAutomationPolicies(ctx caddy.Context, internalNames []string) error {
+func (app *App) createAutomationPolicies(ctx caddy.Context, internalNames, tailscaleNames []string) error {
 	// before we begin, loop through the existing automation policies
 	// and, for any ACMEIssuers we find, make sure they're filled in
 	// with default values that might be specified in our HTTP app; also
@@ -480,6 +485,22 @@ func (app *App) createAutomationPolicies(ctx caddy.Context, internalNames []stri
 		app.tlsApp.Automation = new(caddytls.AutomationConfig)
 	}
 	for _, ap := range app.tlsApp.Automation.Policies {
+		// on-demand policies can have the tailscale manager added implicitly
+		// if there's no explicit manager configured -- for convenience
+		if ap.OnDemand && len(ap.Managers) == 0 {
+			var ts caddytls.Tailscale
+			if err := ts.Provision(ctx); err != nil {
+				return err
+			}
+			ap.Managers = []certmagic.Manager{ts}
+
+			// must reprovision the automation policy so that the underlying
+			// CertMagic config knows about the updated Managers
+			if err := ap.Provision(app.tlsApp); err != nil {
+				return fmt.Errorf("re-provisioning automation policy: %v", err)
+			}
+		}
+
 		// set up default issuer -- honestly, this is only
 		// really necessary because the HTTP app is opinionated
 		// and has settings which could be inferred as new
@@ -501,24 +522,8 @@ func (app *App) createAutomationPolicies(ctx caddy.Context, internalNames []stri
 			}
 		}
 
-		// if no external managers were configured, enable
-		// implicit Tailscale support for convenience
-		if ap.Managers == nil {
-			ts, err := implicitTailscale(ctx)
-			if err != nil {
-				return err
-			}
-			ap.Managers = []certmagic.Manager{ts}
-
-			// must reprovision the automation policy so that the underlying
-			// CertMagic config knows about the updated Managers
-			if err := ap.Provision(app.tlsApp); err != nil {
-				return fmt.Errorf("re-provisioning automation policy: %v", err)
-			}
-		}
-
 		// while we're here, is this the catch-all/base policy?
-		if !foundBasePolicy && len(ap.Subjects) == 0 {
+		if !foundBasePolicy && len(ap.SubjectsRaw) == 0 {
 			basePolicy = ap
 			foundBasePolicy = true
 		}
@@ -529,15 +534,6 @@ func (app *App) createAutomationPolicies(ctx caddy.Context, internalNames []stri
 		basePolicy = new(caddytls.AutomationPolicy)
 	}
 
-	if basePolicy.Managers == nil {
-		// add implicit Tailscale integration, for harmless convenience
-		ts, err := implicitTailscale(ctx)
-		if err != nil {
-			return err
-		}
-		basePolicy.Managers = []certmagic.Manager{ts}
-	}
-
 	// if the basePolicy has an existing ACMEIssuer (particularly to
 	// include any type that embeds/wraps an ACMEIssuer), let's use it
 	// (I guess we just use the first one?), otherwise we'll make one
@@ -634,7 +630,7 @@ func (app *App) createAutomationPolicies(ctx caddy.Context, internalNames []stri
 		// rather they just want to change the CA for the set
 		// of names that would normally use the production API;
 		// anyway, that gets into the weeds a bit...
-		newPolicy.Subjects = internalNames
+		newPolicy.SubjectsRaw = internalNames
 		newPolicy.Issuers = []certmagic.Issuer{internalIssuer}
 		err := app.tlsApp.AddAutomationPolicy(newPolicy)
 		if err != nil {
@@ -642,6 +638,27 @@ func (app *App) createAutomationPolicies(ctx caddy.Context, internalNames []stri
 		}
 	}
 
+	// tailscale names go in their own automation policies because
+	// they require on-demand TLS to be enabled, which we obviously
+	// can't enable for everything
+	if len(tailscaleNames) > 0 {
+		policyCopy := *basePolicy
+		newPolicy := &policyCopy
+
+		var ts caddytls.Tailscale
+		if err := ts.Provision(ctx); err != nil {
+			return err
+		}
+
+		newPolicy.SubjectsRaw = tailscaleNames
+		newPolicy.Issuers = nil
+		newPolicy.Managers = append(newPolicy.Managers, ts)
+		err := app.tlsApp.AddAutomationPolicy(newPolicy)
+		if err != nil {
+			return err
+		}
+	}
+
 	// we just changed a lot of stuff, so double-check that it's all good
 	err := app.tlsApp.Validate()
 	if err != nil {
@@ -720,13 +737,6 @@ func (app *App) automaticHTTPSPhase2() error {
 	return nil
 }
 
-// implicitTailscale returns a new and provisioned Tailscale module configured to be optional.
-func implicitTailscale(ctx caddy.Context) (caddytls.Tailscale, error) {
-	ts := caddytls.Tailscale{Optional: true}
-	err := ts.Provision(ctx)
-	return ts, err
-}
-
 func isTailscaleDomain(name string) bool {
 	return strings.HasSuffix(strings.ToLower(name), ".ts.net")
 }

modules/caddyhttp/caddyauth/basicauth.go

@@ -23,7 +23,6 @@ import (
 	"net/http"
 	"strings"
 	"sync"
-	"time"
 
 	"github.com/caddyserver/caddy/v2"
 	"golang.org/x/sync/singleflight"
@@ -31,8 +30,6 @@ import (
 
 func init() {
 	caddy.RegisterModule(HTTPBasicAuth{})
-
-	weakrand.Seed(time.Now().UnixNano())
 }
 
 // HTTPBasicAuth facilitates HTTP basic authentication.

modules/caddyhttp/caddyauth/command.go

@@ -18,20 +18,19 @@ import (
 	"bufio"
 	"bytes"
 	"encoding/base64"
-	"flag"
 	"fmt"
 	"os"
 	"os/signal"
 
 	"github.com/caddyserver/caddy/v2"
 	caddycmd "github.com/caddyserver/caddy/v2/cmd"
+	"github.com/spf13/cobra"
 	"golang.org/x/term"
 )
 
 func init() {
 	caddycmd.RegisterCommand(caddycmd.Command{
 		Name:  "hash-password",
-		Func:  cmdHashPassword,
 		Usage: "[--algorithm <name>] [--salt <string>] [--plaintext <password>]",
 		Short: "Hashes a password and writes base64",
 		Long: `
@@ -50,13 +49,12 @@ be provided (scrypt).
 
 Note that scrypt is deprecated. Please use 'bcrypt' instead.
 `,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("hash-password", flag.ExitOnError)
-			fs.String("algorithm", "bcrypt", "Name of the hash algorithm")
-			fs.String("plaintext", "", "The plaintext password")
-			fs.String("salt", "", "The password salt")
-			return fs
-		}(),
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("plaintext", "p", "", "The plaintext password")
+			cmd.Flags().StringP("salt", "s", "", "The password salt")
+			cmd.Flags().StringP("algorithm", "a", "bcrypt", "Name of the hash algorithm")
+			cmd.RunE = caddycmd.WrapCommandFuncForCobra(cmdHashPassword)
+		},
 	})
 }
 

modules/caddyhttp/celmatcher.go

@@ -191,15 +191,17 @@ func (m MatchExpression) caddyPlaceholderFunc(lhs, rhs ref.Val) ref.Val {
 	celReq, ok := lhs.(celHTTPRequest)
 	if !ok {
 		return types.NewErr(
-			"invalid request of type '%v' to "+placeholderFuncName+"(request, placeholderVarName)",
+			"invalid request of type '%v' to %s(request, placeholderVarName)",
 			lhs.Type(),
+			placeholderFuncName,
 		)
 	}
 	phStr, ok := rhs.(types.String)
 	if !ok {
 		return types.NewErr(
-			"invalid placeholder variable name of type '%v' to "+placeholderFuncName+"(request, placeholderVarName)",
+			"invalid placeholder variable name of type '%v' to %s(request, placeholderVarName)",
 			rhs.Type(),
+			placeholderFuncName,
 		)
 	}
 
@@ -253,7 +255,10 @@ type celPkixName struct{ *pkix.Name }
 func (pn celPkixName) ConvertToNative(typeDesc reflect.Type) (any, error) {
 	return pn.Name, nil
 }
-func (celPkixName) ConvertToType(typeVal ref.Type) ref.Val {
+func (pn celPkixName) ConvertToType(typeVal ref.Type) ref.Val {
+	if typeVal.TypeName() == "string" {
+		return types.String(pn.Name.String())
+	}
 	panic("not implemented")
 }
 func (pn celPkixName) Equal(other ref.Val) ref.Val {
@@ -491,7 +496,7 @@ func celMatcherStringMacroExpander(funcName string) parser.MacroExpander {
 	}
 }
 
-// celMatcherStringMacroExpander validates that the macro is called a single
+// celMatcherJSONMacroExpander validates that the macro is called a single
 // map literal argument.
 //
 // The following function call is returned: <funcName>(request, arg)

modules/caddyhttp/duplex_go120.go

@@ -0,0 +1,25 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !go1.21
+
+package caddyhttp
+
+import (
+	"net/http"
+)
+
+func enableFullDuplex(w http.ResponseWriter) {
+	// Do nothing, Go 1.20 and earlier do not support full duplex
+}

modules/caddyhttp/duplex_go121.go

@@ -0,0 +1,25 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build go1.21
+
+package caddyhttp
+
+import (
+	"net/http"
+)
+
+func enableFullDuplex(w http.ResponseWriter) {
+	http.NewResponseController(w).EnableFullDuplex()
+}

modules/caddyhttp/encode/encode.go

@@ -20,9 +20,11 @@
 package encode
 
 import (
+	"bufio"
 	"fmt"
 	"io"
 	"math"
+	"net"
 	"net/http"
 	"sort"
 	"strconv"
@@ -165,10 +167,10 @@ func (enc *Encode) openResponseWriter(encodingName string, w http.ResponseWriter
 // initResponseWriter initializes the responseWriter instance
 // allocated in openResponseWriter, enabling mid-stack inlining.
 func (enc *Encode) initResponseWriter(rw *responseWriter, encodingName string, wrappedRW http.ResponseWriter) *responseWriter {
-	if httpInterfaces, ok := wrappedRW.(caddyhttp.HTTPInterfaces); ok {
-		rw.HTTPInterfaces = httpInterfaces
+	if rww, ok := wrappedRW.(*caddyhttp.ResponseWriterWrapper); ok {
+		rw.ResponseWriter = rww
 	} else {
-		rw.HTTPInterfaces = &caddyhttp.ResponseWriterWrapper{ResponseWriter: wrappedRW}
+		rw.ResponseWriter = &caddyhttp.ResponseWriterWrapper{ResponseWriter: wrappedRW}
 	}
 	rw.encodingName = encodingName
 	rw.config = enc
@@ -180,7 +182,7 @@ func (enc *Encode) initResponseWriter(rw *responseWriter, encodingName string, w
 // using the encoding represented by encodingName and
 // configured by config.
 type responseWriter struct {
-	caddyhttp.HTTPInterfaces
+	http.ResponseWriter
 	encodingName string
 	w            Encoder
 	config       *Encode
@@ -209,7 +211,21 @@ func (rw *responseWriter) Flush() {
 		// to rw.Write (see bug in #4314)
 		return
 	}
-	rw.HTTPInterfaces.Flush()
+	//nolint:bodyclose
+	http.NewResponseController(rw.ResponseWriter).Flush()
+}
+
+// Hijack implements http.Hijacker. It will flush status code if set. We don't track actual hijacked
+// status assuming http middlewares will track its status.
+func (rw *responseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) {
+	if !rw.wroteHeader {
+		if rw.statusCode != 0 {
+			rw.ResponseWriter.WriteHeader(rw.statusCode)
+		}
+		rw.wroteHeader = true
+	}
+	//nolint:bodyclose
+	return http.NewResponseController(rw.ResponseWriter).Hijack()
 }
 
 // Write writes to the response. If the response qualifies,
@@ -246,7 +262,7 @@ func (rw *responseWriter) Write(p []byte) (int, error) {
 	// by the standard library
 	if !rw.wroteHeader {
 		if rw.statusCode != 0 {
-			rw.HTTPInterfaces.WriteHeader(rw.statusCode)
+			rw.ResponseWriter.WriteHeader(rw.statusCode)
 		}
 		rw.wroteHeader = true
 	}
@@ -254,7 +270,7 @@ func (rw *responseWriter) Write(p []byte) (int, error) {
 	if rw.w != nil {
 		return rw.w.Write(p)
 	} else {
-		return rw.HTTPInterfaces.Write(p)
+		return rw.ResponseWriter.Write(p)
 	}
 }
 
@@ -270,7 +286,7 @@ func (rw *responseWriter) Close() error {
 
 		// issue #5059, don't write status code if not set explicitly.
 		if rw.statusCode != 0 {
-			rw.HTTPInterfaces.WriteHeader(rw.statusCode)
+			rw.ResponseWriter.WriteHeader(rw.statusCode)
 		}
 		rw.wroteHeader = true
 	}
@@ -285,13 +301,18 @@ func (rw *responseWriter) Close() error {
 	return err
 }
 
+// Unwrap returns the underlying ResponseWriter.
+func (rw *responseWriter) Unwrap() http.ResponseWriter {
+	return rw.ResponseWriter
+}
+
 // init should be called before we write a response, if rw.buf has contents.
 func (rw *responseWriter) init() {
 	if rw.Header().Get("Content-Encoding") == "" && isEncodeAllowed(rw.Header()) &&
 		rw.config.Match(rw) {
 
 		rw.w = rw.config.writerPools[rw.encodingName].Get().(Encoder)
-		rw.w.Reset(rw.HTTPInterfaces)
+		rw.w.Reset(rw.ResponseWriter)
 		rw.Header().Del("Content-Length") // https://github.com/golang/go/issues/14975
 		rw.Header().Set("Content-Encoding", rw.encodingName)
 		rw.Header().Add("Vary", "Accept-Encoding")
@@ -410,5 +431,4 @@ var (
 	_ caddy.Provisioner           = (*Encode)(nil)
 	_ caddy.Validator             = (*Encode)(nil)
 	_ caddyhttp.MiddlewareHandler = (*Encode)(nil)
-	_ caddyhttp.HTTPInterfaces    = (*responseWriter)(nil)
 )

modules/caddyhttp/errors.go

@@ -15,27 +15,24 @@
 package caddyhttp
 
 import (
+	"errors"
 	"fmt"
 	weakrand "math/rand"
 	"path"
 	"runtime"
 	"strings"
-	"time"
 
 	"github.com/caddyserver/caddy/v2"
 )
 
-func init() {
-	weakrand.Seed(time.Now().UnixNano())
-}
-
 // Error is a convenient way for a Handler to populate the
 // essential fields of a HandlerError. If err is itself a
 // HandlerError, then any essential fields that are not
 // set will be populated.
 func Error(statusCode int, err error) HandlerError {
 	const idLen = 9
-	if he, ok := err.(HandlerError); ok {
+	var he HandlerError
+	if errors.As(err, &he) {
 		if he.ID == "" {
 			he.ID = randString(idLen, true)
 		}

modules/caddyhttp/fileserver/browse.go

@@ -82,8 +82,8 @@ func (fsrv *FileServer) serveBrowse(root, dirPath string, w http.ResponseWriter,
 
 	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)
 
-	// calling path.Clean here prevents weird breadcrumbs when URL paths are sketchy like /%2e%2e%2f
-	listing, err := fsrv.loadDirectoryContents(r.Context(), dir.(fs.ReadDirFile), root, path.Clean(r.URL.Path), repl)
+	// TODO: not entirely sure if path.Clean() is necessary here but seems like a safe plan (i.e. /%2e%2e%2f) - someone could verify this
+	listing, err := fsrv.loadDirectoryContents(r.Context(), dir.(fs.ReadDirFile), root, path.Clean(r.URL.EscapedPath()), repl)
 	switch {
 	case os.IsPermission(err):
 		return caddyhttp.Error(http.StatusForbidden, err)
@@ -93,7 +93,7 @@ func (fsrv *FileServer) serveBrowse(root, dirPath string, w http.ResponseWriter,
 		return caddyhttp.Error(http.StatusInternalServerError, err)
 	}
 
-	fsrv.browseApplyQueryParams(w, r, &listing)
+	fsrv.browseApplyQueryParams(w, r, listing)
 
 	buf := bufPool.Get().(*bytes.Buffer)
 	buf.Reset()
@@ -137,10 +137,10 @@ func (fsrv *FileServer) serveBrowse(root, dirPath string, w http.ResponseWriter,
 	return nil
 }
 
-func (fsrv *FileServer) loadDirectoryContents(ctx context.Context, dir fs.ReadDirFile, root, urlPath string, repl *caddy.Replacer) (browseTemplateContext, error) {
+func (fsrv *FileServer) loadDirectoryContents(ctx context.Context, dir fs.ReadDirFile, root, urlPath string, repl *caddy.Replacer) (*browseTemplateContext, error) {
 	files, err := dir.ReadDir(10000) // TODO: this limit should probably be configurable
 	if err != nil && err != io.EOF {
-		return browseTemplateContext{}, err
+		return nil, err
 	}
 
 	// user can presumably browse "up" to parent folder if path is longer than "/"
@@ -152,12 +152,20 @@ func (fsrv *FileServer) loadDirectoryContents(ctx context.Context, dir fs.ReadDi
 // browseApplyQueryParams applies query parameters to the listing.
 // It mutates the listing and may set cookies.
 func (fsrv *FileServer) browseApplyQueryParams(w http.ResponseWriter, r *http.Request, listing *browseTemplateContext) {
+	layoutParam := r.URL.Query().Get("layout")
 	sortParam := r.URL.Query().Get("sort")
 	orderParam := r.URL.Query().Get("order")
 	limitParam := r.URL.Query().Get("limit")
 	offsetParam := r.URL.Query().Get("offset")
 
-	// first figure out what to sort by
+	switch layoutParam {
+	case "list", "grid", "":
+		listing.Layout = layoutParam
+	default:
+		listing.Layout = "list"
+	}
+
+	// figure out what to sort by
 	switch sortParam {
 	case "":
 		sortParam = sortByNameDirFirst
@@ -229,7 +237,7 @@ func isSymlink(f fs.FileInfo) bool {
 // features.
 type templateContext struct {
 	templates.TemplateContext
-	browseTemplateContext
+	*browseTemplateContext
 }
 
 // bufPool is used to increase the efficiency of file listings.

modules/caddyhttp/fileserver/browsetplcontext.go

@@ -31,11 +31,16 @@ import (
 	"go.uber.org/zap"
 )
 
-func (fsrv *FileServer) directoryListing(ctx context.Context, entries []fs.DirEntry, canGoUp bool, root, urlPath string, repl *caddy.Replacer) browseTemplateContext {
+func (fsrv *FileServer) directoryListing(ctx context.Context, entries []fs.DirEntry, canGoUp bool, root, urlPath string, repl *caddy.Replacer) *browseTemplateContext {
 	filesToHide := fsrv.transformHidePaths(repl)
 
-	var dirCount, fileCount int
-	fileInfos := []fileInfo{}
+	name, _ := url.PathUnescape(urlPath)
+
+	tplCtx := &browseTemplateContext{
+		Name:    path.Base(name),
+		Path:    urlPath,
+		CanGoUp: canGoUp,
+	}
 
 	for _, entry := range entries {
 		if err := ctx.Err(); err != nil {
@@ -61,9 +66,9 @@ func (fsrv *FileServer) directoryListing(ctx context.Context, entries []fs.DirEn
 		// add the slash after the escape of path to avoid escaping the slash as well
 		if isDir {
 			name += "/"
-			dirCount++
+			tplCtx.NumDirs++
 		} else {
-			fileCount++
+			tplCtx.NumFiles++
 		}
 
 		size := info.Size()
@@ -82,7 +87,7 @@ func (fsrv *FileServer) directoryListing(ctx context.Context, entries []fs.DirEn
 
 		u := url.URL{Path: "./" + name} // prepend with "./" to fix paths with ':' in the name
 
-		fileInfos = append(fileInfos, fileInfo{
+		tplCtx.Items = append(tplCtx.Items, fileInfo{
 			IsDir:     isDir,
 			IsSymlink: fileIsSymlink,
 			Name:      name,
@@ -90,17 +95,11 @@ func (fsrv *FileServer) directoryListing(ctx context.Context, entries []fs.DirEn
 			URL:       u.String(),
 			ModTime:   info.ModTime().UTC(),
 			Mode:      info.Mode(),
+			Tpl:       tplCtx, // a reference up to the template context is useful
 		})
 	}
-	name, _ := url.PathUnescape(urlPath)
-	return browseTemplateContext{
-		Name:     path.Base(name),
-		Path:     urlPath,
-		CanGoUp:  canGoUp,
-		Items:    fileInfos,
-		NumDirs:  dirCount,
-		NumFiles: fileCount,
-	}
+
+	return tplCtx
 }
 
 // browseTemplateContext provides the template context for directory listings.
@@ -134,6 +133,9 @@ type browseTemplateContext struct {
 
 	// Sorting order
 	Order string `json:"order,omitempty"`
+
+	// Display format (list or grid)
+	Layout string `json:"layout,omitempty"`
 }
 
 // Breadcrumbs returns l.Path where every element maps
@@ -227,6 +229,19 @@ type fileInfo struct {
 	Mode      os.FileMode `json:"mode"`
 	IsDir     bool        `json:"is_dir"`
 	IsSymlink bool        `json:"is_symlink"`
+
+	// a pointer to the template context is useful inside nested templates
+	Tpl *browseTemplateContext `json:"-"`
+}
+
+// HasExt returns true if the filename has any of the given suffixes, case-insensitive.
+func (fi fileInfo) HasExt(exts ...string) bool {
+	for _, ext := range exts {
+		if strings.HasSuffix(strings.ToLower(fi.Name), strings.ToLower(ext)) {
+			return true
+		}
+	}
+	return false
 }
 
 // HumanSize returns the size of the file as a

modules/caddyhttp/fileserver/browsetplcontext_test.go

@@ -25,6 +25,45 @@ func TestBreadcrumbs(t *testing.T) {
 	}{
 		{"", []crumb{}},
 		{"/", []crumb{{Text: "/"}}},
+		{"/foo/", []crumb{
+			{Link: "../", Text: "/"},
+			{Link: "", Text: "foo"},
+		}},
+		{"/foo/bar/", []crumb{
+			{Link: "../../", Text: "/"},
+			{Link: "../", Text: "foo"},
+			{Link: "", Text: "bar"},
+		}},
+		{"/foo bar/", []crumb{
+			{Link: "../", Text: "/"},
+			{Link: "", Text: "foo bar"},
+		}},
+		{"/foo bar/baz/", []crumb{
+			{Link: "../../", Text: "/"},
+			{Link: "../", Text: "foo bar"},
+			{Link: "", Text: "baz"},
+		}},
+		{"/100%25 test coverage/is a lie/", []crumb{
+			{Link: "../../", Text: "/"},
+			{Link: "../", Text: "100% test coverage"},
+			{Link: "", Text: "is a lie"},
+		}},
+		{"/AC%2FDC/", []crumb{
+			{Link: "../", Text: "/"},
+			{Link: "", Text: "AC/DC"},
+		}},
+		{"/foo/%2e%2e%2f/bar", []crumb{
+			{Link: "../../../", Text: "/"},
+			{Link: "../../", Text: "foo"},
+			{Link: "../", Text: "../"},
+			{Link: "", Text: "bar"},
+		}},
+		{"/foo/../bar", []crumb{
+			{Link: "../../../", Text: "/"},
+			{Link: "../../", Text: "foo"},
+			{Link: "../", Text: ".."},
+			{Link: "", Text: "bar"},
+		}},
 		{"foo/bar/baz", []crumb{
 			{Link: "../../", Text: "foo"},
 			{Link: "../", Text: "bar"},
@@ -51,16 +90,16 @@ func TestBreadcrumbs(t *testing.T) {
 		}},
 	}
 
-	for _, d := range testdata {
+	for testNum, d := range testdata {
 		l := browseTemplateContext{Path: d.path}
 		actual := l.Breadcrumbs()
 		if len(actual) != len(d.expected) {
-			t.Errorf("wrong size output, got %d elements but expected %d", len(actual), len(d.expected))
+			t.Errorf("Test %d: Got %d components but expected %d; got: %+v", testNum, len(actual), len(d.expected), actual)
 			continue
 		}
 		for i, c := range actual {
 			if c != d.expected[i] {
-				t.Errorf("got %#v but expected %#v at index %d", c, d.expected[i], i)
+				t.Errorf("Test %d crumb %d: got %#v but expected %#v at index %d", testNum, i, c, d.expected[i], i)
 			}
 		}
 	}

modules/caddyhttp/fileserver/command.go

@@ -16,8 +16,9 @@ package fileserver
 
 import (
 	"encoding/json"
-	"flag"
+	"io"
 	"log"
+	"os"
 	"strconv"
 	"time"
 
@@ -27,13 +28,13 @@ import (
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 	caddytpl "github.com/caddyserver/caddy/v2/modules/caddyhttp/templates"
 	"github.com/caddyserver/certmagic"
+	"github.com/spf13/cobra"
 	"go.uber.org/zap"
 )
 
 func init() {
 	caddycmd.RegisterCommand(caddycmd.Command{
 		Name:  "file-server",
-		Func:  cmdFileServer,
 		Usage: "[--domain <example.com>] [--root <path>] [--listen <addr>] [--browse] [--access-log]",
 		Short: "Spins up a production-ready file server",
 		Long: `
@@ -49,17 +50,25 @@ using this option.
 
 If --browse is enabled, requests for folders without an index file will
 respond with a file listing.`,
-		Flags: func() *flag.FlagSet {
-			fs := flag.NewFlagSet("file-server", flag.ExitOnError)
-			fs.String("domain", "", "Domain name at which to serve the files")
-			fs.String("root", "", "The path to the root of the site")
-			fs.String("listen", "", "The address to which to bind the listener")
-			fs.Bool("browse", false, "Enable directory browsing")
-			fs.Bool("templates", false, "Enable template rendering")
-			fs.Bool("access-log", false, "Enable the access log")
-			fs.Bool("debug", false, "Enable verbose debug logs")
-			return fs
-		}(),
+		CobraFunc: func(cmd *cobra.Command) {
+			cmd.Flags().StringP("domain", "d", "", "Domain name at which to serve the files")
+			cmd.Flags().StringP("root", "r", "", "The path to the root of the site")
+			cmd.Flags().StringP("listen", "", "", "The address to which to bind the listener")
+			cmd.Flags().BoolP("browse", "b", false, "Enable directory browsing")
+			cmd.Flags().BoolP("templates", "t", false, "Enable template rendering")
+			cmd.Flags().BoolP("access-log", "", false, "Enable the access log")
+			cmd.Flags().BoolP("debug", "v", false, "Enable verbose debug logs")
+			cmd.RunE = caddycmd.WrapCommandFuncForCobra(cmdFileServer)
+			cmd.AddCommand(&cobra.Command{
+				Use:     "export-template",
+				Short:   "Exports the default file browser template",
+				Example: "caddy file-server export-template > browse.html",
+				RunE: func(cmd *cobra.Command, args []string) error {
+					_, err := io.WriteString(os.Stdout, defaultBrowseTemplate)
+					return err
+				},
+			})
+		},
 	})
 }
 
@@ -136,7 +145,9 @@ func cmdFileServer(fs caddycmd.Flags) (int, error) {
 	if debug {
 		cfg.Logging = &caddy.Logging{
 			Logs: map[string]*caddy.CustomLog{
-				"default": {Level: zap.DebugLevel.CapitalString()},
+				"default": {
+					BaseLog: caddy.BaseLog{Level: zap.DebugLevel.CapitalString()},
+				},
 			},
 		}
 	}

modules/caddyhttp/fileserver/matcher.go

@@ -558,7 +558,7 @@ func indexFold(haystack, needle string) int {
 	return -1
 }
 
-// isCELMapLiteral returns whether the expression resolves to a map literal containing
+// isCELTryFilesLiteral returns whether the expression resolves to a map literal containing
 // only string keys with or a placeholder call.
 func isCELTryFilesLiteral(e *exprpb.Expr) bool {
 	switch e.GetExprKind().(type) {

modules/caddyhttp/fileserver/staticfiles.go

@@ -29,7 +29,6 @@ import (
 	"runtime"
 	"strconv"
 	"strings"
-	"time"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
@@ -38,8 +37,6 @@ import (
 )
 
 func init() {
-	weakrand.Seed(time.Now().UnixNano())
-
 	caddy.RegisterModule(FileServer{})
 }
 
@@ -250,7 +247,8 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 
 	root := repl.ReplaceAll(fsrv.Root, ".")
 
-	filename := caddyhttp.SanitizedPathJoin(root, r.URL.Path)
+	// remove any trailing `/` as it breaks fs.ValidPath() in the stdlib
+	filename := strings.TrimSuffix(caddyhttp.SanitizedPathJoin(root, r.URL.Path), "/")
 
 	fsrv.logger.Debug("sanitized path join",
 		zap.String("site_root", root),
@@ -355,7 +353,9 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 	}
 
 	var file fs.File
-	var etag string
+
+	// etag is usually unset, but if the user knows what they're doing, let them override it
+	etag := w.Header().Get("Etag")
 
 	// check for precompressed files
 	for _, ae := range encode.AcceptedEncodings(r, fsrv.PrecompressedOrder) {
@@ -387,7 +387,9 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 		// don't assign info = compressedInfo because sidecars are kind
 		// of transparent; however we do need to set the Etag:
 		// https://caddy.community/t/gzipped-sidecar-file-wrong-same-etag/16793
-		etag = calculateEtag(compressedInfo)
+		if etag == "" {
+			etag = calculateEtag(compressedInfo)
+		}
 
 		break
 	}
@@ -407,7 +409,9 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 		}
 		defer file.Close()
 
-		etag = calculateEtag(info)
+		if etag == "" {
+			etag = calculateEtag(info)
+		}
 	}
 
 	// at this point, we're serving a file; Go std lib supports only
@@ -420,7 +424,9 @@ func (fsrv *FileServer) ServeHTTP(w http.ResponseWriter, r *http.Request, next c
 
 	// set the Etag - note that a conditional If-None-Match request is handled
 	// by http.ServeContent below, which checks against this Etag value
-	w.Header().Set("Etag", etag)
+	if etag != "" {
+		w.Header().Set("Etag", etag)
+	}
 
 	if w.Header().Get("Content-Type") == "" {
 		mtyp := mime.TypeByExtension(filepath.Ext(filename))
@@ -607,7 +613,11 @@ func (fsrv *FileServer) notFound(w http.ResponseWriter, r *http.Request, next ca
 // Prefix the etag with "W/" to convert it into a weak etag.
 // See: https://tools.ietf.org/html/rfc7232#section-2.3
 func calculateEtag(d os.FileInfo) string {
-	t := strconv.FormatInt(d.ModTime().Unix(), 36)
+	mtime := d.ModTime().Unix()
+	if mtime == 0 || mtime == 1 {
+		return "" // not useful anyway; see issue #5548
+	}
+	t := strconv.FormatInt(mtime, 36)
 	s := strconv.FormatInt(d.Size(), 36)
 	return `"` + t + s + `"`
 }
@@ -635,6 +645,12 @@ func (wr statusOverrideResponseWriter) WriteHeader(int) {
 	wr.ResponseWriter.WriteHeader(wr.code)
 }
 
+// Unwrap returns the underlying ResponseWriter, necessary for
+// http.ResponseController to work correctly.
+func (wr statusOverrideResponseWriter) Unwrap() http.ResponseWriter {
+	return wr.ResponseWriter
+}
+
 // osFS is a simple fs.FS implementation that uses the local
 // file system. (We do not use os.DirFS because we do our own
 // rooting or path prefixing without being constrained to a single

modules/caddyhttp/headers/caddyfile.go

@@ -32,19 +32,20 @@ func init() {
 // parseCaddyfile sets up the handler for response headers from
 // Caddyfile tokens. Syntax:
 //
-//	header [<matcher>] [[+|-|?]<field> [<value|regexp>] [<replacement>]] {
-//	    [+]<field> [<value|regexp> [<replacement>]]
-//	    ?<field> <default_value>
-//	    -<field>
-//	    [defer]
+//	header [<matcher>] [[+|-|?|>]<field> [<value|regexp>] [<replacement>]] {
+//		[+]<field> [<value|regexp> [<replacement>]]
+//		?<field> <default_value>
+//		-<field>
+//		><field>
+//		[defer]
 //	}
 //
 // Either a block can be opened or a single header field can be configured
 // in the first line, but not both in the same directive. Header operations
 // are deferred to write-time if any headers are being deleted or if the
 // 'defer' subdirective is used. + appends a header value, - deletes a field,
-// and ? conditionally sets a value only if the header field is not already
-// set.
+// ? conditionally sets a value only if the header field is not already set,
+// and > sets a field with defer enabled.
 func parseCaddyfile(h httpcaddyfile.Helper) ([]httpcaddyfile.ConfigValue, error) {
 	if !h.Next() {
 		return nil, h.ArgErr()
@@ -246,10 +247,14 @@ func applyHeaderOp(ops *HeaderOps, respHeaderOps *RespHeaderOps, field, value, r
 		respHeaderOps.Set.Set(field, value)
 
 	case replacement != "": // replace
+		// allow defer shortcut for replace syntax
+		if strings.HasPrefix(field, ">") && respHeaderOps != nil {
+			respHeaderOps.Deferred = true
+		}
 		if ops.Replace == nil {
 			ops.Replace = make(map[string][]Replacement)
 		}
-		field = strings.TrimLeft(field, "+-?")
+		field = strings.TrimLeft(field, "+-?>")
 		ops.Replace[field] = append(
 			ops.Replace[field],
 			Replacement{
@@ -258,6 +263,15 @@ func applyHeaderOp(ops *HeaderOps, respHeaderOps *RespHeaderOps, field, value, r
 			},
 		)
 
+	case strings.HasPrefix(field, ">"): // set (overwrite) with defer
+		if ops.Set == nil {
+			ops.Set = make(http.Header)
+		}
+		ops.Set.Set(field[1:], value)
+		if respHeaderOps != nil {
+			respHeaderOps.Deferred = true
+		}
+
 	default: // set (overwrite)
 		if ops.Set == nil {
 			ops.Set = make(http.Header)

modules/caddyhttp/headers/headers.go

@@ -192,6 +192,19 @@ type RespHeaderOps struct {
 
 // ApplyTo applies ops to hdr using repl.
 func (ops HeaderOps) ApplyTo(hdr http.Header, repl *caddy.Replacer) {
+	// before manipulating headers in other ways, check if there
+	// is configuration to delete all headers, and do that first
+	// because if a header is to be added, we don't want to delete
+	// it also
+	for _, fieldName := range ops.Delete {
+		fieldName = repl.ReplaceKnown(fieldName, "")
+		if fieldName == "*" {
+			for existingField := range hdr {
+				delete(hdr, existingField)
+			}
+		}
+	}
+
 	// add
 	for fieldName, vals := range ops.Add {
 		fieldName = repl.ReplaceKnown(fieldName, "")
@@ -215,6 +228,9 @@ func (ops HeaderOps) ApplyTo(hdr http.Header, repl *caddy.Replacer) {
 	// delete
 	for _, fieldName := range ops.Delete {
 		fieldName = strings.ToLower(repl.ReplaceKnown(fieldName, ""))
+		if fieldName == "*" {
+			continue // handled above
+		}
 		switch {
 		case strings.HasPrefix(fieldName, "*") && strings.HasSuffix(fieldName, "*"):
 			for existingField := range hdr {
@@ -355,5 +371,5 @@ func (rww *responseWriterWrapper) Write(d []byte) (int, error) {
 var (
 	_ caddy.Provisioner           = (*Handler)(nil)
 	_ caddyhttp.MiddlewareHandler = (*Handler)(nil)
-	_ caddyhttp.HTTPInterfaces    = (*responseWriterWrapper)(nil)
+	_ http.ResponseWriter         = (*responseWriterWrapper)(nil)
 )

modules/caddyhttp/http2listener.go

@@ -0,0 +1,102 @@
+package caddyhttp
+
+import (
+	"context"
+	"crypto/tls"
+	weakrand "math/rand"
+	"net"
+	"net/http"
+	"sync/atomic"
+	"time"
+
+	"golang.org/x/net/http2"
+)
+
+// http2Listener wraps the listener to solve the following problems:
+// 1. server h2 natively without using h2c hack when listener handles tls connection but
+// don't return *tls.Conn
+// 2. graceful shutdown. the shutdown logic is copied from stdlib http.Server, it's an extra maintenance burden but
+// whatever, the shutdown logic maybe extracted to be used with h2c graceful shutdown. http2.Server supports graceful shutdown
+// sending GO_AWAY frame to connected clients, but doesn't track connection status. It requires explicit call of http2.ConfigureServer
+type http2Listener struct {
+	cnt uint64
+	net.Listener
+	server   *http.Server
+	h2server *http2.Server
+}
+
+type connectionStateConn interface {
+	net.Conn
+	ConnectionState() tls.ConnectionState
+}
+
+func (h *http2Listener) Accept() (net.Conn, error) {
+	for {
+		conn, err := h.Listener.Accept()
+		if err != nil {
+			return nil, err
+		}
+
+		if csc, ok := conn.(connectionStateConn); ok {
+			// *tls.Conn will return empty string because it's only populated after handshake is complete
+			if csc.ConnectionState().NegotiatedProtocol == http2.NextProtoTLS {
+				go h.serveHttp2(csc)
+				continue
+			}
+		}
+
+		return conn, nil
+	}
+}
+
+func (h *http2Listener) serveHttp2(csc connectionStateConn) {
+	atomic.AddUint64(&h.cnt, 1)
+	h.runHook(csc, http.StateNew)
+	defer func() {
+		csc.Close()
+		atomic.AddUint64(&h.cnt, ^uint64(0))
+		h.runHook(csc, http.StateClosed)
+	}()
+	h.h2server.ServeConn(csc, &http2.ServeConnOpts{
+		Context:    h.server.ConnContext(context.Background(), csc),
+		BaseConfig: h.server,
+		Handler:    h.server.Handler,
+	})
+}
+
+const shutdownPollIntervalMax = 500 * time.Millisecond
+
+func (h *http2Listener) Shutdown(ctx context.Context) error {
+	pollIntervalBase := time.Millisecond
+	nextPollInterval := func() time.Duration {
+		// Add 10% jitter.
+		//nolint:gosec
+		interval := pollIntervalBase + time.Duration(weakrand.Intn(int(pollIntervalBase/10)))
+		// Double and clamp for next time.
+		pollIntervalBase *= 2
+		if pollIntervalBase > shutdownPollIntervalMax {
+			pollIntervalBase = shutdownPollIntervalMax
+		}
+		return interval
+	}
+
+	timer := time.NewTimer(nextPollInterval())
+	defer timer.Stop()
+	for {
+		if atomic.LoadUint64(&h.cnt) == 0 {
+			return nil
+		}
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-timer.C:
+			timer.Reset(nextPollInterval())
+		}
+	}
+}
+
+func (h *http2Listener) runHook(conn net.Conn, state http.ConnState) {
+	if h.server.ConnState != nil {
+		h.server.ConnState(conn, state)
+	}
+}

modules/caddyhttp/invoke.go

@@ -0,0 +1,56 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyhttp
+
+import (
+	"fmt"
+	"net/http"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func init() {
+	caddy.RegisterModule(Invoke{})
+}
+
+// Invoke implements a handler that compiles and executes a
+// named route that was defined on the server.
+//
+// EXPERIMENTAL: Subject to change or removal.
+type Invoke struct {
+	// Name is the key of the named route to execute
+	Name string `json:"name,omitempty"`
+}
+
+// CaddyModule returns the Caddy module information.
+func (Invoke) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "http.handlers.invoke",
+		New: func() caddy.Module { return new(Invoke) },
+	}
+}
+
+func (invoke *Invoke) ServeHTTP(w http.ResponseWriter, r *http.Request, next Handler) error {
+	server := r.Context().Value(ServerCtxKey).(*Server)
+	if route, ok := server.NamedRoutes[invoke.Name]; ok {
+		return route.Compile(next).ServeHTTP(w, r)
+	}
+	return fmt.Errorf("invoke: route '%s' not found", invoke.Name)
+}
+
+// Interface guards
+var (
+	_ MiddlewareHandler = (*Invoke)(nil)
+)

modules/caddyhttp/ip_matchers.go

@@ -0,0 +1,344 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyhttp
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"net/netip"
+	"reflect"
+	"strings"
+
+	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	"github.com/google/cel-go/cel"
+	"github.com/google/cel-go/common/types/ref"
+	"go.uber.org/zap"
+)
+
+// MatchRemoteIP matches requests by the remote IP address,
+// i.e. the IP address of the direct connection to Caddy.
+type MatchRemoteIP struct {
+	// The IPs or CIDR ranges to match.
+	Ranges []string `json:"ranges,omitempty"`
+
+	// If true, prefer the first IP in the request's X-Forwarded-For
+	// header, if present, rather than the immediate peer's IP, as
+	// the reference IP against which to match. Note that it is easy
+	// to spoof request headers. Default: false
+	// DEPRECATED: This is insecure, MatchClientIP should be used instead.
+	Forwarded bool `json:"forwarded,omitempty"`
+
+	// cidrs and zones vars should aligned always in the same
+	// length and indexes for matching later
+	cidrs  []*netip.Prefix
+	zones  []string
+	logger *zap.Logger
+}
+
+// MatchClientIP matches requests by the client IP address,
+// i.e. the resolved address, considering trusted proxies.
+type MatchClientIP struct {
+	// The IPs or CIDR ranges to match.
+	Ranges []string `json:"ranges,omitempty"`
+
+	// cidrs and zones vars should aligned always in the same
+	// length and indexes for matching later
+	cidrs  []*netip.Prefix
+	zones  []string
+	logger *zap.Logger
+}
+
+func init() {
+	caddy.RegisterModule(MatchRemoteIP{})
+	caddy.RegisterModule(MatchClientIP{})
+}
+
+// CaddyModule returns the Caddy module information.
+func (MatchRemoteIP) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "http.matchers.remote_ip",
+		New: func() caddy.Module { return new(MatchRemoteIP) },
+	}
+}
+
+// UnmarshalCaddyfile implements caddyfile.Unmarshaler.
+func (m *MatchRemoteIP) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
+	for d.Next() {
+		for d.NextArg() {
+			if d.Val() == "forwarded" {
+				if len(m.Ranges) > 0 {
+					return d.Err("if used, 'forwarded' must be first argument")
+				}
+				m.Forwarded = true
+				continue
+			}
+			if d.Val() == "private_ranges" {
+				m.Ranges = append(m.Ranges, PrivateRangesCIDR()...)
+				continue
+			}
+			m.Ranges = append(m.Ranges, d.Val())
+		}
+		if d.NextBlock(0) {
+			return d.Err("malformed remote_ip matcher: blocks are not supported")
+		}
+	}
+	return nil
+}
+
+// CELLibrary produces options that expose this matcher for use in CEL
+// expression matchers.
+//
+// Example:
+//
+//	expression remote_ip('forwarded', '192.168.0.0/16', '172.16.0.0/12', '10.0.0.0/8')
+func (MatchRemoteIP) CELLibrary(ctx caddy.Context) (cel.Library, error) {
+	return CELMatcherImpl(
+		// name of the macro, this is the function name that users see when writing expressions.
+		"remote_ip",
+		// name of the function that the macro will be rewritten to call.
+		"remote_ip_match_request_list",
+		// internal data type of the MatchPath value.
+		[]*cel.Type{cel.ListType(cel.StringType)},
+		// function to convert a constant list of strings to a MatchPath instance.
+		func(data ref.Val) (RequestMatcher, error) {
+			refStringList := reflect.TypeOf([]string{})
+			strList, err := data.ConvertToNative(refStringList)
+			if err != nil {
+				return nil, err
+			}
+
+			m := MatchRemoteIP{}
+
+			for _, input := range strList.([]string) {
+				if input == "forwarded" {
+					if len(m.Ranges) > 0 {
+						return nil, errors.New("if used, 'forwarded' must be first argument")
+					}
+					m.Forwarded = true
+					continue
+				}
+				m.Ranges = append(m.Ranges, input)
+			}
+
+			err = m.Provision(ctx)
+			return m, err
+		},
+	)
+}
+
+// Provision parses m's IP ranges, either from IP or CIDR expressions.
+func (m *MatchRemoteIP) Provision(ctx caddy.Context) error {
+	m.logger = ctx.Logger()
+	cidrs, zones, err := provisionCidrsZonesFromRanges(m.Ranges)
+	if err != nil {
+		return err
+	}
+	m.cidrs = cidrs
+	m.zones = zones
+
+	if m.Forwarded {
+		m.logger.Warn("remote_ip's forwarded mode is deprecated; use the 'client_ip' matcher instead")
+	}
+
+	return nil
+}
+
+// Match returns true if r matches m.
+func (m MatchRemoteIP) Match(r *http.Request) bool {
+	address := r.RemoteAddr
+	if m.Forwarded {
+		if fwdFor := r.Header.Get("X-Forwarded-For"); fwdFor != "" {
+			address = strings.TrimSpace(strings.Split(fwdFor, ",")[0])
+		}
+	}
+	clientIP, zoneID, err := parseIPZoneFromString(address)
+	if err != nil {
+		m.logger.Error("getting remote IP", zap.Error(err))
+		return false
+	}
+	matches, zoneFilter := matchIPByCidrZones(clientIP, zoneID, m.cidrs, m.zones)
+	if !matches && !zoneFilter {
+		m.logger.Debug("zone ID from remote IP did not match", zap.String("zone", zoneID))
+	}
+	return matches
+}
+
+// CaddyModule returns the Caddy module information.
+func (MatchClientIP) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "http.matchers.client_ip",
+		New: func() caddy.Module { return new(MatchClientIP) },
+	}
+}
+
+// UnmarshalCaddyfile implements caddyfile.Unmarshaler.
+func (m *MatchClientIP) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
+	for d.Next() {
+		for d.NextArg() {
+			if d.Val() == "private_ranges" {
+				m.Ranges = append(m.Ranges, PrivateRangesCIDR()...)
+				continue
+			}
+			m.Ranges = append(m.Ranges, d.Val())
+		}
+		if d.NextBlock(0) {
+			return d.Err("malformed client_ip matcher: blocks are not supported")
+		}
+	}
+	return nil
+}
+
+// CELLibrary produces options that expose this matcher for use in CEL
+// expression matchers.
+//
+// Example:
+//
+//	expression client_ip('192.168.0.0/16', '172.16.0.0/12', '10.0.0.0/8')
+func (MatchClientIP) CELLibrary(ctx caddy.Context) (cel.Library, error) {
+	return CELMatcherImpl(
+		// name of the macro, this is the function name that users see when writing expressions.
+		"client_ip",
+		// name of the function that the macro will be rewritten to call.
+		"client_ip_match_request_list",
+		// internal data type of the MatchPath value.
+		[]*cel.Type{cel.ListType(cel.StringType)},
+		// function to convert a constant list of strings to a MatchPath instance.
+		func(data ref.Val) (RequestMatcher, error) {
+			refStringList := reflect.TypeOf([]string{})
+			strList, err := data.ConvertToNative(refStringList)
+			if err != nil {
+				return nil, err
+			}
+
+			m := MatchClientIP{
+				Ranges: strList.([]string),
+			}
+
+			err = m.Provision(ctx)
+			return m, err
+		},
+	)
+}
+
+// Provision parses m's IP ranges, either from IP or CIDR expressions.
+func (m *MatchClientIP) Provision(ctx caddy.Context) error {
+	m.logger = ctx.Logger()
+	cidrs, zones, err := provisionCidrsZonesFromRanges(m.Ranges)
+	if err != nil {
+		return err
+	}
+	m.cidrs = cidrs
+	m.zones = zones
+	return nil
+}
+
+// Match returns true if r matches m.
+func (m MatchClientIP) Match(r *http.Request) bool {
+	address := GetVar(r.Context(), ClientIPVarKey).(string)
+	clientIP, zoneID, err := parseIPZoneFromString(address)
+	if err != nil {
+		m.logger.Error("getting client IP", zap.Error(err))
+		return false
+	}
+	matches, zoneFilter := matchIPByCidrZones(clientIP, zoneID, m.cidrs, m.zones)
+	if !matches && !zoneFilter {
+		m.logger.Debug("zone ID from client IP did not match", zap.String("zone", zoneID))
+	}
+	return matches
+}
+
+func provisionCidrsZonesFromRanges(ranges []string) ([]*netip.Prefix, []string, error) {
+	cidrs := []*netip.Prefix{}
+	zones := []string{}
+	for _, str := range ranges {
+		// Exclude the zone_id from the IP
+		if strings.Contains(str, "%") {
+			split := strings.Split(str, "%")
+			str = split[0]
+			// write zone identifiers in m.zones for matching later
+			zones = append(zones, split[1])
+		} else {
+			zones = append(zones, "")
+		}
+		if strings.Contains(str, "/") {
+			ipNet, err := netip.ParsePrefix(str)
+			if err != nil {
+				return nil, nil, fmt.Errorf("parsing CIDR expression '%s': %v", str, err)
+			}
+			cidrs = append(cidrs, &ipNet)
+		} else {
+			ipAddr, err := netip.ParseAddr(str)
+			if err != nil {
+				return nil, nil, fmt.Errorf("invalid IP address: '%s': %v", str, err)
+			}
+			ipNew := netip.PrefixFrom(ipAddr, ipAddr.BitLen())
+			cidrs = append(cidrs, &ipNew)
+		}
+	}
+	return cidrs, zones, nil
+}
+
+func parseIPZoneFromString(address string) (netip.Addr, string, error) {
+	ipStr, _, err := net.SplitHostPort(address)
+	if err != nil {
+		ipStr = address // OK; probably didn't have a port
+	}
+
+	// Some IPv6-Adresses can contain zone identifiers at the end,
+	// which are separated with "%"
+	zoneID := ""
+	if strings.Contains(ipStr, "%") {
+		split := strings.Split(ipStr, "%")
+		ipStr = split[0]
+		zoneID = split[1]
+	}
+
+	ipAddr, err := netip.ParseAddr(ipStr)
+	if err != nil {
+		return netip.IPv4Unspecified(), "", err
+	}
+
+	return ipAddr, zoneID, nil
+}
+
+func matchIPByCidrZones(clientIP netip.Addr, zoneID string, cidrs []*netip.Prefix, zones []string) (bool, bool) {
+	zoneFilter := true
+	for i, ipRange := range cidrs {
+		if ipRange.Contains(clientIP) {
+			// Check if there are zone filters assigned and if they match.
+			if zones[i] == "" || zoneID == zones[i] {
+				return true, false
+			}
+			zoneFilter = false
+		}
+	}
+	return false, zoneFilter
+}
+
+// Interface guards
+var (
+	_ RequestMatcher        = (*MatchRemoteIP)(nil)
+	_ caddy.Provisioner     = (*MatchRemoteIP)(nil)
+	_ caddyfile.Unmarshaler = (*MatchRemoteIP)(nil)
+	_ CELLibraryProducer    = (*MatchRemoteIP)(nil)
+
+	_ RequestMatcher        = (*MatchClientIP)(nil)
+	_ caddy.Provisioner     = (*MatchClientIP)(nil)
+	_ caddyfile.Unmarshaler = (*MatchClientIP)(nil)
+	_ CELLibraryProducer    = (*MatchClientIP)(nil)
+)

modules/caddyhttp/logging.go

@@ -20,6 +20,7 @@ import (
 	"net/http"
 	"strings"
 
+	"github.com/caddyserver/caddy/v2"
 	"go.uber.org/zap"
 	"go.uber.org/zap/zapcore"
 )
@@ -139,6 +140,21 @@ func errLogValues(err error) (status int, msg string, fields []zapcore.Field) {
 	return
 }
 
-// Variable name used to indicate that this request
-// should be omitted from the access logs
-const SkipLogVar = "skip_log"
+// ExtraLogFields is a list of extra fields to log with every request.
+type ExtraLogFields struct {
+	fields []zapcore.Field
+}
+
+// Add adds a field to the list of extra fields to log.
+func (e *ExtraLogFields) Add(field zap.Field) {
+	e.fields = append(e.fields, field)
+}
+
+const (
+	// Variable name used to indicate that this request
+	// should be omitted from the access logs
+	SkipLogVar string = "skip_log"
+
+	// For adding additional fields to the access logs
+	ExtraLogFieldsCtxKey caddy.CtxKey = "extra_log_fields"
+)

modules/caddyhttp/marshalers.go

@@ -40,6 +40,7 @@ func (r LoggableHTTPRequest) MarshalLogObject(enc zapcore.ObjectEncoder) error {
 
 	enc.AddString("remote_ip", ip)
 	enc.AddString("remote_port", port)
+	enc.AddString("client_ip", GetVar(r.Context(), ClientIPVarKey).(string))
 	enc.AddString("proto", r.Proto)
 	enc.AddString("method", r.Method)
 	enc.AddString("host", r.Host)

modules/caddyhttp/matchers.go

@@ -20,12 +20,12 @@ import (
 	"fmt"
 	"net"
 	"net/http"
-	"net/netip"
 	"net/textproto"
 	"net/url"
 	"path"
 	"reflect"
 	"regexp"
+	"runtime"
 	"sort"
 	"strconv"
 	"strings"
@@ -35,7 +35,6 @@ import (
 	"github.com/google/cel-go/cel"
 	"github.com/google/cel-go/common/types"
 	"github.com/google/cel-go/common/types/ref"
-	"go.uber.org/zap"
 )
 
 type (
@@ -176,24 +175,6 @@ type (
 	// "http/2", "http/3", or minimum versions: "http/2+", etc.
 	MatchProtocol string
 
-	// MatchRemoteIP matches requests by client IP (or CIDR range).
-	MatchRemoteIP struct {
-		// The IPs or CIDR ranges to match.
-		Ranges []string `json:"ranges,omitempty"`
-
-		// If true, prefer the first IP in the request's X-Forwarded-For
-		// header, if present, rather than the immediate peer's IP, as
-		// the reference IP against which to match. Note that it is easy
-		// to spoof request headers. Default: false
-		Forwarded bool `json:"forwarded,omitempty"`
-
-		// cidrs and zones vars should aligned always in the same
-		// length and indexes for matching later
-		cidrs  []*netip.Prefix
-		zones  []string
-		logger *zap.Logger
-	}
-
 	// MatchNot matches requests by negating the results of its matcher
 	// sets. A single "not" matcher takes one or more matcher sets. Each
 	// matcher set is OR'ed; in other words, if any matcher set returns
@@ -229,7 +210,6 @@ func init() {
 	caddy.RegisterModule(MatchHeader{})
 	caddy.RegisterModule(MatchHeaderRE{})
 	caddy.RegisterModule(new(MatchProtocol))
-	caddy.RegisterModule(MatchRemoteIP{})
 	caddy.RegisterModule(MatchNot{})
 }
 
@@ -416,7 +396,9 @@ func (m MatchPath) Match(r *http.Request) bool {
 	// security risk (cry) if PHP files end up being served
 	// as static files, exposing the source code, instead of
 	// being matched by *.php to be treated as PHP scripts.
-	reqPath = strings.TrimRight(reqPath, ". ")
+	if runtime.GOOS == "windows" { // issue #5613
+		reqPath = strings.TrimRight(reqPath, ". ")
+	}
 
 	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)
 
@@ -1261,159 +1243,6 @@ func (m MatchNot) Match(r *http.Request) bool {
 	return true
 }
 
-// CaddyModule returns the Caddy module information.
-func (MatchRemoteIP) CaddyModule() caddy.ModuleInfo {
-	return caddy.ModuleInfo{
-		ID:  "http.matchers.remote_ip",
-		New: func() caddy.Module { return new(MatchRemoteIP) },
-	}
-}
-
-// UnmarshalCaddyfile implements caddyfile.Unmarshaler.
-func (m *MatchRemoteIP) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
-	for d.Next() {
-		for d.NextArg() {
-			if d.Val() == "forwarded" {
-				if len(m.Ranges) > 0 {
-					return d.Err("if used, 'forwarded' must be first argument")
-				}
-				m.Forwarded = true
-				continue
-			}
-			if d.Val() == "private_ranges" {
-				m.Ranges = append(m.Ranges, PrivateRangesCIDR()...)
-				continue
-			}
-			m.Ranges = append(m.Ranges, d.Val())
-		}
-		if d.NextBlock(0) {
-			return d.Err("malformed remote_ip matcher: blocks are not supported")
-		}
-	}
-	return nil
-}
-
-// CELLibrary produces options that expose this matcher for use in CEL
-// expression matchers.
-//
-// Example:
-//
-//	expression remote_ip('forwarded', '192.168.0.0/16', '172.16.0.0/12', '10.0.0.0/8')
-func (MatchRemoteIP) CELLibrary(ctx caddy.Context) (cel.Library, error) {
-	return CELMatcherImpl(
-		// name of the macro, this is the function name that users see when writing expressions.
-		"remote_ip",
-		// name of the function that the macro will be rewritten to call.
-		"remote_ip_match_request_list",
-		// internal data type of the MatchPath value.
-		[]*cel.Type{cel.ListType(cel.StringType)},
-		// function to convert a constant list of strings to a MatchPath instance.
-		func(data ref.Val) (RequestMatcher, error) {
-			refStringList := reflect.TypeOf([]string{})
-			strList, err := data.ConvertToNative(refStringList)
-			if err != nil {
-				return nil, err
-			}
-
-			m := MatchRemoteIP{}
-
-			for _, input := range strList.([]string) {
-				if input == "forwarded" {
-					if len(m.Ranges) > 0 {
-						return nil, errors.New("if used, 'forwarded' must be first argument")
-					}
-					m.Forwarded = true
-					continue
-				}
-				m.Ranges = append(m.Ranges, input)
-			}
-
-			err = m.Provision(ctx)
-			return m, err
-		},
-	)
-}
-
-// Provision parses m's IP ranges, either from IP or CIDR expressions.
-func (m *MatchRemoteIP) Provision(ctx caddy.Context) error {
-	m.logger = ctx.Logger()
-	for _, str := range m.Ranges {
-		// Exclude the zone_id from the IP
-		if strings.Contains(str, "%") {
-			split := strings.Split(str, "%")
-			str = split[0]
-			// write zone identifiers in m.zones for matching later
-			m.zones = append(m.zones, split[1])
-		} else {
-			m.zones = append(m.zones, "")
-		}
-		if strings.Contains(str, "/") {
-			ipNet, err := netip.ParsePrefix(str)
-			if err != nil {
-				return fmt.Errorf("parsing CIDR expression '%s': %v", str, err)
-			}
-			m.cidrs = append(m.cidrs, &ipNet)
-		} else {
-			ipAddr, err := netip.ParseAddr(str)
-			if err != nil {
-				return fmt.Errorf("invalid IP address: '%s': %v", str, err)
-			}
-			ipNew := netip.PrefixFrom(ipAddr, ipAddr.BitLen())
-			m.cidrs = append(m.cidrs, &ipNew)
-		}
-	}
-	return nil
-}
-
-func (m MatchRemoteIP) getClientIP(r *http.Request) (netip.Addr, string, error) {
-	remote := r.RemoteAddr
-	zoneID := ""
-	if m.Forwarded {
-		if fwdFor := r.Header.Get("X-Forwarded-For"); fwdFor != "" {
-			remote = strings.TrimSpace(strings.Split(fwdFor, ",")[0])
-		}
-	}
-	ipStr, _, err := net.SplitHostPort(remote)
-	if err != nil {
-		ipStr = remote // OK; probably didn't have a port
-	}
-	// Some IPv6-Adresses can contain zone identifiers at the end,
-	// which are separated with "%"
-	if strings.Contains(ipStr, "%") {
-		split := strings.Split(ipStr, "%")
-		ipStr = split[0]
-		zoneID = split[1]
-	}
-	ipAddr, err := netip.ParseAddr(ipStr)
-	if err != nil {
-		return netip.IPv4Unspecified(), "", err
-	}
-	return ipAddr, zoneID, nil
-}
-
-// Match returns true if r matches m.
-func (m MatchRemoteIP) Match(r *http.Request) bool {
-	clientIP, zoneID, err := m.getClientIP(r)
-	if err != nil {
-		m.logger.Error("getting client IP", zap.Error(err))
-		return false
-	}
-	zoneFilter := true
-	for i, ipRange := range m.cidrs {
-		if ipRange.Contains(clientIP) {
-			// Check if there are zone filters assigned and if they match.
-			if m.zones[i] == "" || zoneID == m.zones[i] {
-				return true
-			}
-			zoneFilter = false
-		}
-	}
-	if !zoneFilter {
-		m.logger.Debug("zone ID from remote did not match", zap.String("zone", zoneID))
-	}
-	return false
-}
-
 // MatchRegexp is an embedable type for matching
 // using regular expressions. It adds placeholders
 // to the request's replacer.
@@ -1588,8 +1417,6 @@ var (
 	_ RequestMatcher    = (*MatchHeaderRE)(nil)
 	_ caddy.Provisioner = (*MatchHeaderRE)(nil)
 	_ RequestMatcher    = (*MatchProtocol)(nil)
-	_ RequestMatcher    = (*MatchRemoteIP)(nil)
-	_ caddy.Provisioner = (*MatchRemoteIP)(nil)
 	_ RequestMatcher    = (*MatchNot)(nil)
 	_ caddy.Provisioner = (*MatchNot)(nil)
 	_ caddy.Provisioner = (*MatchRegexp)(nil)
@@ -1602,7 +1429,6 @@ var (
 	_ caddyfile.Unmarshaler = (*MatchHeader)(nil)
 	_ caddyfile.Unmarshaler = (*MatchHeaderRE)(nil)
 	_ caddyfile.Unmarshaler = (*MatchProtocol)(nil)
-	_ caddyfile.Unmarshaler = (*MatchRemoteIP)(nil)
 	_ caddyfile.Unmarshaler = (*VarsMatcher)(nil)
 	_ caddyfile.Unmarshaler = (*MatchVarsRE)(nil)
 
@@ -1614,7 +1440,6 @@ var (
 	_ CELLibraryProducer = (*MatchHeader)(nil)
 	_ CELLibraryProducer = (*MatchHeaderRE)(nil)
 	_ CELLibraryProducer = (*MatchProtocol)(nil)
-	_ CELLibraryProducer = (*MatchRemoteIP)(nil)
 	// _ CELLibraryProducer = (*VarsMatcher)(nil)
 	// _ CELLibraryProducer = (*MatchVarsRE)(nil)
 

modules/caddyhttp/matchers_test.go

@@ -21,6 +21,7 @@ import (
 	"net/http/httptest"
 	"net/url"
 	"os"
+	"runtime"
 	"testing"
 
 	"github.com/caddyserver/caddy/v2"
@@ -253,11 +254,6 @@ func TestPathMatcher(t *testing.T) {
 			input:  "/FOOOO",
 			expect: true,
 		},
-		{
-			match:  MatchPath{"*.php"},
-			input:  "/foo/index.php. .",
-			expect: true,
-		},
 		{
 			match:  MatchPath{"/foo/bar.txt"},
 			input:  "/foo/BAR.txt",
@@ -435,8 +431,10 @@ func TestPathMatcher(t *testing.T) {
 
 func TestPathMatcherWindows(t *testing.T) {
 	// only Windows has this bug where it will ignore
-	// trailing dots and spaces in a filename, but we
-	// test for it on all platforms to be more consistent
+	// trailing dots and spaces in a filename
+	if runtime.GOOS != "windows" {
+		return
+	}
 
 	req := &http.Request{URL: &url.URL{Path: "/index.php . . .."}}
 	repl := caddy.NewReplacer()

modules/caddyhttp/proxyprotocol/listenerwrapper.go

@@ -0,0 +1,68 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package proxyprotocol
+
+import (
+	"fmt"
+	"net"
+	"time"
+
+	"github.com/caddyserver/caddy/v2"
+	"github.com/mastercactapus/proxyprotocol"
+)
+
+// ListenerWrapper provides PROXY protocol support to Caddy by implementing
+// the caddy.ListenerWrapper interface. It must be loaded before the `tls` listener.
+//
+// Credit goes to https://github.com/mastercactapus/caddy2-proxyprotocol for having
+// initially implemented this as a plugin.
+type ListenerWrapper struct {
+	// Timeout specifies an optional maximum time for
+	// the PROXY header to be received.
+	// If zero, timeout is disabled. Default is 5s.
+	Timeout caddy.Duration `json:"timeout,omitempty"`
+
+	// Allow is an optional list of CIDR ranges to
+	// allow/require PROXY headers from.
+	Allow []string `json:"allow,omitempty"`
+
+	rules []proxyprotocol.Rule
+}
+
+// Provision sets up the listener wrapper.
+func (pp *ListenerWrapper) Provision(ctx caddy.Context) error {
+	rules := make([]proxyprotocol.Rule, 0, len(pp.Allow))
+	for _, s := range pp.Allow {
+		_, n, err := net.ParseCIDR(s)
+		if err != nil {
+			return fmt.Errorf("invalid subnet '%s': %w", s, err)
+		}
+		rules = append(rules, proxyprotocol.Rule{
+			Timeout: time.Duration(pp.Timeout),
+			Subnet:  n,
+		})
+	}
+
+	pp.rules = rules
+
+	return nil
+}
+
+// WrapListener adds PROXY protocol support to the listener.
+func (pp *ListenerWrapper) WrapListener(l net.Listener) net.Listener {
+	pl := proxyprotocol.NewListener(l, time.Duration(pp.Timeout))
+	pl.SetFilter(pp.rules)
+	return pl
+}