metrics: Always track method label in uppercase (#3742)

* metrics: Always track method label in uppercase

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Just use strings.ToUpper for clarity

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

.github/CONTRIBUTING.md

@@ -0,0 +1,184 @@
+Contributing to Caddy
+=====================
+
+Welcome! Thank you for choosing to be a part of our community. Caddy wouldn't be great without your involvement!
+
+For starters, we invite you to join [the Caddy forum](https://caddy.community) where you can hang out with other Caddy users and developers.
+
+## Common Tasks
+
+- [Contributing code](#contributing-code)
+- [Writing a Caddy module](#writing-a-caddy-module)
+- [Asking or answering questions for help using Caddy](#getting-help-using-caddy)
+- [Reporting a bug](#reporting-bugs)
+- [Suggesting an enhancement or a new feature](#suggesting-features)
+- [Improving documentation](#improving-documentation)
+
+Other menu items:
+
+- [Values](#values)
+- [Coordinated Disclosure](#coordinated-disclosure)
+- [Thank You](#thank-you)
+
+
+### Contributing code
+
+You can have a huge impact on the project by helping with its code. To contribute code to Caddy, open a [pull request](https://github.com/caddyserver/caddy/pulls) (PR). If you're new to our community, that's okay: **we gladly welcome pull requests from anyone, regardless of your native language or coding experience.** You can get familiar with Caddy's code base by using [code search at Sourcegraph](https://sourcegraph.com/github.com/caddyserver/caddy/-/search).
+
+We hold contributions to a high standard for quality :bowtie:, so don't be surprised if we ask for revisions—even if it seems small or insignificant. Please don't take it personally. :blue_heart: If your change is on the right track, we can guide you to make it mergable.
+
+Here are some of the expectations we have of contributors:
+
+- **Open an issue to propose your change first.** This way we can avoid confusion, coordinate what everyone is working on, and ensure that any changes are in-line with the project's goals and the best interests of its users. We can also discuss the best possible implementation. If there's already an issue about it, comment on the existing issue to claim it.
+
+- **Keep pull requests small.** Smaller PRs are more likely to be merged because they are easier to review! We might ask you to break up large PRs into smaller ones. [An example of what we want to avoid.](https://twitter.com/iamdevloper/status/397664295875805184)
+
+- **Keep related commits together in a PR.** We do want pull requests to be small, but you should also keep multiple related commits in the same PR if they rely on each other.
+
+- **Write tests.** Tests are essential! Written properly, they ensure your change works, and that other changes in the future won't break your change. CI checks should pass.
+
+- **Benchmarks should be included for optimizations.** Optimizations sometimes make code harder to read or have changes that are less than obvious. They should be proven with benchmarks or profiling.
+
+- **[Squash](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html) insignificant commits.** Every commit should be significant. Commits which merely rewrite a comment or fix a typo can be combined into another commit that has more substance. Interactive rebase can do this, or a simpler way is `git reset --soft <diverging-commit>` then `git commit -s`.
+
+- **Own your contributions.** Caddy is a growing project, and it's much better when individual contributors help maintain their change after it is merged.
+
+- **Use comments properly.** We expect good godoc comments for package-level functions, types, and values. Comments are also useful whenever the purpose for a line of code is not obvious.
+
+We often grant [collaborator status](#collaborator-instructions) to contributors who author one or more significant, high-quality PRs that are merged into the code base!
+
+
+#### HOW TO MAKE A PULL REQUEST TO CADDY
+
+Contributing to Go projects on GitHub is fun and easy. We recommend the following workflow:
+
+1. [Fork this repo](https://github.com/caddyserver/caddy). This makes a copy of the code you can write to.
+
+2. If you don't already have this repo (caddyserver/caddy.git) repo on your computer, get it with `go get github.com/caddyserver/caddy/v2`.
+
+3. Tell git that it can push the caddyserver/caddy.git repo to your fork by adding a remote: `git remote add myfork https://github.com/<your-username>/caddy.git`
+
+4. Make your changes in the caddyserver/caddy.git repo on your computer.
+
+5. Push your changes to your fork: `git push myfork`
+
+6. [Create a pull request](https://github.com/caddyserver/caddy/pull/new/master) to merge your changes into caddyserver/caddy @ master. (Click "compare across forks" and change the head fork.)
+
+This workflow is nice because you don't have to change import paths. You can get fancier by using different branches if you want.
+
+
+### Writing a Caddy module
+
+Caddy can do more with modules! Anyone can write one. Caddy modules are Go libraries that get compiled into Caddy, extending its feature set. They can add directives to the Caddyfile, add new configuration adapters, and even implement new server types (e.g. HTTP, DNS).
+
+[Learn how to write a module here](https://caddyserver.com/docs/extending-caddy). You should also share and discuss your module idea [on the forums](https://caddy.community) to have people test it out. We don't use the Caddy issue tracker for third-party modules.
+
+
+### Getting help using Caddy
+
+If you have a question about using Caddy, [ask on our forum](https://caddy.community)! There will be more people there who can help you than just the Caddy developers who follow our issue tracker. Issues are not the place for usage questions.
+
+Many people on the forums could benefit from your experience and expertise, too. Once you've been helped, consider giving back by answering other people's questions and participating in other discussions.
+
+
+### Reporting bugs
+
+Like every software, Caddy has its flaws. If you find one, [search the issues](https://github.com/caddyserver/caddy/issues) to see if it has already been reported. If not, [open a new issue](https://github.com/caddyserver/caddy/issues/new) and describe the bug, and somebody will look into it! (This repository is only for Caddy and its standard modules.)
+
+**You can help stop bugs in their tracks!** Speed up the patch by identifying the bug in the code. This can sometimes be done by adding `fmt.Println()` statements (or similar) in relevant code paths to narrow down where the problem may be. It's a good way to [introduce yourself to the Go language](https://tour.golang.org), too.
+
+Please follow the issue template so we have all the needed information. Unredacted&mdash;yes, actual values matter. We need to be able to repeat the bug using your instructions. Please simplify the issue as much as possible. The burden is on you to convince us that it is actually a bug in Caddy. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you!
+
+We suggest reading [How to Report Bugs Effectively](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html).
+
+Please be kind. :smile: Remember that Caddy comes at no cost to you, and you're getting free support when we fix your issues. If we helped you, please consider helping someone else!
+
+#### Bug reporting expectations
+
+Maintainers---or more generally, developers---need three things to act on bugs:
+
+1. To agree or be convinced that it's a bug (reporter's responsibility).
+	- A bug is undesired or surprising behavior which violates documentation or the spec.
+
+2. To be able to understand what is happening (mostly reporter's responsibility).
+	- If the reporter can provide satisfactory instructions such that a developer can reproduce the bug, the developer will likely be able to understand the bug, write a test case, and implement a fix.
+	- Otherwise, the burden is on the reporter to test possible solutions. This is discouraged because it loosens the feedback loop, slows down debugging efforts, obscures the true nature of the problem from the developers, and is unlikely to result in new test cases.
+
+3. A solution, or ideas toward a solution (mostly maintainer's responsibility).
+	- Sometimes the best solution is a documentation change.
+	- Usually the developers have the best domain knowledge for inventing a solution, but reporters may have ideas or preferences for how they would like the software to work.
+	- Security, correctness, and project goals/vision all take priority over a user's preferences.
+	- It's simply good business to yield a solution that satisfies the users, and it's even better business to leave them impressed.
+
+Thus, at the very least, the reporter is expected to:
+
+1. Convince the reader that it's a bug (if it's not obvious).
+2. Reduce the problem down to the minimum specific steps required to reproduce it.
+
+The maintainer is usually able to do the rest; but of course the reporter may invest additional effort to speed up the process.
+
+
+
+### Suggesting features
+
+First, [search to see if your feature has already been requested](https://github.com/caddyserver/caddy/issues). If it has, you can add a :+1: reaction to vote for it. If your feature idea is new, open an issue to request the feature. Please describe your idea thoroughly so that we know how to implement it! Really vague requests may not be helpful or actionable and, without clarification, will have to be closed.
+
+While we really do value your requests and implement many of them, not all features are a good fit for Caddy. Most of those [make good modules](#writing-a-caddy-module), which can be made by anyone! But if a feature is not in the best interest of the Caddy project or its users in general, we may politely decline to implement it into Caddy core.
+
+
+### Improving documentation
+
+Caddy's documentation is available at [https://caddyserver.com/docs](https://caddyserver.com/docs) and its source is in the [website repo](https://github.com/caddyserver/website). If you would like to make a fix to the docs, please submit an issue there describing the change to make.
+
+Note that third-party module documentation is not hosted by the Caddy website, other than basic usage examples. They are managed by the individual module authors, and you will have to contact them to change their documentation.
+
+
+
+## Collaborator Instructions
+
+Collaborators have push rights to the repository. We grant this permission after one or more successful, high-quality PRs are merged! We thank them for their help.The expectations we have of collaborators are:
+
+- **Help review pull requests.** Be meticulous, but also kind. We love our contributors, but we critique the contribution to make it better. Multiple, thorough reviews make for the best contributions! Here are some questions to consider:
+	- Can the change be made more elegant?
+	- Is this a maintenance burden?
+	- What assumptions does the code make?
+	- Is it well-tested?
+	- Is the change a good fit for the project?
+	- Does it actually fix the problem or is it creating a special case instead?
+	- Does the change incur any new dependencies? (Avoid these!)
+
+- **Answer issues.** If every collaborator helped out with issues, we could count the number of open issues on two hands. This means getting involved in the discussion, investigating the code, and yes, debugging it. It's fun. Really! :smile: Please, please help with open issues. Granted, some issues need to be done before others. And of course some are larger than others: you don't have to do it all yourself. Work with other collaborators as a team!
+
+- **Do not merge pull requests until they have been approved by one or two other collaborators.** If a project owner approves the PR, it can be merged (as long as the conversation has finished too).
+
+- **Prefer squashed commits over a messy merge.** If there are many little commits, please [squash the commits](https://stackoverflow.com/a/11732910/1048862) so we don't clutter the commit history.
+
+- **Don't accept new dependencies lightly.** Dependencies can make the world crash and burn, but they are sometimes necessary. Choose carefully. Extremely small dependencies (a few lines of code) can be inlined. The rest may not be needed. For those that are, Caddy uses [go modules](https://github.com/golang/go/wiki/Modules). All external dependencies must be installed as modules, and _Caddy must not export any types defined by those dependencies_. Check this diligently!
+
+- **Be extra careful in some areas of the code.** There are some critical areas in the Caddy code base that we review extra meticulously: the `caddyhttp` and `caddytls` packages especially.
+
+- **Make sure tests test the actual thing.** Double-check that the tests fail without the change, and pass with it. It's important that they assert what they're purported to assert.
+
+- **Recommended reading**
+	- [CodeReviewComments](https://github.com/golang/go/wiki/CodeReviewComments) for an idea of what we look for in good, clean Go code
+	- [Linus Torvalds describes a good commit message](https://gist.github.com/matthewhudson/1475276)
+	- [Best Practices for Maintainers](https://opensource.guide/best-practices/)
+	- [Shrinking Code Review](https://alexgaynor.net/2015/dec/29/shrinking-code-review/)
+
+
+
+## Values
+
+- A person is always more important than code. People don't like being handled "efficiently". But we can still process issues and pull requests efficiently while being kind, patient, and considerate.
+
+- The ends justify the means, if the means are good. A good tree won't produce bad fruit. But if we cut corners or are hasty in our process, the end result will not be good.
+
+
+## Security Policy
+
+If you think you've found a security vulnerability, please refer to our [Security Policy](https://github.com/caddyserver/caddy/security/policy) document.
+
+
+## Thank you
+
+Thanks for your help! Caddy would not be what it is today without your contributions.

.github/SECURITY.md

@@ -0,0 +1,27 @@
+# Security Policy
+
+The Caddy project would like to make sure that it stays on top of all practically-exploitable vulnerabilities.
+
+Some security problems are more the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please report only vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing or BGP hijacks a vulnerability in the Caddy web server).
+
+Please note that we consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems.
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x     | :white_check_mark: |
+| 1.x     | :white_check_mark: (EOL 1 Oct 2020) |
+| < 1.x   | :x:                |
+
+## Reporting a Vulnerability
+
+Please email Matt Holt (the author) directly: matt [at] ardanlabs [dot com].
+
+We'll need enough information to verify the bug and make a patch. It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, resources permitting. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. Thank you for understanding.
+
+Please also understand that due to our nature as an open source project, we do not have a budget to award security bounties. We can only thank you.
+
+If your report is valid and a patch is released, we will not reveal your identity by default. If you wish to be credited, please give us the name to use.
+
+Thanks for responsibly helping Caddy&mdash;and thousands of websites&mdash;be more secure!

.github/workflows/ci.yml

@@ -1,13 +1,13 @@
 # Used as inspiration: https://github.com/mvdan/github-actions-golang
 
-name: Cross-Platform
+name: Tests
 
 on:
   push:
-    branches: 
+    branches:
       - master
   pull_request:
-    branches: 
+    branches:
       - master
 
 jobs:
@@ -17,7 +17,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ ubuntu-latest, macos-latest, windows-latest ]
-        go-version: [ 1.14.x ]
+        go: [ '1.14', '1.15' ]
 
         # Set some variables per OS, usable via ${{ matrix.VAR }}
         # CADDY_BIN_PATH: the path to the compiled Caddy binary, for artifact publishing
@@ -39,9 +39,9 @@ jobs:
 
     steps:
     - name: Install Go
-      uses: actions/setup-go@v1
+      uses: actions/setup-go@v2
       with:
-        go-version: ${{ matrix.go-version }}
+        go-version: ${{ matrix.go }}
 
     - name: Checkout code
       uses: actions/checkout@v2
@@ -66,6 +66,15 @@ jobs:
         env
         # Calculate the short SHA1 hash of the git commit
         echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
+        echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+    - name: Cache the build cache
+      uses: actions/cache@v2
+      with:
+        path: ${{ steps.vars.outputs.go_cache }}
+        key: ${{ runner.os }}-${{ matrix.go }}-go-ci-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-${{ matrix.go }}-go-ci
 
     - name: Get dependencies
       run: |
@@ -77,12 +86,12 @@ jobs:
       env:
         CGO_ENABLED: 0
       run: |
-        go build -trimpath -a -ldflags="-w -s" -v
+        go build -trimpath -ldflags="-w -s" -v
 
     - name: Publish Build Artifact
       uses: actions/upload-artifact@v1
       with:
-        name: caddy_v2_${{ runner.os }}_${{ steps.vars.outputs.short_sha }}
+        name: caddy_${{ runner.os }}_go${{ matrix.go }}_${{ steps.vars.outputs.short_sha }}
         path: ${{ matrix.CADDY_BIN_PATH }}
 
     # Commented bits below were useful to allow the job to continue
@@ -111,6 +120,34 @@ jobs:
     #     echo "step_test ${{ steps.step_test.outputs.status }}\n"
     #     exit 1
 
+  s390x-test:
+    name: test (s390x on IBM Z)
+    runs-on: ubuntu-latest
+    if: github.event.pull_request.head.repo.full_name == github.repository
+    continue-on-error: true  # August 2020: s390x VM is down due to weather and power issues
+    steps:
+      - name: Checkout code into the Go module directory
+        uses: actions/checkout@v2
+      - name: Run Tests
+        run: |
+          mkdir -p ~/.ssh && echo -e "${SSH_KEY//_/\\n}" > ~/.ssh/id_ecdsa && chmod og-rwx ~/.ssh/id_ecdsa
+
+          # short sha is enough?
+          short_sha=$(git rev-parse --short HEAD)
+
+          # The environment is fresh, so there's no point in keeping accepting and adding the key.
+          rsync -arz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress --delete --exclude '.git' . caddy-ci@ci-s390x.caddyserver.com:/var/tmp/"$short_sha"
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -t caddy-ci@ci-s390x.caddyserver.com "cd /var/tmp/$short_sha; CGO_ENABLED=0 /usr/local/go/bin/go test -v ./..."
+          test_result=$?
+
+          # There's no need leaving the files around
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null caddy-ci@ci-s390x.caddyserver.com "rm -rf /var/tmp/'$short_sha'"
+
+          echo "Test exit code: $test_result"
+          exit $test_result
+        env:
+          SSH_KEY: ${{ secrets.S390X_SSH_KEY }}
+
   # From https://github.com/reviewdog/action-golangci-lint
   golangci-lint:
     name: runner / golangci-lint
@@ -124,3 +161,15 @@ jobs:
         # uses: docker://reviewdog/action-golangci-lint:v1 # pre-build docker image
         with:
           github_token: ${{ secrets.github_token }}
+
+  goreleaser-check:
+    runs-on: ubuntu-latest
+    steps:
+      - name: checkout
+        uses: actions/checkout@v2
+      - uses: goreleaser/goreleaser-action@v2
+        with:
+          version: latest
+          args: check
+        env:
+          TAG: ${{ steps.vars.outputs.version_tag }}

.github/workflows/cross-build.yml

@@ -0,0 +1,60 @@
+name: Cross-Build
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  cross-build-test:
+    strategy:
+      fail-fast: false
+      matrix:
+        goos: ['android', 'linux', 'solaris', 'illumos', 'dragonfly', 'freebsd', 'openbsd', 'plan9', 'windows', 'darwin', 'netbsd']
+        go: [ '1.14', '1.15' ]
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: ${{ matrix.go }}
+
+      - name: Print Go version and environment
+        id: vars
+        run: |
+          printf "Using go at: $(which go)\n"
+          printf "Go version: $(go version)\n"
+          printf "\n\nGo environment:\n\n"
+          go env
+          printf "\n\nSystem environment:\n\n"
+          env
+          echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+      - name: Cache the build cache
+        uses: actions/cache@v2
+        with:
+          path: ${{ steps.vars.outputs.go_cache }}
+          key: cross-build-go${{ matrix.go }}-${{ matrix.goos }}-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            cross-build-go${{ matrix.go }}-${{ matrix.goos }}
+
+      - name: Checkout code into the Go module directory
+        uses: actions/checkout@v2
+
+      - name: Run Build
+        env:
+          CGO_ENABLED: 0
+          GOOS: ${{ matrix.goos }}
+        shell: bash
+        continue-on-error: true
+        working-directory: ./cmd/caddy
+        run: |
+          GOOS=$GOOS go build -trimpath -o caddy-"$GOOS"-amd64 2> /dev/null
+          if [ $? -ne 0 ]; then
+            echo "::warning ::$GOOS Build Failed"
+            exit 0
+          fi

.github/workflows/fuzzing.yml

@@ -12,14 +12,14 @@ jobs:
     strategy:
       matrix:
         os: [ ubuntu-latest ]
-        go-version: [ 1.14.x ]
+        go: [ '1.14' ]
     runs-on: ${{ matrix.os }}
 
     steps:
     - name: Install Go
-      uses: actions/setup-go@v1
+      uses: actions/setup-go@v2
       with:
-        go-version: ${{ matrix.go-version }}
+        go-version: ${{ matrix.go }}
 
     - name: Checkout code
       uses: actions/checkout@v2
@@ -48,14 +48,12 @@ jobs:
 
         declare -A fuzzers_funcs=(\
           ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="FuzzParseAddress" \
-          ["./caddyconfig/caddyfile/parse_fuzz.go"]="FuzzParseCaddyfile" \
           ["./listeners_fuzz.go"]="FuzzParseNetworkAddress" \
           ["./replacer_fuzz.go"]="FuzzReplacer" \
         )
 
         declare -A fuzzers_targets=(\
           ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="parse-address" \
-          ["./caddyconfig/caddyfile/parse_fuzz.go"]="parse-caddyfile" \
           ["./listeners_fuzz.go"]="parse-network-address" \
           ["./replacer_fuzz.go"]="replacer" \
         )

.github/workflows/release.yml

@@ -11,14 +11,14 @@ jobs:
     strategy:
       matrix:
         os: [ ubuntu-latest ]
-        go-version: [ 1.14.x ]
+        go: [ '1.15' ]
     runs-on: ${{ matrix.os }}
 
     steps:
     - name: Install Go
-      uses: actions/setup-go@v1
+      uses: actions/setup-go@v2
       with:
-        go-version: ${{ matrix.go-version }}
+        go-version: ${{ matrix.go }}
 
     - name: Checkout code
       uses: actions/checkout@v2
@@ -27,6 +27,7 @@ jobs:
     - name: Unshallowify the repo clone
       run: git fetch --prune --unshallow
 
+    # https://github.community/t5/GitHub-Actions/How-to-get-just-the-tag-name/m-p/32167/highlight/true#M1027
     - name: Print Go version and environment
       id: vars
       run: |
@@ -36,18 +37,47 @@ jobs:
         go env
         printf "\n\nSystem environment:\n\n"
         env
-    
-    # https://github.community/t5/GitHub-Actions/How-to-get-just-the-tag-name/m-p/32167/highlight/true#M1027
-    - name: Get the version
-      id: get_version
-      run: echo "::set-output name=version_tag::${GITHUB_REF/refs\/tags\//}"
+        echo "::set-output name=version_tag::${GITHUB_REF/refs\/tags\//}"
+        echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
+        echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+        # Parse semver
+        TAG=${GITHUB_REF/refs\/tags\//}
+        SEMVER_RE='[^0-9]*\([0-9]*\)[.]\([0-9]*\)[.]\([0-9]*\)\([0-9A-Za-z\.-]*\)'
+        TAG_MAJOR=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\1#"`
+        TAG_MINOR=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\2#"`
+        TAG_PATCH=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\3#"`
+        TAG_SPECIAL=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\4#"`
+        echo "::set-output name=tag_major::${TAG_MAJOR}"
+        echo "::set-output name=tag_minor::${TAG_MINOR}"
+        echo "::set-output name=tag_patch::${TAG_PATCH}"
+        echo "::set-output name=tag_special::${TAG_SPECIAL}"
+
+    - name: Cache the build cache
+      uses: actions/cache@v2
+      with:
+        path: ${{ steps.vars.outputs.go_cache }}
+        key: ${{ runner.os }}-go${{ matrix.go }}-release-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-go${{ matrix.go }}-release
 
     # GoReleaser will take care of publishing those artifacts into the release
     - name: Run GoReleaser
-      uses: goreleaser/goreleaser-action@v1
+      uses: goreleaser/goreleaser-action@v2
       with:
         version: latest
         args: release --rm-dist
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        TAG: ${{ steps.get_version.outputs.version_tag }}
+        TAG: ${{ steps.vars.outputs.version_tag }}
+
+    # Only publish on non-special tags (e.g. non-beta)
+    - name: Publish .deb to Gemfury
+      if: ${{ steps.vars.outputs.tag_special == '' }}
+      env:
+        GEMFURY_PUSH_TOKEN: ${{ secrets.GEMFURY_PUSH_TOKEN }}
+      run: |
+        for filename in dist/*.deb; do
+          curl -F package=@"$filename" https://${GEMFURY_PUSH_TOKEN}:@push.fury.io/caddy/
+        done
+

.github/workflows/release_published.yml

@@ -0,0 +1,34 @@
+name: Release Published
+
+# Event payload: https://developer.github.com/webhooks/event-payloads/#release
+on:
+  release:
+    types: [published]
+
+jobs:
+  release:
+    name: Release Published
+    strategy:
+      matrix:
+        os: [ ubuntu-latest ]
+    runs-on: ${{ matrix.os }}
+
+    steps:
+
+    # See https://github.com/peter-evans/repository-dispatch
+    - name: Trigger event on caddyserver/dist
+      uses: peter-evans/repository-dispatch@v1
+      with:
+        token: ${{ secrets.REPO_DISPATCH_TOKEN }}
+        repository: caddyserver/dist
+        event-type: release-tagged
+        client-payload: '{"tag": "${{ github.event.release.tag_name }}"}'
+
+    - name: Trigger event on caddyserver/caddy-docker
+      uses: peter-evans/repository-dispatch@v1
+      with:
+        token: ${{ secrets.REPO_DISPATCH_TOKEN }}
+        repository: caddyserver/caddy-docker
+        event-type: release-tagged
+        client-payload: '{"tag": "${{ github.event.release.tag_name }}"}'
+

.gitignore

@@ -20,4 +20,4 @@ vendor
 # goreleaser artifacts
 dist
 caddy-build
-caddy-dist
+caddy-dist

.goreleaser.yml

@@ -1,14 +1,19 @@
 before:
   hooks:
+    # The build is done in this particular way to build Caddy in a designated directory named in .gitignore.
+    # This is so we can run goreleaser on tag without Git complaining of being dirty. The main.go in cmd/caddy directory 
+    # cannot be built within that directory due to changes necessary for the build causing Git to be dirty, which
+    # subsequently causes gorleaser to refuse running.
     - mkdir -p caddy-build
     - cp cmd/caddy/main.go caddy-build/main.go
     - cp ./go.mod caddy-build/go.mod
-    - sed -i.bkp s/github.com\/caddyserver\/caddy\/v2/caddy/g ./caddy-build/go.mod
+    - sed -i.bkp 's|github.com/caddyserver/caddy/v2|caddy|g' ./caddy-build/go.mod
     # GoReleaser doesn't seem to offer {{.Tag}} at this stage, so we have to embed it into the env
     # so we run: TAG=$(git describe --abbrev=0) goreleaser release --rm-dist --skip-publish --skip-validate
     - go mod edit -require=github.com/caddyserver/caddy/v2@{{.Env.TAG}} ./caddy-build/go.mod
     - git clone --depth 1 https://github.com/caddyserver/dist caddy-dist
     - go mod download
+
 builds:
 - env:
   - CGO_ENABLED=0
@@ -25,16 +30,35 @@ builds:
   - amd64
   - arm
   - arm64
+  - s390x
+  - ppc64le
   goarm:
+  - 5
   - 6
   - 7
   ignore:
     - goos: darwin
       goarch: arm
+    - goos: darwin
+      goarch: ppc64le
+    - goos: darwin
+      goarch: s390x
+    - goos: windows
+      goarch: ppc64le
+    - goos: windows
+      goarch: s390x
+    - goos: freebsd
+      goarch: ppc64le
+    - goos: freebsd
+      goarch: s390x
+    - goos: freebsd
+      goarch: arm
+      goarm: 5
   flags:
   - -trimpath
   ldflags:
   - -s -w
+
 archives:
   - format_overrides:
       - goos: windows
@@ -43,12 +67,44 @@ archives:
       darwin: mac
 checksum:
   algorithm: sha512
+
+nfpms:
+  - id: default
+    package_name: caddy
+
+    vendor: Light Code Labs
+    homepage: https://caddyserver.com
+    maintainer: Matthew Holt <mholt@users.noreply.github.com>
+    description: |
+      Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go
+    license: Apache 2.0
+
+    formats:
+      - deb
+      # - rpm
+
+    bindir: /usr/bin
+    files:
+      ./caddy-dist/init/caddy.service: /lib/systemd/system/caddy.service
+      ./caddy-dist/init/caddy-api.service: /lib/systemd/system/caddy-api.service
+      ./caddy-dist/welcome/index.html: /usr/share/caddy/index.html
+      ./caddy-dist/scripts/completions/bash-completion: /etc/bash_completion.d/caddy
+    config_files:
+      ./caddy-dist/config/Caddyfile: /etc/caddy/Caddyfile
+
+    scripts:
+      postinstall: ./caddy-dist/scripts/postinstall.sh
+      preremove: ./caddy-dist/scripts/preremove.sh
+      postremove: ./caddy-dist/scripts/postremove.sh
+
+
 release:
   github:
     owner: caddyserver
     name: caddy
   draft: true
   prerelease: auto
+
 changelog:
   sort: asc
   filters:
@@ -56,5 +112,6 @@ changelog:
     - '^chore:'
     - '^ci:'
     - '^docs?:'
+    - '^readme:'
     - '^tests?:'
     - '^\w+\s+' # a hack to remove commit messages without colons thus don't correspond to a package

README.md

@@ -5,17 +5,16 @@
 <p align="center">Caddy is an extensible server platform that uses TLS by default.</p>
 <p align="center">
 	<a href="https://github.com/caddyserver/caddy/actions?query=workflow%3ACross-Platform"><img src="https://github.com/caddyserver/caddy/workflows/Cross-Platform/badge.svg"></a>
-	<a href="https://pkg.go.dev/github.com/caddyserver/caddy/v2"><img src="https://img.shields.io/badge/godoc-reference-blue.svg"></a>
-	<a href="https://app.fuzzit.dev/orgs/caddyserver-gh/dashboard"><img src="https://app.fuzzit.dev/badge?org_id=caddyserver-gh"></a>
+	<a href="https://pkg.go.dev/github.com/caddyserver/caddy/v2"><img src="https://img.shields.io/badge/godoc-reference-%23007d9c.svg"></a>
 	<br>
 	<a href="https://twitter.com/caddyserver" title="@caddyserver on Twitter"><img src="https://img.shields.io/badge/twitter-@caddyserver-55acee.svg" alt="@caddyserver on Twitter"></a>
 	<a href="https://caddy.community" title="Caddy Forum"><img src="https://img.shields.io/badge/community-forum-ff69b4.svg" alt="Caddy Forum"></a>
 	<a href="https://sourcegraph.com/github.com/caddyserver/caddy?badge" title="Caddy on Sourcegraph"><img src="https://sourcegraph.com/github.com/caddyserver/caddy/-/badge.svg" alt="Caddy on Sourcegraph"></a>
 </p>
 <p align="center">
-	<a href="https://github.com/caddyserver/caddy/releases">Download</a> ·
+	<a href="https://github.com/caddyserver/caddy/releases">Releases</a> ·
 	<a href="https://caddyserver.com/docs/">Documentation</a> ·
-	<a href="https://caddy.community">Community</a>
+	<a href="https://caddy.community">Get Help</a>
 </p>
 
 
@@ -26,7 +25,7 @@
 - [Build from source](#build-from-source)
 	- [For development](#for-development)
 	- [With version information and/or plugins](#with-version-information-andor-plugins)
-- [Getting started](#getting-started)
+- [Quick start](#quick-start)
 - [Overview](#overview)
 - [Full documentation](#full-documentation)
 - [Getting help](#getting-help)
@@ -43,7 +42,7 @@
 
 - **Easy configuration** with the [Caddyfile](https://caddyserver.com/docs/caddyfile)
 - **Powerful configuration** with its [native JSON config](https://caddyserver.com/docs/json/)
-- **Dynamic configuration** with the [JSON API](https://caddyserver.com/api)
+- **Dynamic configuration** with the [JSON API](https://caddyserver.com/docs/api)
 - [**Config adapters**](https://caddyserver.com/docs/config-adapters) if you don't like JSON
 - **Automatic HTTPS** by default
 	- [Let's Encrypt](https://letsencrypt.org) for public sites
@@ -51,7 +50,7 @@
 	- Can coordinate with other Caddy instances in a cluster
 - **Stays up when other servers go down** due to TLS/OCSP/certificate-related issues
 - **HTTP/1.1, HTTP/2, and experimental HTTP/3** support
-- **Highly extensible** [modular architecture](https://caddyserver.com/docs/extending-caddy) lets Caddy do anything without bloat
+- **Highly extensible** [modular architecture](https://caddyserver.com/docs/architecture) lets Caddy do anything without bloat
 - **Runs anywhere** with **no external dependencies** (not even libc)
 - Written in Go, a language with higher **memory safety guarantees** than other servers
 - Actually **fun to use**
@@ -64,7 +63,6 @@
 Requirements:
 
 - [Go 1.14 or newer](https://golang.org/dl/)
-- Do NOT disable [Go modules](https://github.com/golang/go/wiki/Modules) (`export GO111MODULE=on`)
 
 ### For development
  
@@ -78,10 +76,10 @@ _**Note:** These steps [will not embed proper version information](https://githu
 
 ### With version information and/or plugins
 
-Using [our builder tool](https://github.com/caddyserver/xcaddy)...
+Using [our builder tool, `xcaddy`](https://github.com/caddyserver/xcaddy)...
 
 ```
-$ xcaddy --version CADDY_VERSION
+$ xcaddy build
 ```
 
 ...the following steps are automated:
@@ -90,8 +88,9 @@ $ xcaddy --version CADDY_VERSION
 2. Change into it: `cd caddy`
 3. Copy [Caddy's main.go](https://github.com/caddyserver/caddy/blob/master/cmd/caddy/main.go) into the empty folder. Add imports for any custom plugins you want to add.
 4. Initialize a Go module: `go mod init caddy`
-5. Pin Caddy version: `go get github.com/caddyserver/caddy/v2@TAG` replacing `TAG` with a git tag or commit. You can also pin any plugin versions similarly.
-6. Compile: `go build`
+5. (Optional) Pin Caddy version: `go get github.com/caddyserver/caddy/v2@version` replacing `version` with a git tag or commit.
+6. (Optional) Add plugins by adding their import: `_ "import/path/here"`
+7. Compile: `go build`
 
 
 
@@ -119,7 +118,7 @@ The primary way to configure Caddy is through [its API](https://caddyserver.com/
 
 Caddy exposes an unprecedented level of control compared to any web server in existence. In Caddy, you are usually setting the actual values of the initialized types in memory that power everything from your HTTP handlers and TLS handshakes to your storage medium. Caddy is also ridiculously extensible, with a powerful plugin system that makes vast improvements over other web servers.
 
-To wield the power of this design, you need to know how the config document is structured. Please see the [our documentation site](https://caddyserver.com/docs/) for details about [Caddy's config structure](https://caddyserver.com/docs/json/).
+To wield the power of this design, you need to know how the config document is structured. Please see [our documentation site](https://caddyserver.com/docs/) for details about [Caddy's config structure](https://caddyserver.com/docs/json/).
 
 Nearly all of Caddy's configuration is contained in a single config document, rather than being scattered across CLI flags and env variables and a configuration file as with other web servers. This makes managing your server config more straightforward and reduces hidden variables/factors.
 
@@ -140,7 +139,7 @@ The docs are also open source. You can contribute to them here: https://github.c
 
 - Individuals can exchange help for free on our community forum at https://caddy.community. Remember that people give help out of their spare time and good will. The best way to get help is to give it first!
 
-Please use our [issue tracker](/caddyserver/caddy/issues) only for bug reports and feature requests, i.e. actionable development items (support questions will usually be referred to the forums).
+Please use our [issue tracker](https://github.com/caddyserver/caddy/issues) only for bug reports and feature requests, i.e. actionable development items (support questions will usually be referred to the forums).
 
 
 

admin.go

@@ -18,9 +18,12 @@ import (
 	"bytes"
 	"context"
 	"encoding/json"
+	"errors"
 	"expvar"
 	"fmt"
 	"io"
+	"io/ioutil"
+	"net"
 	"net/http"
 	"net/http/pprof"
 	"net/url"
@@ -32,6 +35,7 @@ import (
 	"sync"
 	"time"
 
+	"github.com/prometheus/client_golang/prometheus"
 	"go.uber.org/zap"
 )
 
@@ -59,10 +63,11 @@ type AdminConfig struct {
 	// default.
 	EnforceOrigin bool `json:"enforce_origin,omitempty"`
 
-	// The list of allowed origins for API requests. Only used if
-	// `enforce_origin` is true. If not set, the listener address
-	// will be the default value. If set but empty, no origins will
-	// be allowed.
+	// The list of allowed origins/hosts for API requests. Only needed
+	// if accessing the admin endpoint from a host different from the
+	// socket's network interface or if `enforce_origin` is true. If not
+	// set, the listener address will be the default value. If set but
+	// empty, no origins will be allowed.
 	Origins []string `json:"origins,omitempty"`
 
 	// Options related to configuration management.
@@ -78,58 +83,78 @@ type ConfigSettings struct {
 
 // listenAddr extracts a singular listen address from ac.Listen,
 // returning the network and the address of the listener.
-func (admin AdminConfig) listenAddr() (string, string, error) {
+func (admin AdminConfig) listenAddr() (NetworkAddress, error) {
 	input := admin.Listen
 	if input == "" {
 		input = DefaultAdminListen
 	}
 	listenAddr, err := ParseNetworkAddress(input)
 	if err != nil {
-		return "", "", fmt.Errorf("parsing admin listener address: %v", err)
+		return NetworkAddress{}, fmt.Errorf("parsing admin listener address: %v", err)
 	}
 	if listenAddr.PortRangeSize() != 1 {
-		return "", "", fmt.Errorf("admin endpoint must have exactly one address; cannot listen on %v", listenAddr)
+		return NetworkAddress{}, fmt.Errorf("admin endpoint must have exactly one address; cannot listen on %v", listenAddr)
 	}
-	return listenAddr.Network, listenAddr.JoinHostPort(0), nil
+	return listenAddr, nil
 }
 
 // newAdminHandler reads admin's config and returns an http.Handler suitable
 // for use in an admin endpoint server, which will be listening on listenAddr.
-func (admin AdminConfig) newAdminHandler(listenAddr string) adminHandler {
+func (admin AdminConfig) newAdminHandler(addr NetworkAddress) adminHandler {
 	muxWrap := adminHandler{
 		enforceOrigin:  admin.EnforceOrigin,
-		allowedOrigins: admin.allowedOrigins(listenAddr),
+		enforceHost:    !addr.isWildcardInterface(),
+		allowedOrigins: admin.allowedOrigins(addr),
 		mux:            http.NewServeMux(),
 	}
 
+	addRouteWithMetrics := func(pattern string, handlerLabel string, h http.Handler) {
+		labels := prometheus.Labels{"path": pattern, "handler": handlerLabel}
+		h = instrumentHandlerCounter(
+			adminMetrics.requestCount.MustCurryWith(labels),
+			h,
+		)
+		muxWrap.mux.Handle(pattern, h)
+	}
 	// addRoute just calls muxWrap.mux.Handle after
 	// wrapping the handler with error handling
-	addRoute := func(pattern string, h AdminHandler) {
+	addRoute := func(pattern string, handlerLabel string, h AdminHandler) {
 		wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 			err := h.ServeHTTP(w, r)
+			if err != nil {
+				labels := prometheus.Labels{
+					"path":    pattern,
+					"handler": handlerLabel,
+					"method":  strings.ToUpper(r.Method),
+				}
+				adminMetrics.requestErrors.With(labels).Inc()
+			}
 			muxWrap.handleError(w, r, err)
 		})
-		muxWrap.mux.Handle(pattern, wrapper)
+		addRouteWithMetrics(pattern, handlerLabel, wrapper)
 	}
 
+	const handlerLabel = "admin"
+
 	// register standard config control endpoints
-	addRoute("/"+rawConfigKey+"/", AdminHandlerFunc(handleConfig))
-	addRoute("/id/", AdminHandlerFunc(handleConfigID))
-	addRoute("/stop", AdminHandlerFunc(handleStop))
+	addRoute("/"+rawConfigKey+"/", handlerLabel, AdminHandlerFunc(handleConfig))
+	addRoute("/id/", handlerLabel, AdminHandlerFunc(handleConfigID))
+	addRoute("/stop", handlerLabel, AdminHandlerFunc(handleStop))
 
 	// register debugging endpoints
-	muxWrap.mux.HandleFunc("/debug/pprof/", pprof.Index)
-	muxWrap.mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
-	muxWrap.mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
-	muxWrap.mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
-	muxWrap.mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
-	muxWrap.mux.Handle("/debug/vars", expvar.Handler())
+	addRouteWithMetrics("/debug/pprof/", handlerLabel, http.HandlerFunc(pprof.Index))
+	addRouteWithMetrics("/debug/pprof/cmdline", handlerLabel, http.HandlerFunc(pprof.Cmdline))
+	addRouteWithMetrics("/debug/pprof/profile", handlerLabel, http.HandlerFunc(pprof.Profile))
+	addRouteWithMetrics("/debug/pprof/symbol", handlerLabel, http.HandlerFunc(pprof.Symbol))
+	addRouteWithMetrics("/debug/pprof/trace", handlerLabel, http.HandlerFunc(pprof.Trace))
+	addRouteWithMetrics("/debug/vars", handlerLabel, expvar.Handler())
 
 	// register third-party module endpoints
 	for _, m := range GetModules("admin.api") {
 		router := m.New().(AdminRouter)
+		handlerLabel := m.ID.Name()
 		for _, route := range router.Routes() {
-			addRoute(route.Pattern, route.Handler)
+			addRoute(route.Pattern, handlerLabel, route.Handler)
 		}
 	}
 
@@ -140,14 +165,30 @@ func (admin AdminConfig) newAdminHandler(listenAddr string) adminHandler {
 // If admin.Origins is nil (null), the provided listen address
 // will be used as the default origin. If admin.Origins is
 // empty, no origins will be allowed, effectively bricking the
-// endpoint, but whatever.
-func (admin AdminConfig) allowedOrigins(listen string) []string {
+// endpoint for non-unix-socket endpoints, but whatever.
+func (admin AdminConfig) allowedOrigins(addr NetworkAddress) []string {
 	uniqueOrigins := make(map[string]struct{})
 	for _, o := range admin.Origins {
 		uniqueOrigins[o] = struct{}{}
 	}
 	if admin.Origins == nil {
-		uniqueOrigins[listen] = struct{}{}
+		if addr.isLoopback() {
+			if addr.IsUnixNetwork() {
+				// RFC 2616, Section 14.26:
+				// "A client MUST include a Host header field in all HTTP/1.1 request
+				// messages. If the requested URI does not include an Internet host
+				// name for the service being requested, then the Host header field MUST
+				// be given with an empty value."
+				uniqueOrigins[""] = struct{}{}
+			} else {
+				uniqueOrigins[net.JoinHostPort("localhost", addr.port())] = struct{}{}
+				uniqueOrigins[net.JoinHostPort("::1", addr.port())] = struct{}{}
+				uniqueOrigins[net.JoinHostPort("127.0.0.1", addr.port())] = struct{}{}
+			}
+		}
+		if !addr.IsUnixNetwork() {
+			uniqueOrigins[addr.JoinHostPort(0)] = struct{}{}
+		}
 	}
 	allowed := make([]string, 0, len(uniqueOrigins))
 	for origin := range uniqueOrigins {
@@ -195,14 +236,14 @@ func replaceAdmin(cfg *Config) error {
 	}
 
 	// extract a singular listener address
-	netw, addr, err := adminConfig.listenAddr()
+	addr, err := adminConfig.listenAddr()
 	if err != nil {
 		return err
 	}
 
 	handler := adminConfig.newAdminHandler(addr)
 
-	ln, err := Listen(netw, addr)
+	ln, err := Listen(addr.Network, addr.JoinHostPort(0))
 	if err != nil {
 		return err
 	}
@@ -215,14 +256,22 @@ func replaceAdmin(cfg *Config) error {
 		MaxHeaderBytes:    1024 * 64,
 	}
 
-	go adminServer.Serve(ln)
+	adminLogger := Log().Named("admin")
+	go func() {
+		if err := adminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+			adminLogger.Error("admin server shutdown for unknown reason", zap.Error(err))
+		}
+	}()
 
-	Log().Named("admin").Info(
-		"admin endpoint started",
-		zap.String("address", addr),
+	adminLogger.Info("admin endpoint started",
+		zap.String("address", addr.String()),
 		zap.Bool("enforce_origin", adminConfig.EnforceOrigin),
-		zap.Strings("origins", handler.allowedOrigins),
-	)
+		zap.Strings("origins", handler.allowedOrigins))
+
+	if !handler.enforceHost {
+		adminLogger.Warn("admin endpoint on open interface; host checking disabled",
+			zap.String("address", addr.String()))
+	}
 
 	return nil
 }
@@ -254,6 +303,7 @@ type AdminRoute struct {
 
 type adminHandler struct {
 	enforceOrigin  bool
+	enforceHost    bool
 	allowedOrigins []string
 	mux            *http.ServeMux
 }
@@ -263,6 +313,7 @@ type adminHandler struct {
 func (h adminHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	Log().Named("admin.api").Info("received request",
 		zap.String("method", r.Method),
+		zap.String("host", r.Host),
 		zap.String("uri", r.RequestURI),
 		zap.String("remote_addr", r.RemoteAddr),
 		zap.Reflect("headers", r.Header),
@@ -274,14 +325,24 @@ func (h adminHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 // be called more than once per request, for example if a request
 // is rewritten (i.e. internal redirect).
 func (h adminHandler) serveHTTP(w http.ResponseWriter, r *http.Request) {
-	if h.enforceOrigin {
+	if strings.Contains(r.Header.Get("Upgrade"), "websocket") {
+		// I've never been able demonstrate a vulnerability myself, but apparently
+		// WebSocket connections originating from browsers aren't subject to CORS
+		// restrictions, so we'll just be on the safe side
+		h.handleError(w, r, fmt.Errorf("websocket connections aren't allowed"))
+		return
+	}
+
+	if h.enforceHost {
 		// DNS rebinding mitigation
 		err := h.checkHost(r)
 		if err != nil {
 			h.handleError(w, r, err)
 			return
 		}
+	}
 
+	if h.enforceOrigin {
 		// cross-site mitigation
 		origin, err := h.checkOrigin(r)
 		if err != nil {
@@ -494,16 +555,19 @@ func handleStop(w http.ResponseWriter, r *http.Request) error {
 	if err != nil {
 		Log().Named("admin.api").Error("unload error", zap.Error(err))
 	}
-	go func() {
-		err := stopAdminServer(adminServer)
-		var exitCode int
-		if err != nil {
-			exitCode = ExitCodeFailedQuit
-			Log().Named("admin.api").Error("failed to stop admin server gracefully", zap.Error(err))
-		}
-		Log().Named("admin.api").Info("stopping now, bye!! 👋")
-		os.Exit(exitCode)
-	}()
+	if adminServer != nil {
+		// use goroutine so that we can finish responding to API request
+		go func() {
+			err := stopAdminServer(adminServer)
+			var exitCode int
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				Log().Named("admin.api").Error("failed to stop admin server gracefully", zap.Error(err))
+			}
+			Log().Named("admin.api").Info("stopping now, bye!! 👋")
+			os.Exit(exitCode)
+		}()
+	}
 	return nil
 }
 
@@ -517,13 +581,6 @@ func handleUnload(w http.ResponseWriter, r *http.Request) error {
 			Err:  fmt.Errorf("method not allowed"),
 		}
 	}
-	currentCfgMu.RLock()
-	hasCfg := currentCfg != nil
-	currentCfgMu.RUnlock()
-	if !hasCfg {
-		Log().Named("admin.api").Info("nothing to unload")
-		return nil
-	}
 	Log().Named("admin.api").Info("unloading")
 	if err := stopAndCleanup(); err != nil {
 		Log().Named("admin.api").Error("error unloading", zap.Error(err))
@@ -772,12 +829,27 @@ var (
 	}
 )
 
+// PIDFile writes a pidfile to the file at filename. It
+// will get deleted before the process gracefully exits.
+func PIDFile(filename string) error {
+	pid := []byte(strconv.Itoa(os.Getpid()) + "\n")
+	err := ioutil.WriteFile(filename, pid, 0644)
+	if err != nil {
+		return err
+	}
+	pidfile = filename
+	return nil
+}
+
 // idRegexp is used to match ID fields and their associated values
 // in the config. It also matches adjacent commas so that syntax
 // can be preserved no matter where in the object the field appears.
 // It supports string and most numeric values.
 var idRegexp = regexp.MustCompile(`(?m),?\s*"` + idKey + `"\s*:\s*(-?[0-9]+(\.[0-9]+)?|(?U)".*")\s*,?`)
 
+// pidfile is the name of the pidfile, if any.
+var pidfile string
+
 const (
 	rawConfigKey = "config"
 	idKey        = "@id"

caddy.go

@@ -470,6 +470,9 @@ func stopAndCleanup() error {
 		return err
 	}
 	certmagic.CleanUpOwnLocks()
+	if pidfile != "" {
+		return os.Remove(pidfile)
+	}
 	return nil
 }
 
@@ -486,7 +489,7 @@ func Validate(cfg *Config) error {
 // Duration can be an integer or a string. An integer is
 // interpreted as nanoseconds. If a string, it is a Go
 // time.Duration value such as `300ms`, `1.5h`, or `2h45m`;
-// valid units are `ns`, `us`/`µs`, `ms`, `s`, `m`, and `h`.
+// valid units are `ns`, `us`/`µs`, `ms`, `s`, `m`, `h`, and `d`.
 type Duration time.Duration
 
 // UnmarshalJSON satisfies json.Unmarshaler.
@@ -497,7 +500,7 @@ func (d *Duration) UnmarshalJSON(b []byte) error {
 	var dur time.Duration
 	var err error
 	if b[0] == byte('"') && b[len(b)-1] == byte('"') {
-		dur, err = time.ParseDuration(strings.Trim(string(b), `"`))
+		dur, err = ParseDuration(strings.Trim(string(b), `"`))
 	} else {
 		err = json.Unmarshal(b, &dur)
 	}
@@ -505,6 +508,34 @@ func (d *Duration) UnmarshalJSON(b []byte) error {
 	return err
 }
 
+// ParseDuration parses a duration string, adding
+// support for the "d" unit meaning number of days,
+// where a day is assumed to be 24h.
+func ParseDuration(s string) (time.Duration, error) {
+	var inNumber bool
+	var numStart int
+	for i := 0; i < len(s); i++ {
+		ch := s[i]
+		if ch == 'd' {
+			daysStr := s[numStart:i]
+			days, err := strconv.ParseFloat(daysStr, 64)
+			if err != nil {
+				return 0, err
+			}
+			hours := days * 24.0
+			hoursStr := strconv.FormatFloat(hours, 'f', -1, 64)
+			s = s[:numStart] + hoursStr + "h" + s[i+1:]
+			i--
+			continue
+		}
+		if !inNumber {
+			numStart = i
+		}
+		inNumber = (ch >= '0' && ch <= '9') || ch == '.' || ch == '-' || ch == '+'
+	}
+	return time.ParseDuration(s)
+}
+
 // GoModule returns the build info of this Caddy
 // build from debug.BuildInfo (requires Go modules).
 // If no version information is available, a non-nil

caddy_test.go

@@ -0,0 +1,74 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddy
+
+import (
+	"testing"
+	"time"
+)
+
+func TestParseDuration(t *testing.T) {
+	const day = 24 * time.Hour
+	for i, tc := range []struct {
+		input  string
+		expect time.Duration
+	}{
+		{
+			input:  "3h",
+			expect: 3 * time.Hour,
+		},
+		{
+			input:  "1d",
+			expect: day,
+		},
+		{
+			input:  "1d30m",
+			expect: day + 30*time.Minute,
+		},
+		{
+			input:  "1m2d",
+			expect: time.Minute + day*2,
+		},
+		{
+			input:  "1m2d30s",
+			expect: time.Minute + day*2 + 30*time.Second,
+		},
+		{
+			input:  "1d2d",
+			expect: 3 * day,
+		},
+		{
+			input:  "1.5d",
+			expect: time.Duration(1.5 * float64(day)),
+		},
+		{
+			input:  "4m1.25d",
+			expect: 4*time.Minute + time.Duration(1.25*float64(day)),
+		},
+		{
+			input:  "-1.25d12h",
+			expect: time.Duration(-1.25*float64(day)) - 12*time.Hour,
+		},
+	} {
+		actual, err := ParseDuration(tc.input)
+		if err != nil {
+			t.Errorf("Test %d ('%s'): Got error: %v", i, tc.input, err)
+			continue
+		}
+		if actual != tc.expect {
+			t.Errorf("Test %d ('%s'): Expected=%s Actual=%s", i, tc.input, tc.expect, actual)
+		}
+	}
+}

caddyconfig/caddyfile/dispenser.go

@@ -17,6 +17,8 @@ package caddyfile
 import (
 	"errors"
 	"fmt"
+	"io"
+	"log"
 	"strings"
 )
 
@@ -37,6 +39,16 @@ func NewDispenser(tokens []Token) *Dispenser {
 	}
 }
 
+// NewTestDispenser parses input into tokens and creates a new
+// Dispenser for test purposes only; any errors are fatal.
+func NewTestDispenser(input string) *Dispenser {
+	tokens, err := allTokens("Testfile", []byte(input))
+	if err != nil && err != io.EOF {
+		log.Fatalf("getting all tokens from input: %v", err)
+	}
+	return NewDispenser(tokens)
+}
+
 // Next loads the next token. Returns true if a token
 // was loaded; false otherwise. If false, all tokens
 // have been consumed.

caddyconfig/caddyfile/dispenser_test.go

@@ -15,8 +15,6 @@
 package caddyfile
 
 import (
-	"io"
-	"log"
 	"reflect"
 	"strings"
 	"testing"
@@ -306,13 +304,3 @@ func TestDispenser_ArgErr_Err(t *testing.T) {
 		t.Errorf("Expected error message with custom message in it ('foobar'); got '%v'", err)
 	}
 }
-
-// NewTestDispenser parses input into tokens and creates a new
-// Disenser for test purposes only; any errors are fatal.
-func NewTestDispenser(input string) *Dispenser {
-	tokens, err := allTokens("Testfile", []byte(input))
-	if err != nil && err != io.EOF {
-		log.Fatalf("getting all tokens from input: %v", err)
-	}
-	return NewDispenser(tokens)
-}

caddyconfig/caddyfile/formatter.go

@@ -131,9 +131,6 @@ func Format(input []byte) []byte {
 		//////////////////////////////////////////////////////////
 
 		if ch == '#' {
-			if !spacePrior && !beginningOfLine {
-				write(' ')
-			}
 			comment = true
 		}
 

caddyconfig/caddyfile/formatter_test.go

@@ -201,7 +201,7 @@ c
 }
 
 d {
-	e #f
+	e#f
 	# g
 }
 
@@ -229,7 +229,7 @@ bar"
 j {
 "\"k\" l m"
 }`,
-			expect: `"a \"b\" " #c
+			expect: `"a \"b\" "#c
 d
 
 e {
@@ -305,6 +305,11 @@ bar "{\"key\":34}"`,
 
 baz`,
 		},
+		{
+			description: "hash within string is not a comment",
+			input:       `redir / /some/#/path`,
+			expect:      `redir / /some/#/path`,
+		},
 	} {
 		// the formatter should output a trailing newline,
 		// even if the tests aren't written to expect that

caddyconfig/caddyfile/lexer.go

@@ -16,6 +16,7 @@ package caddyfile
 
 import (
 	"bufio"
+	"bytes"
 	"io"
 	"unicode"
 )
@@ -73,7 +74,7 @@ func (l *lexer) load(input io.Reader) error {
 // a token was loaded; false otherwise.
 func (l *lexer) next() bool {
 	var val []rune
-	var comment, quoted, escaped bool
+	var comment, quoted, btQuoted, escaped bool
 
 	makeToken := func() bool {
 		l.token.Text = string(val)
@@ -92,13 +93,13 @@ func (l *lexer) next() bool {
 			panic(err)
 		}
 
-		if !escaped && ch == '\\' {
+		if !escaped && !btQuoted && ch == '\\' {
 			escaped = true
 			continue
 		}
 
-		if quoted {
-			if escaped {
+		if quoted || btQuoted {
+			if quoted && escaped {
 				// all is literal in quoted area,
 				// so only escape quotes
 				if ch != '"' {
@@ -106,7 +107,10 @@ func (l *lexer) next() bool {
 				}
 				escaped = false
 			} else {
-				if ch == '"' {
+				if quoted && ch == '"' {
+					return makeToken()
+				}
+				if btQuoted && ch == '`' {
 					return makeToken()
 				}
 			}
@@ -138,7 +142,7 @@ func (l *lexer) next() bool {
 			continue
 		}
 
-		if ch == '#' {
+		if ch == '#' && len(val) == 0 {
 			comment = true
 		}
 		if comment {
@@ -151,6 +155,10 @@ func (l *lexer) next() bool {
 				quoted = true
 				continue
 			}
+			if ch == '`' {
+				btQuoted = true
+				continue
+			}
 		}
 
 		if escaped {
@@ -161,3 +169,21 @@ func (l *lexer) next() bool {
 		val = append(val, ch)
 	}
 }
+
+// Tokenize takes bytes as input and lexes it into
+// a list of tokens that can be parsed as a Caddyfile.
+// Also takes a filename to fill the token's File as
+// the source of the tokens, which is important to
+// determine relative paths for `import` directives.
+func Tokenize(input []byte, filename string) ([]Token, error) {
+	l := lexer{}
+	if err := l.load(bytes.NewReader(input)); err != nil {
+		return nil, err
+	}
+	var tokens []Token
+	for l.next() {
+		l.token.File = filename
+		tokens = append(tokens, l.token)
+	}
+	return tokens, nil
+}

caddyconfig/caddyfile/lexer_test.go

@@ -15,37 +15,35 @@
 package caddyfile
 
 import (
-	"log"
-	"strings"
 	"testing"
 )
 
 type lexerTestCase struct {
-	input    string
+	input    []byte
 	expected []Token
 }
 
 func TestLexer(t *testing.T) {
 	testCases := []lexerTestCase{
 		{
-			input: `host:123`,
+			input: []byte(`host:123`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 			},
 		},
 		{
-			input: `host:123
+			input: []byte(`host:123
 
-					directive`,
+					directive`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 3, Text: "directive"},
 			},
 		},
 		{
-			input: `host:123 {
+			input: []byte(`host:123 {
 						directive
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -54,7 +52,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `host:123 { directive }`,
+			input: []byte(`host:123 { directive }`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -63,12 +61,12 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `host:123 {
+			input: []byte(`host:123 {
 						#comment
 						directive
 						# comment
 						foobar # another comment
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -78,8 +76,28 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `a "quoted value" b
-					foobar`,
+			input: []byte(`host:123 {
+						# hash inside string is not a comment
+						redir / /some/#/path
+					}`),
+			expected: []Token{
+				{Line: 1, Text: "host:123"},
+				{Line: 1, Text: "{"},
+				{Line: 3, Text: "redir"},
+				{Line: 3, Text: "/"},
+				{Line: 3, Text: "/some/#/path"},
+				{Line: 4, Text: "}"},
+			},
+		},
+		{
+			input: []byte("# comment at beginning of file\n# comment at beginning of line\nhost:123"),
+			expected: []Token{
+				{Line: 3, Text: "host:123"},
+			},
+		},
+		{
+			input: []byte(`a "quoted value" b
+					foobar`),
 			expected: []Token{
 				{Line: 1, Text: "a"},
 				{Line: 1, Text: "quoted value"},
@@ -88,7 +106,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `A "quoted \"value\" inside" B`,
+			input: []byte(`A "quoted \"value\" inside" B`),
 			expected: []Token{
 				{Line: 1, Text: "A"},
 				{Line: 1, Text: `quoted "value" inside`},
@@ -96,7 +114,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "An escaped \"newline\\\ninside\" quotes",
+			input: []byte("An escaped \"newline\\\ninside\" quotes"),
 			expected: []Token{
 				{Line: 1, Text: "An"},
 				{Line: 1, Text: "escaped"},
@@ -105,7 +123,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "An escaped newline\\\noutside quotes",
+			input: []byte("An escaped newline\\\noutside quotes"),
 			expected: []Token{
 				{Line: 1, Text: "An"},
 				{Line: 1, Text: "escaped"},
@@ -115,7 +133,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "line1\\\nescaped\nline2\nline3",
+			input: []byte("line1\\\nescaped\nline2\nline3"),
 			expected: []Token{
 				{Line: 1, Text: "line1"},
 				{Line: 1, Text: "escaped"},
@@ -124,7 +142,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "line1\\\nescaped1\\\nescaped2\nline4\nline5",
+			input: []byte("line1\\\nescaped1\\\nescaped2\nline4\nline5"),
 			expected: []Token{
 				{Line: 1, Text: "line1"},
 				{Line: 1, Text: "escaped1"},
@@ -134,34 +152,34 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `"unescapable\ in quotes"`,
+			input: []byte(`"unescapable\ in quotes"`),
 			expected: []Token{
 				{Line: 1, Text: `unescapable\ in quotes`},
 			},
 		},
 		{
-			input: `"don't\escape"`,
+			input: []byte(`"don't\escape"`),
 			expected: []Token{
 				{Line: 1, Text: `don't\escape`},
 			},
 		},
 		{
-			input: `"don't\\escape"`,
+			input: []byte(`"don't\\escape"`),
 			expected: []Token{
 				{Line: 1, Text: `don't\\escape`},
 			},
 		},
 		{
-			input: `un\escapable`,
+			input: []byte(`un\escapable`),
 			expected: []Token{
 				{Line: 1, Text: `un\escapable`},
 			},
 		},
 		{
-			input: `A "quoted value with line
+			input: []byte(`A "quoted value with line
 					break inside" {
 						foobar
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "A"},
 				{Line: 1, Text: "quoted value with line\n\t\t\t\t\tbreak inside"},
@@ -171,13 +189,13 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `"C:\php\php-cgi.exe"`,
+			input: []byte(`"C:\php\php-cgi.exe"`),
 			expected: []Token{
 				{Line: 1, Text: `C:\php\php-cgi.exe`},
 			},
 		},
 		{
-			input: `empty "" string`,
+			input: []byte(`empty "" string`),
 			expected: []Token{
 				{Line: 1, Text: `empty`},
 				{Line: 1, Text: ``},
@@ -185,7 +203,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "skip those\r\nCR characters",
+			input: []byte("skip those\r\nCR characters"),
 			expected: []Token{
 				{Line: 1, Text: "skip"},
 				{Line: 1, Text: "those"},
@@ -194,30 +212,54 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "\xEF\xBB\xBF:8080", // test with leading byte order mark
+			input: []byte("\xEF\xBB\xBF:8080"), // test with leading byte order mark
 			expected: []Token{
 				{Line: 1, Text: ":8080"},
 			},
 		},
+		{
+			input: []byte("simple `backtick quoted` string"),
+			expected: []Token{
+				{Line: 1, Text: `simple`},
+				{Line: 1, Text: `backtick quoted`},
+				{Line: 1, Text: `string`},
+			},
+		},
+		{
+			input: []byte("multiline `backtick\nquoted\n` string"),
+			expected: []Token{
+				{Line: 1, Text: `multiline`},
+				{Line: 1, Text: "backtick\nquoted\n"},
+				{Line: 3, Text: `string`},
+			},
+		},
+		{
+			input: []byte("nested `\"quotes inside\" backticks` string"),
+			expected: []Token{
+				{Line: 1, Text: `nested`},
+				{Line: 1, Text: `"quotes inside" backticks`},
+				{Line: 1, Text: `string`},
+			},
+		},
+		{
+			input: []byte("reverse-nested \"`backticks` inside\" quotes"),
+			expected: []Token{
+				{Line: 1, Text: `reverse-nested`},
+				{Line: 1, Text: "`backticks` inside"},
+				{Line: 1, Text: `quotes`},
+			},
+		},
 	}
 
 	for i, testCase := range testCases {
-		actual := tokenize(testCase.input)
+		actual, err := Tokenize(testCase.input, "")
+		if err != nil {
+			t.Errorf("%v", err)
+		}
 		lexerCompare(t, i, testCase.expected, actual)
 	}
 }
 
-func tokenize(input string) (tokens []Token) {
-	l := lexer{}
-	if err := l.load(strings.NewReader(input)); err != nil {
-		log.Printf("[ERROR] load failed: %v", err)
-	}
-	for l.next() {
-		tokens = append(tokens, l.token)
-	}
-	return
-}
-
 func lexerCompare(t *testing.T, n int, expected, actual []Token) {
 	if len(expected) != len(actual) {
 		t.Errorf("Test case %d: expected %d token(s) but got %d", n, len(expected), len(actual))

caddyconfig/caddyfile/parse.go

@@ -20,7 +20,10 @@ import (
 	"log"
 	"os"
 	"path/filepath"
+	"strconv"
 	"strings"
+
+	"github.com/caddyserver/caddy/v2"
 )
 
 // Parse parses the input just enough to group tokens, in
@@ -84,16 +87,10 @@ func allTokens(filename string, input []byte) ([]Token, error) {
 	if err != nil {
 		return nil, err
 	}
-	l := new(lexer)
-	err = l.load(bytes.NewReader(input))
+	tokens, err := Tokenize(input, filename)
 	if err != nil {
 		return nil, err
 	}
-	var tokens []Token
-	for l.next() {
-		l.token.File = filename
-		tokens = append(tokens, l.token)
-	}
 	return tokens, nil
 }
 
@@ -292,11 +289,19 @@ func (p *parser) doImport() error {
 	if importPattern == "" {
 		return p.Err("Import requires a non-empty filepath")
 	}
-	if p.NextArg() {
-		return p.Err("Import takes only one argument (glob pattern or file)")
+
+	// grab remaining args as placeholder replacements
+	args := p.RemainingArgs()
+
+	// add args to the replacer
+	repl := caddy.NewReplacer()
+	for index, arg := range args {
+		repl.Set("args."+strconv.Itoa(index), arg)
 	}
-	// splice out the import directive and its argument (2 tokens total)
-	tokensBefore := p.tokens[:p.cursor-1]
+
+	// splice out the import directive and its arguments
+	// (2 tokens, plus the length of args)
+	tokensBefore := p.tokens[:p.cursor-1-len(args)]
 	tokensAfter := p.tokens[p.cursor+1:]
 	var importedTokens []Token
 
@@ -348,10 +353,20 @@ func (p *parser) doImport() error {
 		}
 	}
 
+	// copy the tokens so we don't overwrite p.definedSnippets
+	tokensCopy := make([]Token, len(importedTokens))
+	copy(tokensCopy, importedTokens)
+
+	// run the argument replacer on the tokens
+	for index, token := range tokensCopy {
+		token.Text = repl.ReplaceKnown(token.Text, "")
+		tokensCopy[index] = token
+	}
+
 	// splice the imported tokens in the place of the import statement
 	// and rewind cursor so Next() will land on first imported token
-	p.tokens = append(tokensBefore, append(importedTokens, tokensAfter...)...)
-	p.cursor--
+	p.tokens = append(tokensBefore, append(tokensCopy, tokensAfter...)...)
+	p.cursor -= len(args) + 1
 
 	return nil
 }

caddyconfig/caddyfile/parse_test.go

@@ -182,14 +182,17 @@ func TestParseOneAndImport(t *testing.T) {
 			"host1",
 		}, []int{1, 2}},
 
-		{`import testdata/import_test1.txt testdata/import_test2.txt`, true, []string{}, []int{}},
-
 		{`import testdata/not_found.txt`, true, []string{}, []int{}},
 
 		{`""`, false, []string{}, []int{}},
 
 		{``, false, []string{}, []int{}},
 
+		// import with args
+		{`import testdata/import_args0.txt a`, false, []string{"a"}, []int{}},
+		{`import testdata/import_args1.txt a b`, false, []string{"a", "b"}, []int{}},
+		{`import testdata/import_args*.txt a b`, false, []string{"a"}, []int{2}},
+
 		// test cases found by fuzzing!
 		{`import }{$"`, true, []string{}, []int{}},
 		{`import /*/*.txt`, true, []string{}, []int{}},
@@ -210,6 +213,7 @@ func TestParseOneAndImport(t *testing.T) {
 			t.Errorf("Test %d: Expected no error, but got: %v", i, err)
 		}
 
+		// t.Logf("%+v\n", result)
 		if len(result.Keys) != len(test.keys) {
 			t.Errorf("Test %d: Expected %d keys, got %d",
 				i, len(test.keys), len(result.Keys))

caddyconfig/caddyfile/testdata/import_args0.txt

@@ -0,0 +1 @@
+{args.0}

caddyconfig/caddyfile/testdata/import_args1.txt

@@ -0,0 +1 @@
+{args.0} {args.1}

caddyconfig/configadapters.go

@@ -51,14 +51,13 @@ func JSON(val interface{}, warnings *[]Warning) json.RawMessage {
 	return b
 }
 
-// JSONModuleObject is like JSON, except it marshals val into a JSON object
-// and then adds a key to that object named fieldName with the value fieldVal.
-// This is useful for JSON-encoding module values where the module name has to
-// be described within the object by a certain key; for example,
-// "responder": "file_server" for a file server HTTP responder. The val must
-// encode into a map[string]interface{} (i.e. it must be a struct or map),
-// and any errors are converted into warnings, so this can be conveniently
-// used when filling a struct. For correct code, there should be no errors.
+// JSONModuleObject is like JSON(), except it marshals val into a JSON object
+// with an added key named fieldName with the value fieldVal. This is useful
+// for encoding module values where the module name has to be described within
+// the object by a certain key; for example, `"handler": "file_server"` for a
+// file server HTTP handler (fieldName="handler" and fieldVal="file_server").
+// The val parameter must encode into a map[string]interface{} (i.e. it must be
+// a struct or map). Any errors are converted into warnings.
 func JSONModuleObject(val interface{}, fieldName, fieldVal string, warnings *[]Warning) json.RawMessage {
 	// encode to a JSON object first
 	enc, err := json.Marshal(val)
@@ -101,13 +100,14 @@ func JSONIndent(val interface{}) ([]byte, error) {
 }
 
 // RegisterAdapter registers a config adapter with the given name.
-// This should usually be done at init-time.
-func RegisterAdapter(name string, adapter Adapter) error {
+// This should usually be done at init-time. It panics if the
+// adapter cannot be registered successfully.
+func RegisterAdapter(name string, adapter Adapter) {
 	if _, ok := configAdapters[name]; ok {
-		return fmt.Errorf("%s: already registered", name)
+		panic(fmt.Errorf("%s: already registered", name))
 	}
 	configAdapters[name] = adapter
-	return caddy.RegisterModule(adapterModule{name, adapter})
+	caddy.RegisterModule(adapterModule{name, adapter})
 }
 
 // GetAdapter returns the adapter with the given name,

caddyconfig/httpcaddyfile/addresses.go

@@ -20,6 +20,7 @@ import (
 	"reflect"
 	"strconv"
 	"strings"
+	"unicode"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
@@ -333,8 +334,8 @@ func (a Address) Normalize() Address {
 
 	return Address{
 		Original: a.Original,
-		Scheme:   strings.ToLower(a.Scheme),
-		Host:     strings.ToLower(host),
+		Scheme:   lowerExceptPlaceholders(a.Scheme),
+		Host:     lowerExceptPlaceholders(host),
 		Port:     a.Port,
 		Path:     path,
 	}
@@ -361,3 +362,31 @@ func (a Address) Key() string {
 	}
 	return res
 }
+
+// lowerExceptPlaceholders lowercases s except within
+// placeholders (substrings in non-escaped '{ }' spans).
+// See https://github.com/caddyserver/caddy/issues/3264
+func lowerExceptPlaceholders(s string) string {
+	var sb strings.Builder
+	var escaped, inPlaceholder bool
+	for _, ch := range s {
+		if ch == '\\' && !escaped {
+			escaped = true
+			sb.WriteRune(ch)
+			continue
+		}
+		if ch == '{' && !escaped {
+			inPlaceholder = true
+		}
+		if ch == '}' && inPlaceholder && !escaped {
+			inPlaceholder = false
+		}
+		if inPlaceholder {
+			sb.WriteRune(ch)
+		} else {
+			sb.WriteRune(unicode.ToLower(ch))
+		}
+		escaped = false
+	}
+	return sb.String()
+}

caddyconfig/httpcaddyfile/addresses_test.go

@@ -108,6 +108,10 @@ func TestKeyNormalization(t *testing.T) {
 		input  string
 		expect string
 	}{
+		{
+			input:  "example.com",
+			expect: "example.com",
+		},
 		{
 			input:  "http://host:1234/path",
 			expect: "http://host:1234/path",
@@ -124,6 +128,22 @@ func TestKeyNormalization(t *testing.T) {
 			input:  "A:2015/Path",
 			expect: "a:2015/Path",
 		},
+		{
+			input:  "sub.{env.MY_DOMAIN}",
+			expect: "sub.{env.MY_DOMAIN}",
+		},
+		{
+			input:  "sub.ExAmPle",
+			expect: "sub.example",
+		},
+		{
+			input:  "sub.\\{env.MY_DOMAIN\\}",
+			expect: "sub.\\{env.my_domain\\}",
+		},
+		{
+			input:  "sub.{env.MY_DOMAIN}.com",
+			expect: "sub.{env.MY_DOMAIN}.com",
+		},
 		{
 			input:  ":80",
 			expect: ":80",
@@ -156,7 +176,7 @@ func TestKeyNormalization(t *testing.T) {
 			continue
 		}
 		if actual := addr.Normalize().Key(); actual != tc.expect {
-			t.Errorf("Test %d: Normalized key for address '%s' was '%s' but expected '%s'", i, tc.input, actual, tc.expect)
+			t.Errorf("Test %d: Input '%s': Expected '%s' but got '%s'", i, tc.input, tc.expect, actual)
 		}
 
 	}

caddyconfig/httpcaddyfile/builtins.go

@@ -15,8 +15,11 @@
 package httpcaddyfile
 
 import (
+	"encoding/base64"
+	"encoding/pem"
 	"fmt"
 	"html"
+	"io/ioutil"
 	"net/http"
 	"reflect"
 	"strings"
@@ -26,6 +29,8 @@ import (
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
 	"go.uber.org/zap/zapcore"
 )
 
@@ -59,12 +64,21 @@ func parseBind(h Helper) ([]ConfigValue, error) {
 //         protocols <min> [<max>]
 //         ciphers   <cipher_suites...>
 //         curves    <curves...>
+//         client_auth {
+//             mode                   [request|require|verify_if_given|require_and_verify]
+//             trusted_ca_cert        <base64_der>
+//             trusted_ca_cert_file   <filename>
+//             trusted_leaf_cert      <base64_der>
+//             trusted_leaf_cert_file <filename>
+//         }
 //         alpn      <values...>
 //         load      <paths...>
 //         ca        <acme_ca_endpoint>
 //         ca_root   <pem_file>
 //         dns       <provider_name>
 //         on_demand
+//         eab    <key_id> <mac_key>
+//         issuer <module_name> ...
 //     }
 //
 func parseTLS(h Helper) ([]ConfigValue, error) {
@@ -74,6 +88,7 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 	var certSelector caddytls.CustomCertSelectionPolicy
 	var acmeIssuer *caddytls.ACMEIssuer
 	var internalIssuer *caddytls.InternalIssuer
+	var issuer certmagic.Issuer
 	var onDemand bool
 
 	for h.Next() {
@@ -143,7 +158,7 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 		}
 
 		var hasBlock bool
-		for h.NextBlock(0) {
+		for nesting := h.Nesting(); h.NextBlock(nesting); {
 			hasBlock = true
 
 			switch h.Val() {
@@ -181,6 +196,57 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 					cp.Curves = append(cp.Curves, h.Val())
 				}
 
+			case "client_auth":
+				cp.ClientAuthentication = &caddytls.ClientAuthentication{}
+				for nesting := h.Nesting(); h.NextBlock(nesting); {
+					subdir := h.Val()
+					switch subdir {
+					case "mode":
+						if !h.Args(&cp.ClientAuthentication.Mode) {
+							return nil, h.ArgErr()
+						}
+						if h.NextArg() {
+							return nil, h.ArgErr()
+						}
+
+					case "trusted_ca_cert",
+						"trusted_leaf_cert":
+						if !h.NextArg() {
+							return nil, h.ArgErr()
+						}
+						if subdir == "trusted_ca_cert" {
+							cp.ClientAuthentication.TrustedCACerts = append(cp.ClientAuthentication.TrustedCACerts, h.Val())
+						} else {
+							cp.ClientAuthentication.TrustedLeafCerts = append(cp.ClientAuthentication.TrustedLeafCerts, h.Val())
+						}
+
+					case "trusted_ca_cert_file",
+						"trusted_leaf_cert_file":
+						if !h.NextArg() {
+							return nil, h.ArgErr()
+						}
+						filename := h.Val()
+						certDataPEM, err := ioutil.ReadFile(filename)
+						if err != nil {
+							return nil, err
+						}
+						block, _ := pem.Decode(certDataPEM)
+						if block == nil || block.Type != "CERTIFICATE" {
+							return nil, h.Errf("no CERTIFICATE pem block found in %s", h.Val())
+						}
+						if subdir == "trusted_ca_cert_file" {
+							cp.ClientAuthentication.TrustedCACerts = append(cp.ClientAuthentication.TrustedCACerts,
+								base64.StdEncoding.EncodeToString(block.Bytes))
+						} else {
+							cp.ClientAuthentication.TrustedLeafCerts = append(cp.ClientAuthentication.TrustedLeafCerts,
+								base64.StdEncoding.EncodeToString(block.Bytes))
+						}
+
+					default:
+						return nil, h.Errf("unknown subdirective for client_auth: %s", subdir)
+					}
+				}
+
 			case "alpn":
 				args := h.RemainingArgs()
 				if len(args) == 0 {
@@ -201,22 +267,65 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 				}
 				acmeIssuer.CA = arg[0]
 
-			case "dns":
-				if !h.Next() {
+			case "eab":
+				arg := h.RemainingArgs()
+				if len(arg) != 2 {
 					return nil, h.ArgErr()
 				}
 				if acmeIssuer == nil {
 					acmeIssuer = new(caddytls.ACMEIssuer)
 				}
+				acmeIssuer.ExternalAccount = &acme.EAB{
+					KeyID:  arg[0],
+					MACKey: arg[1],
+				}
+
+			case "issuer":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				modName := h.Val()
+				mod, err := caddy.GetModule("tls.issuance." + modName)
+				if err != nil {
+					return nil, h.Errf("getting issuer module '%s': %v", modName, err)
+				}
+				unm, ok := mod.New().(caddyfile.Unmarshaler)
+				if !ok {
+					return nil, h.Errf("issuer module '%s' is not a Caddyfile unmarshaler", mod.ID)
+				}
+				err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+				if err != nil {
+					return nil, err
+				}
+				issuer, ok = unm.(certmagic.Issuer)
+				if !ok {
+					return nil, h.Errf("module %s is not a certmagic.Issuer", mod.ID)
+				}
+
+			case "dns":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
 				provName := h.Val()
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
 				if acmeIssuer.Challenges == nil {
 					acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
+					acmeIssuer.Challenges.DNS = new(caddytls.DNSChallengeConfig)
 				}
-				dnsProvModule, err := caddy.GetModule("tls.dns." + provName)
+				dnsProvModule, err := caddy.GetModule("dns.providers." + provName)
 				if err != nil {
 					return nil, h.Errf("getting DNS provider module named '%s': %v", provName, err)
 				}
-				acmeIssuer.Challenges.DNSRaw = caddyconfig.JSONModuleObject(dnsProvModule.New(), "provider", provName, h.warnings)
+				dnsProvModuleInstance := dnsProvModule.New()
+				if unm, ok := dnsProvModuleInstance.(caddyfile.Unmarshaler); ok {
+					err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+					if err != nil {
+						return nil, err
+					}
+				}
+				acmeIssuer.Challenges.DNS.ProviderRaw = caddyconfig.JSONModuleObject(dnsProvModuleInstance, "name", provName, h.warnings)
 
 			case "ca_root":
 				arg := h.RemainingArgs()
@@ -251,13 +360,13 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 	// certificate loaders
 	if len(fileLoader) > 0 {
 		configVals = append(configVals, ConfigValue{
-			Class: "tls.certificate_loader",
+			Class: "tls.cert_loader",
 			Value: fileLoader,
 		})
 	}
 	if len(folderLoader) > 0 {
 		configVals = append(configVals, ConfigValue{
-			Class: "tls.certificate_loader",
+			Class: "tls.cert_loader",
 			Value: folderLoader,
 		})
 	}
@@ -267,7 +376,24 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 		// the logic to support this would be complex
 		return nil, h.Err("cannot use both ACME and internal issuers in same server block")
 	}
-	if acmeIssuer != nil {
+	if issuer != nil && (acmeIssuer != nil || internalIssuer != nil) {
+		// similarly, the logic to support this would be complex
+		return nil, h.Err("when defining an issuer, all its config must be in its block, rather than from separate tls subdirectives")
+	}
+	switch {
+	case issuer != nil:
+		configVals = append(configVals, ConfigValue{
+			Class: "tls.cert_issuer",
+			Value: issuer,
+		})
+
+	case internalIssuer != nil:
+		configVals = append(configVals, ConfigValue{
+			Class: "tls.cert_issuer",
+			Value: internalIssuer,
+		})
+
+	case acmeIssuer != nil:
 		// fill in global defaults, if configured
 		if email := h.Option("email"); email != nil && acmeIssuer.Email == "" {
 			acmeIssuer.Email = email.(string)
@@ -278,15 +404,9 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 		if caPemFile := h.Option("acme_ca_root"); caPemFile != nil {
 			acmeIssuer.TrustedRootsPEMFiles = append(acmeIssuer.TrustedRootsPEMFiles, caPemFile.(string))
 		}
-
 		configVals = append(configVals, ConfigValue{
 			Class: "tls.cert_issuer",
-			Value: acmeIssuer,
-		})
-	} else if internalIssuer != nil {
-		configVals = append(configVals, ConfigValue{
-			Class: "tls.cert_issuer",
-			Value: internalIssuer,
+			Value: disambiguateACMEIssuer(acmeIssuer),
 		})
 	}
 
@@ -397,36 +517,23 @@ func parseRespond(h Helper) (caddyhttp.MiddlewareHandler, error) {
 func parseRoute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	sr := new(caddyhttp.Subroute)
 
-	for h.Next() {
-		for nesting := h.Nesting(); h.NextBlock(nesting); {
-			dir := h.Val()
+	allResults, err := parseSegmentAsConfig(h)
+	if err != nil {
+		return nil, err
+	}
 
-			dirFunc, ok := registeredDirectives[dir]
-			if !ok {
-				return nil, h.Errf("unrecognized directive: %s", dir)
-			}
-
-			subHelper := h
-			subHelper.Dispenser = h.NewFromNextSegment()
-
-			results, err := dirFunc(subHelper)
-			if err != nil {
-				return nil, h.Errf("parsing caddyfile tokens for '%s': %v", dir, err)
-			}
-			for _, result := range results {
-				switch handler := result.Value.(type) {
-				case caddyhttp.Route:
-					sr.Routes = append(sr.Routes, handler)
-				case caddyhttp.Subroute:
-					// directives which return a literal subroute instead of a route
-					// means they intend to keep those handlers together without
-					// them being reordered; we're doing that anyway since we're in
-					// the route directive, so just append its handlers
-					sr.Routes = append(sr.Routes, handler.Routes...)
-				default:
-					return nil, h.Errf("%s directive returned something other than an HTTP route or subroute: %#v (only handler directives can be used in routes)", dir, result.Value)
-				}
-			}
+	for _, result := range allResults {
+		switch handler := result.Value.(type) {
+		case caddyhttp.Route:
+			sr.Routes = append(sr.Routes, handler)
+		case caddyhttp.Subroute:
+			// directives which return a literal subroute instead of a route
+			// means they intend to keep those handlers together without
+			// them being reordered; we're doing that anyway since we're in
+			// the route directive, so just append its handlers
+			sr.Routes = append(sr.Routes, handler.Routes...)
+		default:
+			return nil, h.Errf("%s directive returned something other than an HTTP route or subroute: %#v (only handler directives can be used in routes)", result.directive, result.Value)
 		}
 	}
 
@@ -434,11 +541,11 @@ func parseRoute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 }
 
 func parseHandle(h Helper) (caddyhttp.MiddlewareHandler, error) {
-	return parseSegmentAsSubroute(h)
+	return ParseSegmentAsSubroute(h)
 }
 
 func parseHandleErrors(h Helper) ([]ConfigValue, error) {
-	subroute, err := parseSegmentAsSubroute(h)
+	subroute, err := ParseSegmentAsSubroute(h)
 	if err != nil {
 		return nil, err
 	}
@@ -461,6 +568,11 @@ func parseHandleErrors(h Helper) ([]ConfigValue, error) {
 func parseLog(h Helper) ([]ConfigValue, error) {
 	var configValues []ConfigValue
 	for h.Next() {
+		// log does not currently support any arguments
+		if h.NextArg() {
+			return nil, h.ArgErr()
+		}
+
 		cl := new(caddy.CustomLog)
 
 		for h.NextBlock(0) {

caddyconfig/httpcaddyfile/builtins_test.go

@@ -0,0 +1,62 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	_ "github.com/caddyserver/caddy/v2/modules/logging"
+)
+
+func TestLogDirectiveSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectWarn  bool
+		expectError bool
+	}{
+		{
+			input: `:8080 {
+				log
+			}
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log {
+					output file foo.log
+				}
+			}
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log /foo {
+					output file foo.log
+				}
+			}
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if len(warnings) > 0 != tc.expectWarn {
+			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
+			continue
+		}
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+	}
+}

caddyconfig/httpcaddyfile/directives.go

@@ -37,6 +37,7 @@ import (
 // The header directive goes second so that headers
 // can be manipulated before doing redirects.
 var directiveOrder = []string{
+	"map",
 	"root",
 
 	"header",
@@ -54,15 +55,19 @@ var directiveOrder = []string{
 	"encode",
 	"templates",
 
-	// special routing directives
+	// special routing & dispatching directives
 	"handle",
+	"handle_path",
 	"route",
+	"push",
 
 	// handlers that typically respond to requests
 	"respond",
+	"metrics",
 	"reverse_proxy",
 	"php_fastcgi",
 	"file_server",
+	"acme_server",
 }
 
 // directiveIsOrdered returns true if dir is
@@ -120,6 +125,17 @@ func RegisterHandlerDirective(dir string, setupFunc UnmarshalHandlerFunc) {
 	})
 }
 
+// RegisterGlobalOption registers a unique global option opt with
+// an associated unmarshaling (setup) function. When the global
+// option opt is encountered in a Caddyfile, setupFunc will be
+// called to unmarshal its tokens.
+func RegisterGlobalOption(opt string, setupFunc UnmarshalGlobalFunc) {
+	if _, ok := registeredGlobalOptions[opt]; ok {
+		panic("global option " + opt + " already registered")
+	}
+	registeredGlobalOptions[opt] = setupFunc
+}
+
 // Helper is a type which helps setup a value from
 // Caddyfile tokens.
 type Helper struct {
@@ -250,89 +266,30 @@ func (h Helper) NewBindAddresses(addrs []string) []ConfigValue {
 	return []ConfigValue{{Class: "bind", Value: addrs}}
 }
 
-// ConfigValue represents a value to be added to the final
-// configuration, or a value to be consulted when building
-// the final configuration.
-type ConfigValue struct {
-	// The kind of value this is. As the config is
-	// being built, the adapter will look in the
-	// "pile" for values belonging to a certain
-	// class when it is setting up a certain part
-	// of the config. The associated value will be
-	// type-asserted and placed accordingly.
-	Class string
-
-	// The value to be used when building the config.
-	// Generally its type is associated with the
-	// name of the Class.
-	Value interface{}
-
-	directive string
-}
-
-func sortRoutes(routes []ConfigValue) {
-	dirPositions := make(map[string]int)
-	for i, dir := range directiveOrder {
-		dirPositions[dir] = i
-	}
-
-	// while we are sorting, we will need to decode a route's path matcher
-	// in order to sub-sort by path length; we can amortize this operation
-	// for efficiency by storing the decoded matchers in a slice
-	decodedMatchers := make([]caddyhttp.MatchPath, len(routes))
-
-	sort.SliceStable(routes, func(i, j int) bool {
-		iDir, jDir := routes[i].directive, routes[j].directive
-		if iDir == jDir {
-			// directives are the same; sub-sort by path matcher length
-			// if there's only one matcher set and one path (common case)
-			iRoute, ok := routes[i].Value.(caddyhttp.Route)
-			if !ok {
-				return false
-			}
-			jRoute, ok := routes[j].Value.(caddyhttp.Route)
-			if !ok {
-				return false
-			}
-
-			// use already-decoded matcher, or decode if it's the first time seeing it
-			iPM, jPM := decodedMatchers[i], decodedMatchers[j]
-			if iPM == nil && len(iRoute.MatcherSetsRaw) == 1 {
-				var pathMatcher caddyhttp.MatchPath
-				_ = json.Unmarshal(iRoute.MatcherSetsRaw[0]["path"], &pathMatcher)
-				decodedMatchers[i] = pathMatcher
-				iPM = pathMatcher
-			}
-			if jPM == nil && len(jRoute.MatcherSetsRaw) == 1 {
-				var pathMatcher caddyhttp.MatchPath
-				_ = json.Unmarshal(jRoute.MatcherSetsRaw[0]["path"], &pathMatcher)
-				decodedMatchers[j] = pathMatcher
-				jPM = pathMatcher
-			}
-
-			// sort by longer path (more specific) first; missing
-			// path matchers are treated as zero-length paths
-			var iPathLen, jPathLen int
-			if iPM != nil {
-				iPathLen = len(iPM[0])
-			}
-			if jPM != nil {
-				jPathLen = len(jPM[0])
-			}
-			return iPathLen > jPathLen
-		}
-
-		return dirPositions[iDir] < dirPositions[jDir]
-	})
-}
-
-// parseSegmentAsSubroute parses the segment such that its subdirectives
+// ParseSegmentAsSubroute parses the segment such that its subdirectives
 // are themselves treated as directives, from which a subroute is built
 // and returned.
-func parseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
+func ParseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	allResults, err := parseSegmentAsConfig(h)
+	if err != nil {
+		return nil, err
+	}
+
+	return buildSubroute(allResults, h.groupCounter)
+}
+
+// parseSegmentAsConfig parses the segment such that its subdirectives
+// are themselves treated as directives, including named matcher definitions,
+// and the raw Config structs are returned.
+func parseSegmentAsConfig(h Helper) ([]ConfigValue, error) {
 	var allResults []ConfigValue
 
 	for h.Next() {
+		// don't allow non-matcher args on the first line
+		if h.NextArg() {
+			return nil, h.ArgErr()
+		}
+
 		// slice the linear list of tokens into top-level segments
 		var segments []caddyfile.Segment
 		for nesting := h.Nesting(); h.NextBlock(nesting); {
@@ -347,13 +304,17 @@ func parseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 		}
 
 		// find and extract any embedded matcher definitions in this scope
-		for i, seg := range segments {
+		for i := 0; i < len(segments); i++ {
+			seg := segments[i]
 			if strings.HasPrefix(seg.Directive(), matcherPrefix) {
+				// parse, then add the matcher to matcherDefs
 				err := parseMatcherDefinitions(caddyfile.NewDispenser(seg), matcherDefs)
 				if err != nil {
 					return nil, err
 				}
+				// remove the matcher segment (consumed), then step back the loop
 				segments = append(segments[:i], segments[i+1:]...)
+				i--
 			}
 		}
 
@@ -380,7 +341,83 @@ func parseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 		}
 	}
 
-	return buildSubroute(allResults, h.groupCounter)
+	return allResults, nil
+}
+
+// ConfigValue represents a value to be added to the final
+// configuration, or a value to be consulted when building
+// the final configuration.
+type ConfigValue struct {
+	// The kind of value this is. As the config is
+	// being built, the adapter will look in the
+	// "pile" for values belonging to a certain
+	// class when it is setting up a certain part
+	// of the config. The associated value will be
+	// type-asserted and placed accordingly.
+	Class string
+
+	// The value to be used when building the config.
+	// Generally its type is associated with the
+	// name of the Class.
+	Value interface{}
+
+	directive string
+}
+
+func sortRoutes(routes []ConfigValue) {
+	dirPositions := make(map[string]int)
+	for i, dir := range directiveOrder {
+		dirPositions[dir] = i
+	}
+
+	sort.SliceStable(routes, func(i, j int) bool {
+		// if the directives are different, just use the established directive order
+		iDir, jDir := routes[i].directive, routes[j].directive
+		if iDir != jDir {
+			return dirPositions[iDir] < dirPositions[jDir]
+		}
+
+		// directives are the same; sub-sort by path matcher length if there's
+		// only one matcher set and one path (this is a very common case and
+		// usually -- but not always -- helpful/expected, oh well; user can
+		// always take manual control of order using handler or route blocks)
+		iRoute, ok := routes[i].Value.(caddyhttp.Route)
+		if !ok {
+			return false
+		}
+		jRoute, ok := routes[j].Value.(caddyhttp.Route)
+		if !ok {
+			return false
+		}
+
+		// decode the path matchers, if there is just one of them
+		var iPM, jPM caddyhttp.MatchPath
+		if len(iRoute.MatcherSetsRaw) == 1 {
+			_ = json.Unmarshal(iRoute.MatcherSetsRaw[0]["path"], &iPM)
+		}
+		if len(jRoute.MatcherSetsRaw) == 1 {
+			_ = json.Unmarshal(jRoute.MatcherSetsRaw[0]["path"], &jPM)
+		}
+
+		// sort by longer path (more specific) first; missing path
+		// matchers or multi-matchers are treated as zero-length paths
+		var iPathLen, jPathLen int
+		if len(iPM) > 0 {
+			iPathLen = len(iPM[0])
+		}
+		if len(jPM) > 0 {
+			jPathLen = len(jPM[0])
+		}
+
+		// if both directives have no path matcher, use whichever one
+		// has any kind of matcher defined first.
+		if iPathLen == 0 && jPathLen == 0 {
+			return len(iRoute.MatcherSetsRaw) > 0 && len(jRoute.MatcherSetsRaw) == 0
+		}
+
+		// sort with the most-specific (longest) path first
+		return iPathLen > jPathLen
+	})
 }
 
 // serverBlock pairs a Caddyfile server block with
@@ -462,6 +499,13 @@ type (
 	// for you. These are passed to a call to
 	// RegisterHandlerDirective.
 	UnmarshalHandlerFunc func(h Helper) (caddyhttp.MiddlewareHandler, error)
+
+	// UnmarshalGlobalFunc is a function which can unmarshal Caddyfile
+	// tokens into a global option config value using a Helper type.
+	// These are passed in a call to RegisterGlobalOption.
+	UnmarshalGlobalFunc func(d *caddyfile.Dispenser) (interface{}, error)
 )
 
 var registeredDirectives = make(map[string]UnmarshalFunc)
+
+var registeredGlobalOptions = make(map[string]UnmarshalGlobalFunc)

caddyconfig/httpcaddyfile/directives_test.go

@@ -14,62 +14,62 @@ func TestHostsFromKeys(t *testing.T) {
 	}{
 		{
 			[]Address{
-				Address{Original: "foo", Host: "foo"},
+				{Original: "foo", Host: "foo"},
 			},
 			[]string{"foo"},
 			[]string{"foo"},
 		},
 		{
 			[]Address{
-				Address{Original: "foo", Host: "foo"},
-				Address{Original: "bar", Host: "bar"},
+				{Original: "foo", Host: "foo"},
+				{Original: "bar", Host: "bar"},
 			},
 			[]string{"bar", "foo"},
 			[]string{"bar", "foo"},
 		},
 		{
 			[]Address{
-				Address{Original: ":2015", Port: "2015"},
+				{Original: ":2015", Port: "2015"},
 			},
 			[]string{}, []string{},
 		},
 		{
 			[]Address{
-				Address{Original: ":443", Port: "443"},
+				{Original: ":443", Port: "443"},
 			},
 			[]string{}, []string{},
 		},
 		{
 			[]Address{
-				Address{Original: "foo", Host: "foo"},
-				Address{Original: ":2015", Port: "2015"},
+				{Original: "foo", Host: "foo"},
+				{Original: ":2015", Port: "2015"},
 			},
 			[]string{}, []string{"foo"},
 		},
 		{
 			[]Address{
-				Address{Original: "example.com:2015", Host: "example.com", Port: "2015"},
+				{Original: "example.com:2015", Host: "example.com", Port: "2015"},
 			},
 			[]string{"example.com"},
 			[]string{"example.com:2015"},
 		},
 		{
 			[]Address{
-				Address{Original: "example.com:80", Host: "example.com", Port: "80"},
+				{Original: "example.com:80", Host: "example.com", Port: "80"},
 			},
 			[]string{"example.com"},
 			[]string{"example.com"},
 		},
 		{
 			[]Address{
-				Address{Original: "https://:2015/foo", Scheme: "https", Port: "2015", Path: "/foo"},
+				{Original: "https://:2015/foo", Scheme: "https", Port: "2015", Path: "/foo"},
 			},
 			[]string{},
 			[]string{},
 		},
 		{
 			[]Address{
-				Address{Original: "https://example.com:2015/foo", Scheme: "https", Host: "example.com", Port: "2015", Path: "/foo"},
+				{Original: "https://example.com:2015/foo", Scheme: "https", Host: "example.com", Port: "2015", Path: "/foo"},
 			},
 			[]string{"example.com"},
 			[]string{"example.com:2015"},

caddyconfig/httpcaddyfile/httptype.go

@@ -18,6 +18,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"reflect"
+	"regexp"
 	"sort"
 	"strconv"
 	"strings"
@@ -38,7 +39,7 @@ type ServerType struct {
 }
 
 // Setup makes a config from the tokens.
-func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
+func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 	options map[string]interface{}) (*caddy.Config, []caddyconfig.Warning, error) {
 	var warnings []caddyconfig.Warning
 	gc := counter{new(int)}
@@ -50,15 +51,18 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 	// chosen to handle a request - we actually will make each
 	// server block's route terminal so that only one will run
 	sbKeys := make(map[string]struct{})
-	serverBlocks := make([]serverBlock, 0, len(originalServerBlocks))
-	for i, sblock := range originalServerBlocks {
+	originalServerBlocks := make([]serverBlock, 0, len(inputServerBlocks))
+	for i, sblock := range inputServerBlocks {
 		for j, k := range sblock.Keys {
+			if j == 0 && strings.HasPrefix(k, "@") {
+				return nil, warnings, fmt.Errorf("cannot define a matcher outside of a site block: '%s'", k)
+			}
 			if _, ok := sbKeys[k]; ok {
 				return nil, warnings, fmt.Errorf("duplicate site address not allowed: '%s' in %v (site block %d, key %d)", k, sblock.Keys, i, j)
 			}
 			sbKeys[k] = struct{}{}
 		}
-		serverBlocks = append(serverBlocks, serverBlock{
+		originalServerBlocks = append(originalServerBlocks, serverBlock{
 			block: sblock,
 			pile:  make(map[string][]ConfigValue),
 		})
@@ -66,39 +70,60 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 
 	// apply any global options
 	var err error
-	serverBlocks, err = st.evaluateGlobalOptionsBlock(serverBlocks, options)
+	originalServerBlocks, err = st.evaluateGlobalOptionsBlock(originalServerBlocks, options)
 	if err != nil {
 		return nil, warnings, err
 	}
 
-	for _, sb := range serverBlocks {
-		// replace shorthand placeholders (which are
-		// convenient when writing a Caddyfile) with
-		// their actual placeholder identifiers or
-		// variable names
-		replacer := strings.NewReplacer(
-			"{dir}", "{http.request.uri.path.dir}",
-			"{file}", "{http.request.uri.path.file}",
-			"{host}", "{http.request.host}",
-			"{hostport}", "{http.request.hostport}",
-			"{method}", "{http.request.method}",
-			"{path}", "{http.request.uri.path}",
-			"{query}", "{http.request.uri.query}",
-			"{remote}", "{http.request.remote}",
-			"{remote_host}", "{http.request.remote.host}",
-			"{remote_port}", "{http.request.remote.port}",
-			"{scheme}", "{http.request.scheme}",
-			"{uri}", "{http.request.uri}",
-			"{tls_cipher}", "{http.request.tls.cipher_suite}",
-			"{tls_version}", "{http.request.tls.version}",
-			"{tls_client_fingerprint}", "{http.request.tls.client.fingerprint}",
-			"{tls_client_issuer}", "{http.request.tls.client.issuer}",
-			"{tls_client_serial}", "{http.request.tls.client.serial}",
-			"{tls_client_subject}", "{http.request.tls.client.subject}",
-		)
+	// replace shorthand placeholders (which are
+	// convenient when writing a Caddyfile) with
+	// their actual placeholder identifiers or
+	// variable names
+	replacer := strings.NewReplacer(
+		"{dir}", "{http.request.uri.path.dir}",
+		"{file}", "{http.request.uri.path.file}",
+		"{host}", "{http.request.host}",
+		"{hostport}", "{http.request.hostport}",
+		"{port}", "{http.request.port}",
+		"{method}", "{http.request.method}",
+		"{path}", "{http.request.uri.path}",
+		"{query}", "{http.request.uri.query}",
+		"{remote}", "{http.request.remote}",
+		"{remote_host}", "{http.request.remote.host}",
+		"{remote_port}", "{http.request.remote.port}",
+		"{scheme}", "{http.request.scheme}",
+		"{uri}", "{http.request.uri}",
+		"{tls_cipher}", "{http.request.tls.cipher_suite}",
+		"{tls_version}", "{http.request.tls.version}",
+		"{tls_client_fingerprint}", "{http.request.tls.client.fingerprint}",
+		"{tls_client_issuer}", "{http.request.tls.client.issuer}",
+		"{tls_client_serial}", "{http.request.tls.client.serial}",
+		"{tls_client_subject}", "{http.request.tls.client.subject}",
+	)
+
+	// these are placeholders that allow a user-defined final
+	// parameters, but we still want to provide a shorthand
+	// for those, so we use a regexp to replace
+	regexpReplacements := []struct {
+		search  *regexp.Regexp
+		replace string
+	}{
+		{regexp.MustCompile(`{query\.([\w-]*)}`), "{http.request.uri.query.$1}"},
+		{regexp.MustCompile(`{labels\.([\w-]*)}`), "{http.request.host.labels.$1}"},
+		{regexp.MustCompile(`{header\.([\w-]*)}`), "{http.request.header.$1}"},
+		{regexp.MustCompile(`{path\.([\w-]*)}`), "{http.request.uri.path.$1}"},
+		{regexp.MustCompile(`{re\.([\w-]*)\.([\w-]*)}`), "{http.regexp.$1.$2}"},
+	}
+
+	for _, sb := range originalServerBlocks {
 		for _, segment := range sb.block.Segments {
 			for i := 0; i < len(segment); i++ {
+				// simple string replacements
 				segment[i].Text = replacer.Replace(segment[i].Text)
+				// complex regexp replacements
+				for _, r := range regexpReplacements {
+					segment[i].Text = r.search.ReplaceAllString(segment[i].Text, r.replace)
+				}
 			}
 		}
 
@@ -147,6 +172,15 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 			if err != nil {
 				return nil, warnings, fmt.Errorf("parsing caddyfile tokens for '%s': %v", dir, err)
 			}
+
+			// As a special case, we want "handle_path" to be sorted
+			// at the same level as "handle", so we force them to use
+			// the same directive name after their parsing is complete.
+			// See https://github.com/caddyserver/caddy/issues/3675#issuecomment-678042377
+			if dir == "handle_path" {
+				dir = "handle"
+			}
+
 			for _, result := range results {
 				result.directive = dir
 				sb.pile[result.Class] = append(sb.pile[result.Class], result)
@@ -155,7 +189,7 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 	}
 
 	// map
-	sbmap, err := st.mapAddressToServerBlocks(serverBlocks, options)
+	sbmap, err := st.mapAddressToServerBlocks(originalServerBlocks, options)
 	if err != nil {
 		return nil, warnings, err
 	}
@@ -193,21 +227,24 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 	// extract any custom logs, and enforce configured levels
 	var customLogs []namedCustomLog
 	var hasDefaultLog bool
-	for _, sb := range serverBlocks {
-		for _, clVal := range sb.pile["custom_log"] {
-			ncl := clVal.Value.(namedCustomLog)
-			if ncl.name == "" {
-				continue
+	for _, p := range pairings {
+		for _, sb := range p.serverBlocks {
+			for _, clVal := range sb.pile["custom_log"] {
+				ncl := clVal.Value.(namedCustomLog)
+				if ncl.name == "" {
+					continue
+				}
+				if ncl.name == "default" {
+					hasDefaultLog = true
+				}
+				if _, ok := options["debug"]; ok && ncl.log.Level == "" {
+					ncl.log.Level = "DEBUG"
+				}
+				customLogs = append(customLogs, ncl)
 			}
-			if ncl.name == "default" {
-				hasDefaultLog = true
-			}
-			if _, ok := options["debug"]; ok && ncl.log.Level == "" {
-				ncl.log.Level = "DEBUG"
-			}
-			customLogs = append(customLogs, ncl)
 		}
 	}
+
 	if !hasDefaultLog {
 		// if the default log was not customized, ensure we
 		// configure it with any applicable options
@@ -233,12 +270,8 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 			storageCvtr.(caddy.Module).CaddyModule().ID.Name(),
 			&warnings)
 	}
-	if adminConfig, ok := options["admin"].(string); ok && adminConfig != "" {
-		if adminConfig == "off" {
-			cfg.Admin = &caddy.AdminConfig{Disabled: true}
-		} else {
-			cfg.Admin = &caddy.AdminConfig{Listen: adminConfig}
-		}
+	if adminConfig, ok := options["admin"].(*caddy.AdminConfig); ok && adminConfig != nil {
+		cfg.Admin = adminConfig
 	}
 	if len(customLogs) > 0 {
 		if cfg.Logging == nil {
@@ -250,17 +283,16 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 			if ncl.name != "" {
 				cfg.Logging.Logs[ncl.name] = ncl.log
 			}
-		}
-	}
-	if len(customLogs) > 0 {
-		if cfg.Logging == nil {
-			cfg.Logging = &caddy.Logging{
-				Logs: make(map[string]*caddy.CustomLog),
-			}
-		}
-		for _, ncl := range customLogs {
-			if ncl.name != "" {
-				cfg.Logging.Logs[ncl.name] = ncl.log
+			// most users seem to prefer not writing access logs
+			// to the default log when they are directed to a
+			// file or have any other special customization
+			if len(ncl.log.Include) > 0 {
+				defaultLog, ok := cfg.Logging.Logs["default"]
+				if !ok {
+					defaultLog = new(caddy.CustomLog)
+					cfg.Logging.Logs["default"] = defaultLog
+				}
+				defaultLog.Exclude = append(defaultLog.Exclude, ncl.log.Include...)
 			}
 		}
 	}
@@ -282,39 +314,18 @@ func (ServerType) evaluateGlobalOptionsBlock(serverBlocks []serverBlock, options
 		var val interface{}
 		var err error
 		disp := caddyfile.NewDispenser(segment)
-		switch dir {
-		case "debug":
-			val = true
-		case "http_port":
-			val, err = parseOptHTTPPort(disp)
-		case "https_port":
-			val, err = parseOptHTTPSPort(disp)
-		case "default_sni":
-			val, err = parseOptSingleString(disp)
-		case "order":
-			val, err = parseOptOrder(disp)
-		case "experimental_http3":
-			val, err = parseOptExperimentalHTTP3(disp)
-		case "storage":
-			val, err = parseOptStorage(disp)
-		case "acme_ca", "acme_dns", "acme_ca_root":
-			val, err = parseOptSingleString(disp)
-		case "email":
-			val, err = parseOptSingleString(disp)
-		case "admin":
-			val, err = parseOptAdmin(disp)
-		case "on_demand_tls":
-			val, err = parseOptOnDemand(disp)
-		case "local_certs":
-			val = true
-		case "key_type":
-			val, err = parseOptSingleString(disp)
-		default:
-			return nil, fmt.Errorf("unrecognized parameter name: %s", dir)
+
+		dirFunc, ok := registeredGlobalOptions[dir]
+		if !ok {
+			tkn := segment[0]
+			return nil, fmt.Errorf("%s:%d: unrecognized global option: %s", tkn.File, tkn.Line, dir)
 		}
+
+		val, err = dirFunc(disp)
 		if err != nil {
-			return nil, fmt.Errorf("%s: %v", dir, err)
+			return nil, fmt.Errorf("parsing caddyfile tokens for '%s': %v", dir, err)
 		}
+
 		options[dir] = val
 	}
 
@@ -336,12 +347,31 @@ func (st *ServerType) serversFromPairings(
 	if hp, ok := options["http_port"].(int); ok {
 		httpPort = strconv.Itoa(hp)
 	}
+	httpsPort := strconv.Itoa(caddyhttp.DefaultHTTPSPort)
+	if hsp, ok := options["https_port"].(int); ok {
+		httpsPort = strconv.Itoa(hsp)
+	}
+	autoHTTPS := "on"
+	if ah, ok := options["auto_https"].(string); ok {
+		autoHTTPS = ah
+	}
 
 	for i, p := range pairings {
 		srv := &caddyhttp.Server{
 			Listen: p.addresses,
 		}
 
+		// handle the auto_https global option
+		if autoHTTPS != "on" {
+			srv.AutoHTTPS = new(caddyhttp.AutoHTTPSConfig)
+			if autoHTTPS == "off" {
+				srv.AutoHTTPS.Disabled = true
+			}
+			if autoHTTPS == "disable_redirects" {
+				srv.AutoHTTPS.DisableRedir = true
+			}
+		}
+
 		// sort server blocks by their keys; this is important because
 		// only the first matching site should be evaluated, and we should
 		// attempt to match most specific site first (host and path), in
@@ -352,7 +382,11 @@ func (st *ServerType) serversFromPairings(
 			// but I don't expect many blocks will have THAT many keys...
 			var iLongestPath, jLongestPath string
 			var iLongestHost, jLongestHost string
+			var iWildcardHost, jWildcardHost bool
 			for _, addr := range p.serverBlocks[i].keys {
+				if strings.Contains(addr.Host, "*.") {
+					iWildcardHost = true
+				}
 				if specificity(addr.Host) > specificity(iLongestHost) {
 					iLongestHost = addr.Host
 				}
@@ -361,6 +395,9 @@ func (st *ServerType) serversFromPairings(
 				}
 			}
 			for _, addr := range p.serverBlocks[j].keys {
+				if strings.Contains(addr.Host, "*.") {
+					jWildcardHost = true
+				}
 				if specificity(addr.Host) > specificity(jLongestHost) {
 					jLongestHost = addr.Host
 				}
@@ -368,13 +405,25 @@ func (st *ServerType) serversFromPairings(
 					jLongestPath = addr.Path
 				}
 			}
+			if specificity(jLongestHost) == 0 {
+				// catch-all blocks (blocks with no hostname) should always go
+				// last, even after blocks with wildcard hosts
+				return true
+			}
+			if iWildcardHost != jWildcardHost {
+				// site blocks that have a key with a wildcard in the hostname
+				// must always be less specific than blocks without one; see
+				// https://github.com/caddyserver/caddy/issues/3410
+				return jWildcardHost && !iWildcardHost
+			}
 			if specificity(iLongestHost) == specificity(jLongestHost) {
 				return len(iLongestPath) > len(jLongestPath)
 			}
 			return specificity(iLongestHost) > specificity(jLongestHost)
 		})
 
-		var hasCatchAllTLSConnPolicy, usesTLS bool
+		var hasCatchAllTLSConnPolicy, addressQualifiesForTLS bool
+		autoHTTPSWillAddConnPolicy := autoHTTPS != "off"
 
 		// create a subroute for each site in the server block
 		for _, sblock := range p.serverBlocks {
@@ -406,16 +455,19 @@ func (st *ServerType) serversFromPairings(
 						}
 					} else {
 						cp.DefaultSNI = defaultSNI
-						hasCatchAllTLSConnPolicy = true
 					}
 
-					srv.TLSConnPolicies = append(srv.TLSConnPolicies, cp)
+					// only append this policy if it actually changes something
+					if !cp.SettingsEmpty() {
+						srv.TLSConnPolicies = append(srv.TLSConnPolicies, cp)
+						hasCatchAllTLSConnPolicy = len(hosts) == 0
+					}
 				}
 			}
 
-			// exclude any hosts that were defined explicitly with
-			// "http://" in the key from automated cert management (issue #2998)
 			for _, addr := range sblock.keys {
+				// exclude any hosts that were defined explicitly with "http://"
+				// in the key from automated cert management (issue #2998)
 				if addr.Scheme == "http" && addr.Host != "" {
 					if srv.AutoHTTPS == nil {
 						srv.AutoHTTPS = new(caddyhttp.AutoHTTPSConfig)
@@ -424,9 +476,30 @@ func (st *ServerType) serversFromPairings(
 						srv.AutoHTTPS.Skip = append(srv.AutoHTTPS.Skip, addr.Host)
 					}
 				}
-				if addr.Scheme != "http" && addr.Host != "" && addr.Port != httpPort {
-					usesTLS = true
+				// we'll need to remember if the address qualifies for auto-HTTPS, so we
+				// can add a TLS conn policy if necessary
+				if addr.Scheme == "https" ||
+					(addr.Scheme != "http" && addr.Host != "" && addr.Port != httpPort) {
+					addressQualifiesForTLS = true
 				}
+				// predict whether auto-HTTPS will add the conn policy for us; if so, we
+				// may not need to add one for this server
+				autoHTTPSWillAddConnPolicy = autoHTTPSWillAddConnPolicy &&
+					(addr.Port == httpsPort || (addr.Port != httpPort && addr.Host != ""))
+			}
+
+			// Look for any config values that provide listener wrappers on the server block
+			for _, listenerConfig := range sblock.pile["listener_wrapper"] {
+				listenerWrapper, ok := listenerConfig.Value.(caddy.ListenerWrapper)
+				if !ok {
+					return nil, fmt.Errorf("config for a listener wrapper did not provide a value that implements caddy.ListenerWrapper")
+				}
+				jsonListenerWrapper := caddyconfig.JSONModuleObject(
+					listenerWrapper,
+					"wrapper",
+					listenerWrapper.(caddy.Module).CaddyModule().ID.Name(),
+					warnings)
+				srv.ListenerWrappersRaw = append(srv.ListenerWrappersRaw, jsonListenerWrapper)
 			}
 
 			// set up each handler directive, making sure to honor directive order
@@ -451,23 +524,46 @@ func (st *ServerType) serversFromPairings(
 			}
 
 			// add log associations
+			// see https://github.com/caddyserver/caddy/issues/3310
+			sblockLogHosts := sblock.hostsFromKeys(true)
 			for _, cval := range sblock.pile["custom_log"] {
 				ncl := cval.Value.(namedCustomLog)
 				if srv.Logs == nil {
-					srv.Logs = &caddyhttp.ServerLogConfig{
-						LoggerNames: make(map[string]string),
-					}
+					srv.Logs = new(caddyhttp.ServerLogConfig)
 				}
 				if sblock.hasHostCatchAllKey() {
-					srv.Logs.LoggerName = ncl.name
+					// all requests for hosts not able to be listed should use
+					// this log because it's a catch-all-hosts server block
+					srv.Logs.DefaultLoggerName = ncl.name
 				} else {
-					for _, h := range sblock.hostsFromKeys(true) {
-						if ncl.name != "" {
+					// map each host to the user's desired logger name
+					for _, h := range sblockLogHosts {
+						// if the custom logger name is non-empty, add it to
+						// the map; otherwise, only map to an empty logger
+						// name if the server block has a catch-all host (in
+						// which case only requests with mapped hostnames will
+						// be access-logged, so it'll be necessary to add them
+						// to the map even if they use default logger)
+						if ncl.name != "" || len(hosts) == 0 {
+							if srv.Logs.LoggerNames == nil {
+								srv.Logs.LoggerNames = make(map[string]string)
+							}
 							srv.Logs.LoggerNames[h] = ncl.name
 						}
 					}
 				}
 			}
+			if srv.Logs != nil && len(sblock.pile["custom_log"]) == 0 {
+				// server has access logs enabled, but this server block does not
+				// enable access logs; therefore, all hosts of this server block
+				// should not be access-logged
+				if len(hosts) == 0 {
+					// if the server block has a catch-all-hosts key, then we should
+					// not log reqs to any host unless it appears in the map
+					srv.Logs.SkipUnmappedHosts = true
+				}
+				srv.Logs.SkipHosts = append(srv.Logs.SkipHosts, sblockLogHosts...)
+			}
 		}
 
 		// a server cannot (natively) serve both HTTP and HTTPS at the
@@ -489,9 +585,9 @@ func (st *ServerType) serversFromPairings(
 		// TODO: maybe a smarter way to handle this might be to just make the
 		// auto-HTTPS logic at provision-time detect if there is any connection
 		// policy missing for any HTTPS-enabled hosts, if so, add it... maybe?
-		if usesTLS &&
+		if addressQualifiesForTLS &&
 			!hasCatchAllTLSConnPolicy &&
-			(len(srv.TLSConnPolicies) > 0 || defaultSNI != "") {
+			(len(srv.TLSConnPolicies) > 0 || !autoHTTPSWillAddConnPolicy || defaultSNI != "") {
 			srv.TLSConnPolicies = append(srv.TLSConnPolicies, &caddytls.ConnectionPolicy{DefaultSNI: defaultSNI})
 		}
 
@@ -551,7 +647,7 @@ func detectConflictingSchemes(srv *caddyhttp.Server, serverBlocks []serverBlock,
 				if err := checkAndSetHTTP(addr); err != nil {
 					return err
 				}
-			} else if addr.Scheme == "https" || addr.Port == httpsPort {
+			} else if addr.Scheme == "https" || addr.Port == httpsPort || len(srv.TLSConnPolicies) > 0 {
 				if err := checkAndSetHTTPS(addr); err != nil {
 					return err
 				}
@@ -566,21 +662,16 @@ func detectConflictingSchemes(srv *caddyhttp.Server, serverBlocks []serverBlock,
 	return nil
 }
 
-// consolidateConnPolicies removes empty TLS connection policies and combines
-// equivalent ones for a cleaner overall output.
+// consolidateConnPolicies sorts any catch-all policy to the end, removes empty TLS connection
+// policies, and combines equivalent ones for a cleaner overall output.
 func consolidateConnPolicies(cps caddytls.ConnectionPolicies) (caddytls.ConnectionPolicies, error) {
-	empty := new(caddytls.ConnectionPolicy)
+	// catch-all policies (those without any matcher) should be at the
+	// end, otherwise it nullifies any more specific policies
+	sort.SliceStable(cps, func(i, j int) bool {
+		return cps[j].MatchersRaw == nil && cps[i].MatchersRaw != nil
+	})
 
 	for i := 0; i < len(cps); i++ {
-		// if the connection policy is empty or has
-		// only matchers, we can remove it entirely
-		empty.MatchersRaw = cps[i].MatchersRaw
-		if reflect.DeepEqual(empty, cps[i]) {
-			cps = append(cps[:i], cps[i+1:]...)
-			i--
-			continue
-		}
-
 		// compare it to the others
 		for j := 0; j < len(cps); j++ {
 			if j == i {
@@ -772,7 +863,18 @@ func buildSubroute(routes []ConfigValue, groupCounter counter) (*caddyhttp.Subro
 		// root directives would overwrite previously-matched ones; they should not cascade
 		"root": {},
 	}
-	for meDir, info := range mutuallyExclusiveDirs {
+
+	// we need to deterministically loop over each of these directives
+	// in order to keep the group numbers consistent
+	keys := make([]string, 0, len(mutuallyExclusiveDirs))
+	for k := range mutuallyExclusiveDirs {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+
+	for _, meDir := range keys {
+		info := mutuallyExclusiveDirs[meDir]
+
 		// see how many instances of the directive there are
 		for _, r := range routes {
 			if r.directive == meDir {
@@ -975,7 +1077,7 @@ func parseMatcherDefinitions(d *caddyfile.Dispenser, matchers map[string]caddy.M
 		// handle more than one segment); otherwise, we'd overwrite other
 		// instances of the matcher in this set
 		tokensByMatcherName := make(map[string][]caddyfile.Token)
-		for nesting := d.Nesting(); d.NextBlock(nesting); {
+		for nesting := d.Nesting(); d.NextArg() || d.NextBlock(nesting); {
 			matcherName := d.Val()
 			tokensByMatcherName[matcherName] = append(tokensByMatcherName[matcherName], d.NextSegment()...)
 		}

caddyconfig/httpcaddyfile/httptype_test.go

@@ -50,6 +50,22 @@ func TestMatcherSyntax(t *testing.T) {
 			expectWarn:  false,
 			expectError: false,
 		},
+		{
+			input: `http://localhost
+			@debug not path /somepath*
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `@matcher {
+				path /matcher-not-allowed/outside-of-site-block/*
+			}
+			http://localhost
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
 	} {
 
 		adapter := caddyfile.Adapter{
@@ -148,6 +164,58 @@ func TestGlobalOptions(t *testing.T) {
 			expectWarn:  false,
 			expectError: true,
 		},
+		{
+			input: `
+				{
+					admin {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 127.0.0.1:2020 {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 192.168.1.1:2020 127.0.0.1:2020 {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
+		{
+			input: `
+				{
+					admin off {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
 	} {
 
 		adapter := caddyfile.Adapter{

caddyconfig/httpcaddyfile/options.go

@@ -16,14 +16,40 @@ package httpcaddyfile
 
 import (
 	"strconv"
-	"time"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
 )
 
-func parseOptHTTPPort(d *caddyfile.Dispenser) (int, error) {
+func init() {
+	RegisterGlobalOption("debug", parseOptTrue)
+	RegisterGlobalOption("http_port", parseOptHTTPPort)
+	RegisterGlobalOption("https_port", parseOptHTTPSPort)
+	RegisterGlobalOption("default_sni", parseOptSingleString)
+	RegisterGlobalOption("order", parseOptOrder)
+	RegisterGlobalOption("experimental_http3", parseOptTrue)
+	RegisterGlobalOption("storage", parseOptStorage)
+	RegisterGlobalOption("acme_ca", parseOptSingleString)
+	RegisterGlobalOption("acme_ca_root", parseOptSingleString)
+	RegisterGlobalOption("acme_dns", parseOptSingleString)
+	RegisterGlobalOption("acme_eab", parseOptACMEEAB)
+	RegisterGlobalOption("cert_issuer", parseOptCertIssuer)
+	RegisterGlobalOption("email", parseOptSingleString)
+	RegisterGlobalOption("admin", parseOptAdmin)
+	RegisterGlobalOption("on_demand_tls", parseOptOnDemand)
+	RegisterGlobalOption("local_certs", parseOptTrue)
+	RegisterGlobalOption("key_type", parseOptSingleString)
+	RegisterGlobalOption("auto_https", parseOptAutoHTTPS)
+}
+
+func parseOptTrue(d *caddyfile.Dispenser) (interface{}, error) {
+	return true, nil
+}
+
+func parseOptHTTPPort(d *caddyfile.Dispenser) (interface{}, error) {
 	var httpPort int
 	for d.Next() {
 		var httpPortStr string
@@ -39,7 +65,7 @@ func parseOptHTTPPort(d *caddyfile.Dispenser) (int, error) {
 	return httpPort, nil
 }
 
-func parseOptHTTPSPort(d *caddyfile.Dispenser) (int, error) {
+func parseOptHTTPSPort(d *caddyfile.Dispenser) (interface{}, error) {
 	var httpsPort int
 	for d.Next() {
 		var httpsPortStr string
@@ -55,11 +81,7 @@ func parseOptHTTPSPort(d *caddyfile.Dispenser) (int, error) {
 	return httpsPort, nil
 }
 
-func parseOptExperimentalHTTP3(d *caddyfile.Dispenser) (bool, error) {
-	return true, nil
-}
-
-func parseOptOrder(d *caddyfile.Dispenser) ([]string, error) {
+func parseOptOrder(d *caddyfile.Dispenser) (interface{}, error) {
 	newOrder := directiveOrder
 
 	for d.Next() {
@@ -135,15 +157,14 @@ func parseOptOrder(d *caddyfile.Dispenser) ([]string, error) {
 	return newOrder, nil
 }
 
-func parseOptStorage(d *caddyfile.Dispenser) (caddy.StorageConverter, error) {
-	if !d.Next() {
+func parseOptStorage(d *caddyfile.Dispenser) (interface{}, error) {
+	if !d.Next() { // consume option name
 		return nil, d.ArgErr()
 	}
-	args := d.RemainingArgs()
-	if len(args) != 1 {
+	if !d.Next() { // get storage module name
 		return nil, d.ArgErr()
 	}
-	modName := args[0]
+	modName := d.Val()
 	mod, err := caddy.GetModule("caddy.storage." + modName)
 	if err != nil {
 		return nil, d.Errf("getting storage module '%s': %v", modName, err)
@@ -163,7 +184,62 @@ func parseOptStorage(d *caddyfile.Dispenser) (caddy.StorageConverter, error) {
 	return storage, nil
 }
 
-func parseOptSingleString(d *caddyfile.Dispenser) (string, error) {
+func parseOptACMEEAB(d *caddyfile.Dispenser) (interface{}, error) {
+	eab := new(acme.EAB)
+	for d.Next() {
+		if d.NextArg() {
+			return nil, d.ArgErr()
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "key_id":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				eab.KeyID = d.Val()
+
+			case "mac_key":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				eab.MACKey = d.Val()
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
+		}
+	}
+	return eab, nil
+}
+
+func parseOptCertIssuer(d *caddyfile.Dispenser) (interface{}, error) {
+	if !d.Next() { // consume option name
+		return nil, d.ArgErr()
+	}
+	if !d.Next() { // get issuer module name
+		return nil, d.ArgErr()
+	}
+	modName := d.Val()
+	mod, err := caddy.GetModule("tls.issuance." + modName)
+	if err != nil {
+		return nil, d.Errf("getting issuer module '%s': %v", modName, err)
+	}
+	unm, ok := mod.New().(caddyfile.Unmarshaler)
+	if !ok {
+		return nil, d.Errf("issuer module '%s' is not a Caddyfile unmarshaler", mod.ID)
+	}
+	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
+	if err != nil {
+		return nil, err
+	}
+	iss, ok := unm.(certmagic.Issuer)
+	if !ok {
+		return nil, d.Errf("module %s is not a certmagic.Issuer", mod.ID)
+	}
+	return iss, nil
+}
+
+func parseOptSingleString(d *caddyfile.Dispenser) (interface{}, error) {
 	d.Next() // consume parameter name
 	if !d.Next() {
 		return "", d.ArgErr()
@@ -175,21 +251,43 @@ func parseOptSingleString(d *caddyfile.Dispenser) (string, error) {
 	return val, nil
 }
 
-func parseOptAdmin(d *caddyfile.Dispenser) (string, error) {
-	if d.Next() {
-		var listenAddress string
-		if !d.AllArgs(&listenAddress) {
-			return "", d.ArgErr()
+func parseOptAdmin(d *caddyfile.Dispenser) (interface{}, error) {
+	adminCfg := new(caddy.AdminConfig)
+	for d.Next() {
+		if d.NextArg() {
+			listenAddress := d.Val()
+			if listenAddress == "off" {
+				adminCfg.Disabled = true
+				if d.Next() { // Do not accept any remaining options including block
+					return nil, d.Err("No more option is allowed after turning off admin config")
+				}
+			} else {
+				adminCfg.Listen = listenAddress
+				if d.NextArg() { // At most 1 arg is allowed
+					return nil, d.ArgErr()
+				}
+			}
 		}
-		if listenAddress == "" {
-			listenAddress = caddy.DefaultAdminListen
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "enforce_origin":
+				adminCfg.EnforceOrigin = true
+
+			case "origins":
+				adminCfg.Origins = d.RemainingArgs()
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
 		}
-		return listenAddress, nil
 	}
-	return "", nil
+	if adminCfg.Listen == "" && !adminCfg.Disabled {
+		adminCfg.Listen = caddy.DefaultAdminListen
+	}
+	return adminCfg, nil
 }
 
-func parseOptOnDemand(d *caddyfile.Dispenser) (*caddytls.OnDemandConfig, error) {
+func parseOptOnDemand(d *caddyfile.Dispenser) (interface{}, error) {
 	var ond *caddytls.OnDemandConfig
 	for d.Next() {
 		if d.NextArg() {
@@ -210,7 +308,7 @@ func parseOptOnDemand(d *caddyfile.Dispenser) (*caddytls.OnDemandConfig, error)
 				if !d.NextArg() {
 					return nil, d.ArgErr()
 				}
-				dur, err := time.ParseDuration(d.Val())
+				dur, err := caddy.ParseDuration(d.Val())
 				if err != nil {
 					return nil, err
 				}
@@ -248,3 +346,18 @@ func parseOptOnDemand(d *caddyfile.Dispenser) (*caddytls.OnDemandConfig, error)
 	}
 	return ond, nil
 }
+
+func parseOptAutoHTTPS(d *caddyfile.Dispenser) (interface{}, error) {
+	d.Next() // consume parameter name
+	if !d.Next() {
+		return "", d.ArgErr()
+	}
+	val := d.Val()
+	if d.Next() {
+		return "", d.ArgErr()
+	}
+	if val != "off" && val != "disable_redirects" {
+		return "", d.Errf("auto_https must be either 'off' or 'disable_redirects'")
+	}
+	return val, nil
+}

caddyconfig/httpcaddyfile/tlsapp.go

@@ -20,11 +20,15 @@ import (
 	"fmt"
 	"reflect"
 	"sort"
+	"strconv"
+	"strings"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
 	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
 )
 
 func (st ServerType) buildTLSApp(
@@ -36,17 +40,26 @@ func (st ServerType) buildTLSApp(
 	tlsApp := &caddytls.TLS{CertificatesRaw: make(caddy.ModuleMap)}
 	var certLoaders []caddytls.CertificateLoader
 
-	// count how many server blocks have a key with no host,
-	// and find all hosts that share a server block with a
-	// hostless key, so that they don't get forgotten/omitted
+	httpsPort := strconv.Itoa(caddyhttp.DefaultHTTPSPort)
+	if hsp, ok := options["https_port"].(int); ok {
+		httpsPort = strconv.Itoa(hsp)
+	}
+
+	// count how many server blocks have a TLS-enabled key with
+	// no host, and find all hosts that share a server block with
+	// a hostless key, so that they don't get forgotten/omitted
 	// by auto-HTTPS (since they won't appear in route matchers)
-	var serverBlocksWithHostlessKey int
+	var serverBlocksWithTLSHostlessKey int
 	hostsSharedWithHostlessKey := make(map[string]struct{})
 	for _, pair := range pairings {
 		for _, sb := range pair.serverBlocks {
 			for _, addr := range sb.keys {
 				if addr.Host == "" {
-					serverBlocksWithHostlessKey++
+					// this address has no hostname, but if it's explicitly set
+					// to HTTPS, then we need to count it as being TLS-enabled
+					if addr.Scheme == "https" || addr.Port == httpsPort {
+						serverBlocksWithTLSHostlessKey++
+					}
 					// this server block has a hostless key, now
 					// go through and add all the hosts to the set
 					for _, otherAddr := range sb.keys {
@@ -123,8 +136,11 @@ func (st ServerType) buildTLSApp(
 				// issuer, skip, since we intend to adjust only ACME issuers
 				var acmeIssuer *caddytls.ACMEIssuer
 				if ap.Issuer != nil {
-					var ok bool
-					if acmeIssuer, ok = ap.Issuer.(*caddytls.ACMEIssuer); !ok {
+					// ensure we include any issuer that embeds/wraps an underlying ACME issuer
+					type acmeCapable interface{ GetACMEIssuer() *caddytls.ACMEIssuer }
+					if acmeWrapper, ok := ap.Issuer.(acmeCapable); ok {
+						acmeIssuer = acmeWrapper.GetACMEIssuer()
+					} else {
 						break
 					}
 				}
@@ -149,9 +165,11 @@ func (st ServerType) buildTLSApp(
 			}
 
 			if ap != nil {
-				// encode issuer now that it's all set up
-				issuerName := ap.Issuer.(caddy.Module).CaddyModule().ID.Name()
-				ap.IssuerRaw = caddyconfig.JSONModuleObject(ap.Issuer, "module", issuerName, &warnings)
+				if ap.Issuer != nil {
+					// encode issuer now that it's all set up
+					issuerName := ap.Issuer.(caddy.Module).CaddyModule().ID.Name()
+					ap.IssuerRaw = caddyconfig.JSONModuleObject(ap.Issuer, "module", issuerName, &warnings)
+				}
 
 				// first make sure this block is allowed to create an automation policy;
 				// doing so is forbidden if it has a key with no host (i.e. ":443")
@@ -163,12 +181,12 @@ func (st ServerType) buildTLSApp(
 				// this is an example of a poor mapping from Caddyfile to JSON but that's
 				// the least-leaky abstraction I could figure out
 				if len(sblockHosts) == 0 {
-					if serverBlocksWithHostlessKey > 1 {
+					if serverBlocksWithTLSHostlessKey > 1 {
 						// this server block and at least one other has a key with no host,
 						// making the two indistinguishable; it is misleading to define such
 						// a policy within one server block since it actually will apply to
 						// others as well
-						return nil, warnings, fmt.Errorf("cannot make a TLS automation policy from a server block that has a host-less address when there are other server block addresses lacking a host")
+						return nil, warnings, fmt.Errorf("cannot make a TLS automation policy from a server block that has a host-less address when there are other TLS-enabled server block addresses lacking a host")
 					}
 					if catchAllAP == nil {
 						// this server block has a key with no hosts, but there is not yet
@@ -220,7 +238,7 @@ func (st ServerType) buildTLSApp(
 			}
 
 			// certificate loaders
-			if clVals, ok := sblock.pile["tls.certificate_loader"]; ok {
+			if clVals, ok := sblock.pile["tls.cert_loader"]; ok {
 				for _, clVal := range clVals {
 					certLoaders = append(certLoaders, clVal.Value.(caddytls.CertificateLoader))
 				}
@@ -293,9 +311,11 @@ func (st ServerType) buildTLSApp(
 
 	// if there is a global/catch-all automation policy, ensure it goes last
 	if catchAllAP != nil {
-		// first, encode its issuer
-		issuerName := catchAllAP.Issuer.(caddy.Module).CaddyModule().ID.Name()
-		catchAllAP.IssuerRaw = caddyconfig.JSONModuleObject(catchAllAP.Issuer, "module", issuerName, &warnings)
+		// first, encode its issuer, if there is one
+		if catchAllAP.Issuer != nil {
+			issuerName := catchAllAP.Issuer.(caddy.Module).CaddyModule().ID.Name()
+			catchAllAP.IssuerRaw = caddyconfig.JSONModuleObject(catchAllAP.Issuer, "module", issuerName, &warnings)
+		}
 
 		// then append it to the end of the policies list
 		if tlsApp.Automation == nil {
@@ -306,6 +326,9 @@ func (st ServerType) buildTLSApp(
 
 	// do a little verification & cleanup
 	if tlsApp.Automation != nil {
+		// consolidate automation policies that are the exact same
+		tlsApp.Automation.Policies = consolidateAutomationPolicies(tlsApp.Automation.Policies)
+
 		// ensure automation policies don't overlap subjects (this should be
 		// an error at provision-time as well, but catch it in the adapt phase
 		// for convenience)
@@ -318,9 +341,6 @@ func (st ServerType) buildTLSApp(
 				automationHostSet[s] = struct{}{}
 			}
 		}
-
-		// consolidate automation policies that are the exact same
-		tlsApp.Automation.Policies = consolidateAutomationPolicies(tlsApp.Automation.Policies)
 	}
 
 	return tlsApp, warnings, nil
@@ -332,14 +352,18 @@ func (st ServerType) buildTLSApp(
 // returned if there are no default/global options. However, if always is
 // true, a non-nil value will always be returned (unless there is an error).
 func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddyconfig.Warning, always bool) (*caddytls.AutomationPolicy, error) {
+	issuer, hasIssuer := options["cert_issuer"]
+
 	acmeCA, hasACMECA := options["acme_ca"]
-	acmeDNS, hasACMEDNS := options["acme_dns"]
 	acmeCARoot, hasACMECARoot := options["acme_ca_root"]
+	acmeDNS, hasACMEDNS := options["acme_dns"]
+	acmeEAB, hasACMEEAB := options["acme_eab"]
+
 	email, hasEmail := options["email"]
 	localCerts, hasLocalCerts := options["local_certs"]
 	keyType, hasKeyType := options["key_type"]
 
-	hasGlobalAutomationOpts := hasACMECA || hasACMEDNS || hasACMECARoot || hasEmail || hasLocalCerts || hasKeyType
+	hasGlobalAutomationOpts := hasIssuer || hasACMECA || hasACMECARoot || hasACMEDNS || hasACMEEAB || hasEmail || hasLocalCerts || hasKeyType
 
 	// if there are no global options related to automation policies
 	// set, then we can just return right away
@@ -351,8 +375,16 @@ func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddycon
 	}
 
 	ap := new(caddytls.AutomationPolicy)
+	if keyType != nil {
+		ap.KeyType = keyType.(string)
+	}
 
-	if localCerts != nil {
+	if hasIssuer {
+		if hasACMECA || hasACMEDNS || hasACMEEAB || hasEmail || hasLocalCerts {
+			return nil, fmt.Errorf("global options are ambiguous: cert_issuer is confusing when combined with acme_*, email, or local_certs options")
+		}
+		ap.Issuer = issuer.(certmagic.Issuer)
+	} else if localCerts != nil {
 		// internal issuer enabled trumps any ACME configurations; useful in testing
 		ap.Issuer = new(caddytls.InternalIssuer) // we'll encode it later
 	} else {
@@ -368,26 +400,41 @@ func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddycon
 		}
 		if acmeDNS != nil {
 			provName := acmeDNS.(string)
-			dnsProvModule, err := caddy.GetModule("tls.dns." + provName)
+			dnsProvModule, err := caddy.GetModule("dns.providers." + provName)
 			if err != nil {
 				return nil, fmt.Errorf("getting DNS provider module named '%s': %v", provName, err)
 			}
 			mgr.Challenges = &caddytls.ChallengesConfig{
-				DNSRaw: caddyconfig.JSONModuleObject(dnsProvModule.New(), "provider", provName, &warnings),
+				DNS: &caddytls.DNSChallengeConfig{
+					ProviderRaw: caddyconfig.JSONModuleObject(dnsProvModule.New(), "name", provName, &warnings),
+				},
 			}
 		}
 		if acmeCARoot != nil {
 			mgr.TrustedRootsPEMFiles = []string{acmeCARoot.(string)}
 		}
-		if keyType != nil {
-			ap.KeyType = keyType.(string)
+		if acmeEAB != nil {
+			mgr.ExternalAccount = acmeEAB.(*acme.EAB)
 		}
-		ap.Issuer = mgr // we'll encode it later
+		ap.Issuer = disambiguateACMEIssuer(mgr) // we'll encode it later
 	}
 
 	return ap, nil
 }
 
+// disambiguateACMEIssuer returns an issuer based on the properties of acmeIssuer.
+// If acmeIssuer implicitly configures a certain kind of ACMEIssuer (for example,
+// ZeroSSL), the proper wrapper over acmeIssuer will be returned instead.
+func disambiguateACMEIssuer(acmeIssuer *caddytls.ACMEIssuer) certmagic.Issuer {
+	// as a special case, we integrate with ZeroSSL's ACME endpoint if it looks like an
+	// implicit ZeroSSL configuration (this requires a  wrapper type over ACMEIssuer
+	// because of the EAB generation; if EAB is provided, we can use plain ACMEIssuer)
+	if strings.Contains(acmeIssuer.CA, "acme.zerossl.com") && acmeIssuer.ExternalAccount == nil {
+		return &caddytls.ZeroSSLIssuer{ACMEIssuer: acmeIssuer}
+	}
+	return acmeIssuer
+}
+
 // consolidateAutomationPolicies combines automation policies that are the same,
 // for a cleaner overall output.
 func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls.AutomationPolicy {
@@ -421,7 +468,12 @@ func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls
 				} else if len(aps[i].Subjects) > 0 && len(aps[j].Subjects) == 0 {
 					aps = append(aps[:i], aps[i+1:]...)
 				} else {
-					aps[i].Subjects = append(aps[i].Subjects, aps[j].Subjects...)
+					// avoid repeated subjects
+					for _, subj := range aps[j].Subjects {
+						if !sliceContains(aps[i].Subjects, subj) {
+							aps[i].Subjects = append(aps[i].Subjects, subj)
+						}
+					}
 					aps = append(aps[:j], aps[j+1:]...)
 				}
 				i--

caddytest/caddy.ca.cer

@@ -0,0 +1,20 @@
+-----BEGIN CERTIFICATE-----
+MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQEL
+BQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkw
+ODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcN
+AQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU
+7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl0
+3WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45t
+wOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNx
+tdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTU
+ApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAd
+BgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS
+2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5u
+NY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkq
+hkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfK
+D66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEO
+fG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnk
+oNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZ
+ks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdle
+Ih6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ==
+-----END CERTIFICATE-----

caddytest/caddytest.go

@@ -11,6 +11,7 @@ import (
 	"log"
 	"net"
 	"net/http"
+	"net/http/cookiejar"
 	"os"
 	"path"
 	"regexp"
@@ -33,12 +34,18 @@ type Defaults struct {
 	AdminPort int
 	// Certificates we expect to be loaded before attempting to run the tests
 	Certifcates []string
+	// TestRequestTimeout is the time to wait for a http request to
+	TestRequestTimeout time.Duration
+	// LoadRequestTimeout is the time to wait for the config to be loaded against the caddy server
+	LoadRequestTimeout time.Duration
 }
 
 // Default testing values
 var Default = Defaults{
-	AdminPort:   2019,
-	Certifcates: []string{"/caddy.localhost.crt", "/caddy.localhost.key"},
+	AdminPort:          2019,
+	Certifcates:        []string{"/caddy.localhost.crt", "/caddy.localhost.key"},
+	TestRequestTimeout: 5 * time.Second,
+	LoadRequestTimeout: 5 * time.Second,
 }
 
 var (
@@ -46,6 +53,32 @@ var (
 	matchCert = regexp.MustCompile(`(/[\w\d\.]+\.crt)`)
 )
 
+// Tester represents an instance of a test client.
+type Tester struct {
+	Client       *http.Client
+	configLoaded bool
+	t            *testing.T
+}
+
+// NewTester will create a new testing client with an attached cookie jar
+func NewTester(t *testing.T) *Tester {
+
+	jar, err := cookiejar.New(nil)
+	if err != nil {
+		t.Fatalf("failed to create cookiejar: %s", err)
+	}
+
+	return &Tester{
+		Client: &http.Client{
+			Transport: CreateTestingTransport(),
+			Jar:       jar,
+			Timeout:   Default.TestRequestTimeout,
+		},
+		configLoaded: false,
+		t:            t,
+	}
+}
+
 type configLoadError struct {
 	Response string
 }
@@ -59,53 +92,54 @@ func timeElapsed(start time.Time, name string) {
 
 // InitServer this will configure the server with a configurion of a specific
 // type. The configType must be either "json" or the adapter type.
-func InitServer(t *testing.T, rawConfig string, configType string) {
+func (tc *Tester) InitServer(rawConfig string, configType string) {
 
-	if err := initServer(t, rawConfig, configType); err != nil {
-		t.Logf("failed to load config: %s", err)
-		t.Fail()
+	if err := tc.initServer(rawConfig, configType); err != nil {
+		tc.t.Logf("failed to load config: %s", err)
+		tc.t.Fail()
 	}
 }
 
 // InitServer this will configure the server with a configurion of a specific
 // type. The configType must be either "json" or the adapter type.
-func initServer(t *testing.T, rawConfig string, configType string) error {
+func (tc *Tester) initServer(rawConfig string, configType string) error {
 
 	if testing.Short() {
-		t.SkipNow()
+		tc.t.SkipNow()
 		return nil
 	}
 
 	err := validateTestPrerequisites()
 	if err != nil {
-		t.Skipf("skipping tests as failed integration prerequisites. %s", err)
+		tc.t.Skipf("skipping tests as failed integration prerequisites. %s", err)
 		return nil
 	}
 
-	t.Cleanup(func() {
-		if t.Failed() {
+	tc.t.Cleanup(func() {
+		if tc.t.Failed() && tc.configLoaded {
+
 			res, err := http.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
 			if err != nil {
-				t.Log("unable to read the current config")
+				tc.t.Log("unable to read the current config")
+				return
 			}
 			defer res.Body.Close()
 			body, err := ioutil.ReadAll(res.Body)
 
 			var out bytes.Buffer
 			json.Indent(&out, body, "", "  ")
-			t.Logf("----------- failed with config -----------\n%s", out.String())
+			tc.t.Logf("----------- failed with config -----------\n%s", out.String())
 		}
 	})
 
 	rawConfig = prependCaddyFilePath(rawConfig)
 	client := &http.Client{
-		Timeout: time.Second * 2,
+		Timeout: Default.LoadRequestTimeout,
 	}
-
 	start := time.Now()
 	req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/load", Default.AdminPort), strings.NewReader(rawConfig))
 	if err != nil {
-		t.Errorf("failed to create request. %s", err)
+		tc.t.Errorf("failed to create request. %s", err)
 		return err
 	}
 
@@ -117,7 +151,7 @@ func initServer(t *testing.T, rawConfig string, configType string) error {
 
 	res, err := client.Do(req)
 	if err != nil {
-		t.Errorf("unable to contact caddy server. %s", err)
+		tc.t.Errorf("unable to contact caddy server. %s", err)
 		return err
 	}
 	timeElapsed(start, "caddytest: config load time")
@@ -125,7 +159,7 @@ func initServer(t *testing.T, rawConfig string, configType string) error {
 	defer res.Body.Close()
 	body, err := ioutil.ReadAll(res.Body)
 	if err != nil {
-		t.Errorf("unable to read response. %s", err)
+		tc.t.Errorf("unable to read response. %s", err)
 		return err
 	}
 
@@ -133,6 +167,7 @@ func initServer(t *testing.T, rawConfig string, configType string) error {
 		return configLoadError{Response: string(body)}
 	}
 
+	tc.configLoaded = true
 	return nil
 }
 
@@ -184,7 +219,7 @@ func validateTestPrerequisites() error {
 func isCaddyAdminRunning() error {
 	// assert that caddy is running
 	client := &http.Client{
-		Timeout: time.Second * 2,
+		Timeout: Default.LoadRequestTimeout,
 	}
 	_, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
 	if err != nil {
@@ -213,8 +248,8 @@ func prependCaddyFilePath(rawConfig string) string {
 	return r
 }
 
-// creates a testing transport that forces call dialing connections to happen locally
-func createTestingTransport() *http.Transport {
+// CreateTestingTransport creates a testing transport that forces call dialing connections to happen locally
+func CreateTestingTransport() *http.Transport {
 
 	dialer := net.Dialer{
 		Timeout:   5 * time.Second,
@@ -243,90 +278,56 @@ func createTestingTransport() *http.Transport {
 
 // AssertLoadError will load a config and expect an error
 func AssertLoadError(t *testing.T, rawConfig string, configType string, expectedError string) {
-	err := initServer(t, rawConfig, configType)
+
+	tc := NewTester(t)
+
+	err := tc.initServer(rawConfig, configType)
 	if !strings.Contains(err.Error(), expectedError) {
 		t.Errorf("expected error \"%s\" but got \"%s\"", expectedError, err.Error())
 	}
 }
 
-// AssertGetResponse request a URI and assert the status code and the body contains a string
-func AssertGetResponse(t *testing.T, requestURI string, statusCode int, expectedBody string) (*http.Response, string) {
-	resp, body := AssertGetResponseBody(t, requestURI, statusCode)
-	if !strings.Contains(body, expectedBody) {
-		t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", requestURI, expectedBody, body)
-	}
-	return resp, body
-}
-
-// AssertGetResponseBody request a URI and assert the status code matches
-func AssertGetResponseBody(t *testing.T, requestURI string, expectedStatusCode int) (*http.Response, string) {
-
-	client := &http.Client{
-		Transport: createTestingTransport(),
-	}
-
-	resp, err := client.Get(requestURI)
-	if err != nil {
-		t.Errorf("failed to call server %s", err)
-		return nil, ""
-	}
-
-	defer resp.Body.Close()
-
-	if expectedStatusCode != resp.StatusCode {
-		t.Errorf("requesting \"%s\" expected status code: %d but got %d", requestURI, expectedStatusCode, resp.StatusCode)
-	}
-
-	body, err := ioutil.ReadAll(resp.Body)
-	if err != nil {
-		t.Errorf("unable to read the response body %s", err)
-		return nil, ""
-	}
-
-	return resp, string(body)
-}
-
 // AssertRedirect makes a request and asserts the redirection happens
-func AssertRedirect(t *testing.T, requestURI string, expectedToLocation string, expectedStatusCode int) *http.Response {
+func (tc *Tester) AssertRedirect(requestURI string, expectedToLocation string, expectedStatusCode int) *http.Response {
 
 	redirectPolicyFunc := func(req *http.Request, via []*http.Request) error {
 		return http.ErrUseLastResponse
 	}
 
-	client := &http.Client{
-		CheckRedirect: redirectPolicyFunc,
-		Transport:     createTestingTransport(),
-	}
+	// using the existing client, we override the check redirect policy for this test
+	old := tc.Client.CheckRedirect
+	tc.Client.CheckRedirect = redirectPolicyFunc
+	defer func() { tc.Client.CheckRedirect = old }()
 
-	resp, err := client.Get(requestURI)
+	resp, err := tc.Client.Get(requestURI)
 	if err != nil {
-		t.Errorf("failed to call server %s", err)
+		tc.t.Errorf("failed to call server %s", err)
 		return nil
 	}
 
 	if expectedStatusCode != resp.StatusCode {
-		t.Errorf("requesting \"%s\" expected status code: %d but got %d", requestURI, expectedStatusCode, resp.StatusCode)
+		tc.t.Errorf("requesting \"%s\" expected status code: %d but got %d", requestURI, expectedStatusCode, resp.StatusCode)
 	}
 
 	loc, err := resp.Location()
 	if err != nil {
-		t.Errorf("requesting \"%s\" expected location: \"%s\" but got error: %s", requestURI, expectedToLocation, err)
+		tc.t.Errorf("requesting \"%s\" expected location: \"%s\" but got error: %s", requestURI, expectedToLocation, err)
 	}
 
 	if expectedToLocation != loc.String() {
-		t.Errorf("requesting \"%s\" expected location: \"%s\" but got \"%s\"", requestURI, expectedToLocation, loc.String())
+		tc.t.Errorf("requesting \"%s\" expected location: \"%s\" but got \"%s\"", requestURI, expectedToLocation, loc.String())
 	}
 
 	return resp
 }
 
-// AssertAdapt adapts a config and then tests it against an expected result
-func AssertAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) {
+// CompareAdapt adapts a config and then compares it against an expected result
+func CompareAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) bool {
 
 	cfgAdapter := caddyconfig.GetAdapter(adapterName)
 	if cfgAdapter == nil {
-		t.Errorf("unrecognized config adapter '%s'", adapterName)
-		return
+		t.Logf("unrecognized config adapter '%s'", adapterName)
+		return false
 	}
 
 	options := make(map[string]interface{})
@@ -334,8 +335,8 @@ func AssertAdapt(t *testing.T, rawConfig string, adapterName string, expectedRes
 
 	result, warnings, err := cfgAdapter.Adapt([]byte(rawConfig), options)
 	if err != nil {
-		t.Errorf("adapting config using %s adapter: %v", adapterName, err)
-		return
+		t.Logf("adapting config using %s adapter: %v", adapterName, err)
+		return false
 	}
 
 	if len(warnings) > 0 {
@@ -368,6 +369,136 @@ func AssertAdapt(t *testing.T, rawConfig string, adapterName string, expectedRes
 				fmt.Printf(" + %s\n", d.Payload)
 			}
 		}
+		return false
+	}
+	return true
+}
+
+// AssertAdapt adapts a config and then tests it against an expected result
+func AssertAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) {
+	ok := CompareAdapt(t, rawConfig, adapterName, expectedResponse)
+	if !ok {
 		t.Fail()
 	}
 }
+
+// Generic request functions
+
+func applyHeaders(t *testing.T, req *http.Request, requestHeaders []string) {
+	requestContentType := ""
+	for _, requestHeader := range requestHeaders {
+		arr := strings.SplitAfterN(requestHeader, ":", 2)
+		k := strings.TrimRight(arr[0], ":")
+		v := strings.TrimSpace(arr[1])
+		if k == "Content-Type" {
+			requestContentType = v
+		}
+		t.Logf("Request header: %s => %s", k, v)
+		req.Header.Set(k, v)
+	}
+
+	if requestContentType == "" {
+		t.Logf("Content-Type header not provided")
+	}
+}
+
+// AssertResponseCode will execute the request and verify the status code, returns a response for additional assertions
+func (tc *Tester) AssertResponseCode(req *http.Request, expectedStatusCode int) *http.Response {
+
+	resp, err := tc.Client.Do(req)
+	if err != nil {
+		tc.t.Fatalf("failed to call server %s", err)
+	}
+
+	if expectedStatusCode != resp.StatusCode {
+		tc.t.Errorf("requesting \"%s\" expected status code: %d but got %d", req.RequestURI, expectedStatusCode, resp.StatusCode)
+	}
+
+	return resp
+}
+
+// AssertResponse request a URI and assert the status code and the body contains a string
+func (tc *Tester) AssertResponse(req *http.Request, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	resp := tc.AssertResponseCode(req, expectedStatusCode)
+
+	defer resp.Body.Close()
+	bytes, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		tc.t.Fatalf("unable to read the response body %s", err)
+	}
+
+	body := string(bytes)
+
+	if !strings.Contains(body, expectedBody) {
+		tc.t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", req.RequestURI, expectedBody, body)
+	}
+
+	return resp, body
+}
+
+// Verb specific test functions
+
+// AssertGetResponse GET a URI and expect a statusCode and body text
+func (tc *Tester) AssertGetResponse(requestURI string, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("GET", requestURI, nil)
+	if err != nil {
+		tc.t.Fatalf("unable to create request %s", err)
+	}
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertDeleteResponse request a URI and expect a statusCode and body text
+func (tc *Tester) AssertDeleteResponse(requestURI string, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("DELETE", requestURI, nil)
+	if err != nil {
+		tc.t.Fatalf("unable to create request %s", err)
+	}
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPostResponseBody POST to a URI and assert the response code and body
+func (tc *Tester) AssertPostResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("POST", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPutResponseBody PUT to a URI and assert the response code and body
+func (tc *Tester) AssertPutResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("PUT", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPatchResponseBody PATCH to a URI and assert the response code and body
+func (tc *Tester) AssertPatchResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("PATCH", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}

caddytest/integration/autohttps_test.go

@@ -0,0 +1,22 @@
+package integration
+
+import (
+	"net/http"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestAutoHTTPtoHTTPSRedirects(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	localhost:9443
+	respond "Yahaha! You found me!"
+  `, "caddyfile")
+
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
+}

caddytest/integration/caddyfile_adapt/auto_https_disable_redirects.txt

@@ -0,0 +1,34 @@
+{
+	auto_https disable_redirects
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"disable_redirects": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/auto_https_off.txt

@@ -0,0 +1,37 @@
+{
+	auto_https off
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"tls_connection_policies": [
+						{}
+					],
+					"automatic_https": {
+						"disable": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options.txt

@@ -0,0 +1,73 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_ca_root /path/to/ca.crt
+
+	email test@example.com
+	admin off
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	local_certs
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"disabled": true
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuer": {
+							"module": "internal"
+						},
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_acme.txt

@@ -0,0 +1,85 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_eab {
+		key_id  4K2scIVbBpNd-78scadB2g
+		mac_key abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh
+	}
+	acme_ca_root /path/to/ca.crt
+	email test@example.com
+	admin off
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"disabled": true
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuer": {
+							"ca": "https://example.com",
+							"email": "test@example.com",
+							"external_account": {
+								"key_id": "4K2scIVbBpNd-78scadB2g",
+								"mac_key": "abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh"
+							},
+							"module": "acme",
+							"trusted_roots_pem_files": [
+								"/path/to/ca.crt"
+							]
+						},
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_admin.txt

@@ -0,0 +1,81 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_ca_root /path/to/ca.crt
+
+	email test@example.com
+	admin {
+		origins localhost:2019 [::1]:2019 127.0.0.1:2019 192.168.10.128
+	}
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	local_certs
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"listen": "localhost:2019",
+		"origins": [
+			"localhost:2019",
+			"[::1]:2019",
+			"127.0.0.1:2019",
+			"192.168.10.128"
+		]
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuer": {
+							"module": "internal"
+						},
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/handle_path.txt

@@ -0,0 +1,52 @@
+:80
+handle_path /api/v1/* {
+	respond "API v1"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/api/v1/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"strip_path_prefix": "/api/v1"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"body": "API v1",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/handle_path_sorting.txt

@@ -0,0 +1,105 @@
+:80 {
+	handle /api/* {
+		respond "api"
+	}
+
+	handle_path /static/* {
+		respond "static"
+	}
+
+	handle {
+		respond "handle"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"group": "group3",
+							"match": [
+								{
+									"path": [
+										"/static/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"strip_path_prefix": "/static"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"body": "static",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						},
+						{
+							"group": "group3",
+							"match": [
+								{
+									"path": [
+										"/api/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "api",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						},
+						{
+							"group": "group3",
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "handle",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_any_address.txt

@@ -0,0 +1,37 @@
+:80 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/version"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "hello from localhost",
+									"handler": "static_response",
+									"status_code": 200
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_domain.txt

@@ -0,0 +1,59 @@
+http://a.caddy.localhost {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"a.caddy.localhost"
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_localhost.txt

@@ -0,0 +1,54 @@
+localhost:80 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_non_standard_port.txt

@@ -0,0 +1,59 @@
+http://a.caddy.localhost:81 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":81"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"a.caddy.localhost"
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/https_on_domain.txt

@@ -0,0 +1,54 @@
+a.caddy.localhost {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_file.txt

@@ -0,0 +1,49 @@
+example.com
+
+import testdata/import_respond.txt Groot Rocket
+import testdata/import_respond.txt you "the confused man"
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "'I am Groot', hears Rocket",
+													"handler": "static_response"
+												},
+												{
+													"body": "'I am you', hears the confused man",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_snippet.txt

@@ -0,0 +1,83 @@
+(logging) {
+	log {
+		output file /var/log/caddy/{args.0}.access.log
+	}
+}
+
+a.example.com {
+	import logging a.example.com
+}
+
+b.example.com {
+	import logging b.example.com
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0",
+					"http.log.access.log1"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "/var/log/caddy/a.example.com.access.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			},
+			"log1": {
+				"writer": {
+					"filename": "/var/log/caddy/b.example.com.access.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log1"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"b.example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"logs": {
+						"logger_names": {
+							"a.example.com": "log0",
+							"b.example.com": "log1"
+						}
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_filters.txt

@@ -0,0 +1,69 @@
+:80
+
+log {
+	output stdout
+	format filter {
+		wrap console
+		fields {
+			request>headers>Authorization delete
+			request>headers>Server delete
+			request>remote_addr ip_mask {
+				ipv4 24
+				ipv6 32
+			}
+		}
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0"
+				]
+			},
+			"log0": {
+				"writer": {
+					"output": "stdout"
+				},
+				"encoder": {
+					"fields": {
+						"request\u003eheaders\u003eAuthorization": {
+							"filter": "delete"
+						},
+						"request\u003eheaders\u003eServer": {
+							"filter": "delete"
+						},
+						"request\u003eremote_addr": {
+							"filter": "ip_mask",
+							"ipv4_cidr": 24,
+							"ipv6_cidr": 32
+						}
+					},
+					"format": "filter",
+					"wrap": {
+						"format": "console"
+					}
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"logs": {
+						"default_logger_name": "log0"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_roll_days.txt

@@ -0,0 +1,47 @@
+:80
+
+log {
+	output file /var/log/access.log {
+		roll_size 1gb
+		roll_keep 5
+		roll_keep_for 90d
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "/var/log/access.log",
+					"output": "file",
+					"roll_keep": 5,
+					"roll_keep_days": 90,
+					"roll_size_mb": 954
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"logs": {
+						"default_logger_name": "log0"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/matcher_syntax.txt

@@ -0,0 +1,77 @@
+:80 {
+	@matcher {
+		method GET
+	}
+	respond @matcher "get"
+
+	@matcher2 method POST
+	respond @matcher2 "post"
+
+	@matcher3 not method PUT
+	respond @matcher3 "not put"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"method": [
+										"GET"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "get",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"method": [
+										"POST"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "post",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"not": [
+										{
+											"method": [
+												"PUT"
+											]
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "not put",
+									"handler": "static_response"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/matchers_in_route.txt

@@ -0,0 +1,31 @@
+:80 {
+    route {
+        # unused matchers should not panic
+        # see https://github.com/caddyserver/caddy/issues/3745
+        @matcher1 path /path1
+        @matcher2 path /path2
+    }
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "subroute"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/not_block_merging.txt

@@ -0,0 +1,49 @@
+:80
+
+@test {
+	not {
+		header Abc "123"
+		header Bcd "123"
+	}
+}
+respond @test 403
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"not": [
+										{
+											"header": {
+												"Abc": [
+													"123"
+												],
+												"Bcd": [
+													"123"
+												]
+											}
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "static_response",
+									"status_code": 403
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_expanded_form.txt

@@ -0,0 +1,132 @@
+:8886
+
+route {
+	# Add trailing slash for directory requests
+	@canonicalPath {
+		file {
+			try_files {path}/index.php
+		}
+		not path */
+	}
+	redir @canonicalPath {path}/ 308
+
+	# If the requested file does not exist, try index files
+	@indexFiles {
+		file {
+			try_files {path} {path}/index.php index.php
+			split_path .php
+		}
+	}
+	rewrite @indexFiles {http.matchers.file.relative}
+
+	# Proxy PHP files to the FastCGI responder
+	@phpFiles {
+		path *.php
+	}
+	reverse_proxy @phpFiles 127.0.0.1:9000 {
+		transport fastcgi {
+			split .php
+		}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8886"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"headers": {
+														"Location": [
+															"{http.request.uri.path}/"
+														]
+													},
+													"status_code": 308
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"try_files": [
+															"{http.request.uri.path}/index.php"
+														]
+													},
+													"not": [
+														{
+															"path": [
+																"*/"
+															]
+														}
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"uri": "{http.matchers.file.relative}"
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"split_path": [
+															".php"
+														],
+														"try_files": [
+															"{http.request.uri.path}",
+															"{http.request.uri.path}/index.php",
+															"index.php"
+														]
+													}
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"transport": {
+														"protocol": "fastcgi",
+														"split_path": [
+															".php"
+														]
+													},
+													"upstreams": [
+														{
+															"dial": "127.0.0.1:9000"
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"*.php"
+													]
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_index_off.txt

@@ -0,0 +1,66 @@
+:8884
+
+php_fastcgi localhost:9000 {
+    # some php_fastcgi-specific subdirectives
+    split .php .php5
+    env VAR1 value1
+    env VAR2 value2
+    root /var/www
+    index off
+
+    # passed through to reverse_proxy (directive order doesn't matter!)
+    lb_policy random
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"*.php",
+										"*.php5"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"load_balancing": {
+										"selection_policy": {
+											"policy": "random"
+										}
+									},
+									"transport": {
+										"env": {
+											"VAR1": "value1",
+											"VAR2": "value2"
+										},
+										"protocol": "fastcgi",
+										"root": "/var/www",
+										"split_path": [
+											".php",
+											".php5"
+										]
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_matcher.txt

@@ -0,0 +1,112 @@
+:8884
+
+@api host example.com
+php_fastcgi @api localhost:9000
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"headers": {
+														"Location": [
+															"{http.request.uri.path}/"
+														]
+													},
+													"status_code": 308
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"try_files": [
+															"{http.request.uri.path}/index.php"
+														]
+													},
+													"not": [
+														{
+															"path": [
+																"*/"
+															]
+														}
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"uri": "{http.matchers.file.relative}"
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"split_path": [
+															".php"
+														],
+														"try_files": [
+															"{http.request.uri.path}",
+															"{http.request.uri.path}/index.php",
+															"index.php"
+														]
+													}
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"transport": {
+														"protocol": "fastcgi",
+														"split_path": [
+															".php"
+														]
+													},
+													"upstreams": [
+														{
+															"dial": "localhost:9000"
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"*.php"
+													]
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_subdirectives.txt

@@ -0,0 +1,118 @@
+:8884
+
+php_fastcgi localhost:9000 {
+    # some php_fastcgi-specific subdirectives
+    split .php .php5
+    env VAR1 value1
+    env VAR2 value2
+    root /var/www
+    index index.php5
+
+    # passed through to reverse_proxy (directive order doesn't matter!)
+    lb_policy random
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}/index.php5"
+										]
+									},
+									"not": [
+										{
+											"path": [
+												"*/"
+											]
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "static_response",
+									"headers": {
+										"Location": [
+											"{http.request.uri.path}/"
+										]
+									},
+									"status_code": 308
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}",
+											"{http.request.uri.path}/index.php5",
+											"index.php5"
+										],
+										"split_path": [
+											".php",
+											".php5"
+										]
+									}
+								}
+							],
+							"handle": [
+								{
+									"handler": "rewrite",
+									"uri": "{http.matchers.file.relative}"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"path": [
+										"*.php",
+										"*.php5"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"load_balancing": {
+										"selection_policy": {
+											"policy": "random"
+										}
+									},
+									"transport": {
+										"env": {
+											"VAR1": "value1",
+											"VAR2": "value2"
+										},
+										"protocol": "fastcgi",
+										"root": "/var/www",
+										"split_path": [
+											".php",
+											".php5"
+										]
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_empty_non_http_transport.txt

@@ -0,0 +1,36 @@
+:8884
+
+reverse_proxy 127.0.0.1:65535 {
+    transport fastcgi
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"transport": {
+										"protocol": "fastcgi"
+									},
+									"upstreams": [
+										{
+											"dial": "127.0.0.1:65535"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_h2c.txt

@@ -0,0 +1,98 @@
+
+https://example.com {
+	reverse_proxy /path http://localhost:54321 {
+		header_up Host {host}
+		header_up X-Real-IP {remote}
+		header_up X-Forwarded-For {remote}
+		header_up X-Forwarded-Port {server_port}
+		header_up X-Forwarded-Proto "http"
+		transport http {
+			versions h2c 2
+			compression off
+		}
+		buffer_requests
+	}
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"buffer_requests": true,
+													"handler": "reverse_proxy",
+													"headers": {
+														"request": {
+															"set": {
+																"Host": [
+																	"{http.request.host}"
+																],
+																"X-Forwarded-For": [
+																	"{http.request.remote}"
+																],
+																"X-Forwarded-Port": [
+																	"{server_port}"
+																],
+																"X-Forwarded-Proto": [
+																	"http"
+																],
+																"X-Real-Ip": [
+																	"{http.request.remote}"
+																]
+															}
+														}
+													},
+													"transport": {
+														"compression": false,
+														"protocol": "http",
+														"versions": [
+															"h2c",
+															"2"
+														]
+													},
+													"upstreams": [
+														{
+															"dial": "localhost:54321"
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/path"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/shorthand_parameterized_placeholders.txt

@@ -0,0 +1,43 @@
+localhost:80
+respond * "{header.content-type} {labels.0} {query.p} {path.0} {re.name.0}"
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "{http.request.header.content-type} {http.request.host.labels.0} {http.request.uri.query.p} {http.request.uri.path.0} {http.regexp.name.0}",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/sort_directives_with_any_matcher_first.txt

@@ -0,0 +1,55 @@
+:80
+
+file_server
+
+@untrusted not remote_ip 10.1.1.0/24
+file_server @untrusted
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"not": [
+										{
+											"remote_ip": {
+												"ranges": [
+													"10.1.1.0/24"
+												]
+											}
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "file_server",
+									"hide": [
+										"Caddyfile"
+									]
+								}
+							]
+						},
+						{
+							"handle": [
+								{
+									"handler": "file_server",
+									"hide": [
+										"Caddyfile"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_client_auth_cert_file.txt

@@ -0,0 +1,66 @@
+localhost
+
+respond "hello from localhost"
+tls {
+	client_auth {
+		mode request 
+		trusted_ca_cert_file ../caddy.ca.cer
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"tls_connection_policies": [
+						{
+							"match": {
+								"sni": [
+									"localhost"
+								]
+							},
+							"client_authentication": {
+								"trusted_ca_certs": [
+									"MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkwODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl03WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45twOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNxtdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTUApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAdBgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5uNY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfKD66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEOfG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnkoNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdleIh6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ=="
+								],
+								"mode": "request"
+							}
+						},
+						{}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_client_auth_inline_cert.txt

@@ -0,0 +1,66 @@
+localhost
+
+respond "hello from localhost"
+tls {
+	client_auth {
+		mode request 
+		trusted_ca_cert MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkwODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl03WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45twOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNxtdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTUApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAdBgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5uNY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfKD66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEOfG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnkoNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdleIh6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ== 
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"tls_connection_policies": [
+						{
+							"match": {
+								"sni": [
+									"localhost"
+								]
+							},
+							"client_authentication": {
+								"trusted_ca_certs": [
+									"MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkwODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl03WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45twOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNxtdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTUApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAdBgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5uNY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfKD66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEOfG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnkoNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdleIh6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ=="
+								],
+								"mode": "request"
+							}
+						},
+						{}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt_test.go

@@ -1,285 +1,47 @@
 package integration
 
 import (
+	"io/ioutil"
+	"regexp"
+	"strings"
 	"testing"
 
 	"github.com/caddyserver/caddy/v2/caddytest"
 )
 
-func TestHttpOnlyOnLocalhost(t *testing.T) {
-	caddytest.AssertAdapt(t, ` 
-	localhost:80 {
-		respond /version 200 {
-			body "hello from localhost"
-		}
+func TestCaddyfileAdaptToJSON(t *testing.T) {
+	// load the list of test files from the dir
+	files, err := ioutil.ReadDir("./caddyfile_adapt")
+	if err != nil {
+		t.Errorf("failed to read caddyfile_adapt dir: %s", err)
 	}
-  `, "caddyfile", `{
-	"apps": {
-		"http": {
-			"servers": {
-				"srv0": {
-					"listen": [
-						":80"
-					],
-					"routes": [
-						{
-							"match": [
-								{
-									"host": [
-										"localhost"
-									]
-								}
-							],
-							"handle": [
-								{
-									"handler": "subroute",
-									"routes": [
-										{
-											"handle": [
-												{
-													"body": "hello from localhost",
-													"handler": "static_response",
-													"status_code": 200
-												}
-											],
-											"match": [
-												{
-													"path": [
-														"/version"
-													]
-												}
-											]
-										}
-									]
-								}
-							],
-							"terminal": true
-						}
-					]
-				}
-			}
-		}
-	}
-}`)
-}
 
-func TestHttpOnlyOnAnyAddress(t *testing.T) {
-	caddytest.AssertAdapt(t, ` 
-	:80 {
-		respond /version 200 {
-			body "hello from localhost"
-		}
-	}
-  `, "caddyfile", `{
-	"apps": {
-		"http": {
-			"servers": {
-				"srv0": {
-					"listen": [
-						":80"
-					],
-					"routes": [
-						{
-							"match": [
-								{
-									"path": [
-										"/version"
-									]
-								}
-							],
-							"handle": [
-								{
-									"body": "hello from localhost",
-									"handler": "static_response",
-									"status_code": 200
-								}
-							]
-						}
-					]
-				}
-			}
-		}
-	}
-}`)
-}
+	// prep a regexp to fix strings on windows
+	winNewlines := regexp.MustCompile(`\r?\n`)
 
-func TestHttpsOnDomain(t *testing.T) {
-	caddytest.AssertAdapt(t, ` 
-	a.caddy.localhost {
-		respond /version 200 {
-			body "hello from localhost"
+	for _, f := range files {
+		if f.IsDir() {
+			continue
 		}
-	}
-  `, "caddyfile", `{
-	"apps": {
-		"http": {
-			"servers": {
-				"srv0": {
-					"listen": [
-						":443"
-					],
-					"routes": [
-						{
-							"match": [
-								{
-									"host": [
-										"a.caddy.localhost"
-									]
-								}
-							],
-							"handle": [
-								{
-									"handler": "subroute",
-									"routes": [
-										{
-											"handle": [
-												{
-													"body": "hello from localhost",
-													"handler": "static_response",
-													"status_code": 200
-												}
-											],
-											"match": [
-												{
-													"path": [
-														"/version"
-													]
-												}
-											]
-										}
-									]
-								}
-							],
-							"terminal": true
-						}
-					]
-				}
-			}
-		}
-	}
-}`)
-}
 
-func TestHttpOnlyOnDomain(t *testing.T) {
-	caddytest.AssertAdapt(t, ` 
-	http://a.caddy.localhost {
-		respond /version 200 {
-			body "hello from localhost"
+		// read the test file
+		filename := f.Name()
+		data, err := ioutil.ReadFile("./caddyfile_adapt/" + filename)
+		if err != nil {
+			t.Errorf("failed to read %s dir: %s", filename, err)
 		}
-	}
-  `, "caddyfile", `{
-	"apps": {
-		"http": {
-			"servers": {
-				"srv0": {
-					"listen": [
-						":80"
-					],
-					"routes": [
-						{
-							"match": [
-								{
-									"host": [
-										"a.caddy.localhost"
-									]
-								}
-							],
-							"handle": [
-								{
-									"handler": "subroute",
-									"routes": [
-										{
-											"handle": [
-												{
-													"body": "hello from localhost",
-													"handler": "static_response",
-													"status_code": 200
-												}
-											],
-											"match": [
-												{
-													"path": [
-														"/version"
-													]
-												}
-											]
-										}
-									]
-								}
-							],
-							"terminal": true
-						}
-					],
-					"automatic_https": {
-						"skip": [
-							"a.caddy.localhost"
-						]
-					}
-				}
-			}
-		}
-	}
-}`)
-}
 
-func TestHttpOnlyOnNonStandardPort(t *testing.T) {
-	caddytest.AssertAdapt(t, ` 
-	http://a.caddy.localhost:81 {
-		respond /version 200 {
-			body "hello from localhost"
+		// split the Caddyfile (first) and JSON (second) parts
+		parts := strings.Split(string(data), "----------")
+		caddyfile, json := strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1])
+
+		// replace windows newlines in the json with unix newlines
+		json = winNewlines.ReplaceAllString(json, "\n")
+
+		// run the test
+		ok := caddytest.CompareAdapt(t, caddyfile, "caddyfile", json)
+		if !ok {
+			t.Errorf("failed to adapt %s", filename)
 		}
 	}
-  `, "caddyfile", `{
-	"apps": {
-		"http": {
-			"servers": {
-				"srv0": {
-					"listen": [
-						":81"
-					],
-					"routes": [
-						{
-							"match": [
-								{
-									"host": [
-										"a.caddy.localhost"
-									]
-								}
-							],
-							"handle": [
-								{
-									"handler": "subroute",
-									"routes": [
-										{
-											"handle": [
-												{
-													"body": "hello from localhost",
-													"handler": "static_response",
-													"status_code": 200
-												}
-											],
-											"match": [
-												{
-													"path": [
-														"/version"
-													]
-												}
-											]
-										}
-									]
-								}
-							],
-							"terminal": true
-						}
-					],
-					"automatic_https": {
-						"skip": [
-							"a.caddy.localhost"
-						]
-					}
-				}
-			}
-		}
-	}
-}`)
 }

caddytest/integration/caddyfile_test.go

@@ -1,6 +1,8 @@
 package integration
 
 import (
+	"net/http"
+	"net/url"
 	"testing"
 
 	"github.com/caddyserver/caddy/v2/caddytest"
@@ -9,7 +11,8 @@ import (
 func TestRespond(t *testing.T) {
 
 	// arrange
-	caddytest.InitServer(t, ` 
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
   {
     http_port     9080
     https_port    9443
@@ -23,13 +26,14 @@ func TestRespond(t *testing.T) {
   `, "caddyfile")
 
 	// act and assert
-	caddytest.AssertGetResponse(t, "http://localhost:9080/version", 200, "hello from localhost")
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost")
 }
 
 func TestRedirect(t *testing.T) {
 
 	// arrange
-	caddytest.InitServer(t, `
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
   {
     http_port     9080
     https_port    9443
@@ -46,10 +50,10 @@ func TestRedirect(t *testing.T) {
   `, "caddyfile")
 
 	// act and assert
-	caddytest.AssertRedirect(t, "http://localhost:9080/", "http://localhost:9080/hello", 301)
+	tester.AssertRedirect("http://localhost:9080/", "http://localhost:9080/hello", 301)
 
 	// follow redirect
-	caddytest.AssertGetResponse(t, "http://localhost:9080/", 200, "hello from localhost")
+	tester.AssertGetResponse("http://localhost:9080/", 200, "hello from localhost")
 }
 
 func TestDuplicateHosts(t *testing.T) {
@@ -66,3 +70,34 @@ func TestDuplicateHosts(t *testing.T) {
 		"caddyfile",
 		"duplicate site address not allowed")
 }
+
+func TestReadCookie(t *testing.T) {
+
+	localhost, _ := url.Parse("http://localhost")
+	cookie := http.Cookie{
+		Name:  "clientname",
+		Value: "caddytest",
+	}
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.Client.Jar.SetCookies(localhost, []*http.Cookie{&cookie})
+	tester.InitServer(` 
+  {
+    http_port     9080
+    https_port    9443
+  }
+  
+  localhost:9080 {
+    templates {
+      root testdata
+    }
+    file_server {
+      root testdata
+    }
+  }
+  `, "caddyfile")
+
+	// act and assert
+	tester.AssertGetResponse("http://localhost:9080/cookie.html", 200, "<h2>Cookie.ClientName caddytest</h2>")
+}

caddytest/integration/handler_test.go

@@ -0,0 +1,28 @@
+package integration
+
+import (
+	"net/http"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestBrowse(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	http://localhost:9080 {
+		file_server browse
+	}
+  `, "caddyfile")
+
+	req, err := http.NewRequest(http.MethodGet, "http://localhost:9080/", nil)
+	if err != nil {
+		t.Fail()
+		return
+	}
+	tester.AssertResponseCode(req, 200)
+}

caddytest/integration/map_test.go

@@ -0,0 +1,143 @@
+package integration
+
+import (
+	"bytes"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestMap(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`{
+		http_port     9080
+		https_port    9443
+	}
+
+	localhost:9080 {
+
+		map http.request.method dest-name {
+			default unknown
+			G.T get-called
+			POST post-called
+		}
+
+		respond /version 200 {
+			body "hello from localhost {dest-name}"
+		}	
+	}
+	`, "caddyfile")
+
+	// act and assert
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost get-called")
+	tester.AssertPostResponseBody("http://localhost:9080/version", []string{}, bytes.NewBuffer([]byte{}), 200, "hello from localhost post-called")
+}
+
+func TestMapRespondWithDefault(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`{
+		http_port     9080
+		https_port    9443
+		}
+		
+		localhost:9080 {
+	
+			map http.request.method dest-name {
+				default unknown
+				GET get-called
+			}
+		
+			respond /version 200 {
+				body "hello from localhost {dest-name}"
+			}	
+		}
+	`, "caddyfile")
+
+	// act and assert
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost get-called")
+	tester.AssertPostResponseBody("http://localhost:9080/version", []string{}, bytes.NewBuffer([]byte{}), 200, "hello from localhost unknown")
+}
+
+func TestMapAsJson(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`{
+		"apps": {
+			"http": {
+			"http_port": 9080,
+			"https_port": 9443,
+			"servers": {
+				"srv0": {
+				"listen": [
+					":9080"
+				],
+				"routes": [
+					{
+					"handle": [
+						{
+						"handler": "subroute",
+						"routes": [
+							{
+							"handle": [
+								{
+								"handler": "map",
+								"source": "http.request.method",
+								"destination": "dest-name",
+								"default": "unknown",
+								"items": [
+									{
+									"expression": "GET",
+									"value": "get-called"
+									},
+									{
+									"expression": "POST",
+									"value": "post-called"
+									}
+								]
+								}
+							]
+							},
+							{
+							"handle": [
+								{
+								"body": "hello from localhost {dest-name}",
+								"handler": "static_response",
+								"status_code": 200
+								}
+							],
+							"match": [
+								{
+								"path": [
+									"/version"
+								]
+								}
+							]
+							}
+						]
+						}
+					],
+					"match": [
+						{
+						"host": [
+							"localhost"
+						]
+						}
+					],
+					"terminal": true
+					}
+				]
+				}
+			}
+			}
+		}
+		}
+		`, "json")
+
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost get-called")
+	tester.AssertPostResponseBody("http://localhost:9080/version", []string{}, bytes.NewBuffer([]byte{}), 200, "hello from localhost post-called")
+}

caddytest/integration/reverseproxy_test.go

@@ -0,0 +1,97 @@
+package integration
+
+import (
+	"fmt"
+	"io/ioutil"
+	"net"
+	"net/http"
+	"os"
+	"runtime"
+	"strings"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestReverseProxyHealthCheck(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	http://localhost:2020 {
+		respond "Hello, World!"
+	}
+	http://localhost:2021 {
+		respond "ok"
+	}
+	http://localhost:9080 {
+		reverse_proxy {
+			to localhost:2020
+	
+			health_path /health
+			health_port 2021
+			health_interval 2s
+			health_timeout 5s
+		}
+	}
+  `, "caddyfile")
+
+	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")
+}
+
+func TestReverseProxyHealthCheckUnixSocket(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.SkipNow()
+	}
+	tester := caddytest.NewTester(t)
+	f, err := ioutil.TempFile("", "*.sock")
+	if err != nil {
+		t.Errorf("failed to create TempFile: %s", err)
+		return
+	}
+	// a hack to get a file name within a valid path to use as socket
+	socketName := f.Name()
+	os.Remove(f.Name())
+
+	server := http.Server{
+		Handler: http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+			if strings.HasPrefix(req.URL.Path, "/health") {
+				w.Write([]byte("ok"))
+				return
+			}
+			w.Write([]byte("Hello, World!"))
+		}),
+	}
+
+	unixListener, err := net.Listen("unix", socketName)
+	if err != nil {
+		t.Errorf("failed to listen on the socket: %s", err)
+		return
+	}
+	go server.Serve(unixListener)
+	t.Cleanup(func() {
+		server.Close()
+	})
+	runtime.Gosched() // Allow other goroutines to run
+
+	tester.InitServer(fmt.Sprintf(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	http://localhost:9080 {
+		reverse_proxy {
+			to unix/%s
+	
+			health_path /health
+			health_port 2021
+			health_interval 2s
+			health_timeout 5s
+		}
+	}
+	`, socketName), "caddyfile")
+
+	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")
+}

caddytest/integration/sni_test.go

@@ -9,7 +9,8 @@ import (
 func TestDefaultSNI(t *testing.T) {
 
 	// arrange
-	caddytest.InitServer(t, `{
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`{
     "apps": {
       "http": {
         "http_port": 9080,
@@ -98,13 +99,14 @@ func TestDefaultSNI(t *testing.T) {
 
 	// act and assert
 	// makes a request with no sni
-	caddytest.AssertGetResponse(t, "https://127.0.0.1:9443/version", 200, "hello from a")
+	tester.AssertGetResponse("https://127.0.0.1:9443/version", 200, "hello from a")
 }
 
 func TestDefaultSNIWithNamedHostAndExplicitIP(t *testing.T) {
 
 	// arrange
-	caddytest.InitServer(t, ` 
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
   {
     "apps": {
       "http": {
@@ -198,13 +200,14 @@ func TestDefaultSNIWithNamedHostAndExplicitIP(t *testing.T) {
 
 	// act and assert
 	// makes a request with no sni
-	caddytest.AssertGetResponse(t, "https://127.0.0.1:9443/version", 200, "hello from a")
+	tester.AssertGetResponse("https://127.0.0.1:9443/version", 200, "hello from a")
 }
 
 func TestDefaultSNIWithPortMappingOnly(t *testing.T) {
 
 	// arrange
-	caddytest.InitServer(t, ` 
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
   {
     "apps": {
       "http": {
@@ -270,7 +273,7 @@ func TestDefaultSNIWithPortMappingOnly(t *testing.T) {
 
 	// act and assert
 	// makes a request with no sni
-	caddytest.AssertGetResponse(t, "https://127.0.0.1:9443/version", 200, "hello from a")
+	tester.AssertGetResponse("https://127.0.0.1:9443/version", 200, "hello from a")
 }
 
 func TestHttpOnlyOnDomainWithSNI(t *testing.T) {

caddytest/integration/stream_test.go

@@ -0,0 +1,437 @@
+package integration
+
+import (
+	"compress/gzip"
+	"context"
+	"crypto/rand"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+	"net/http/httputil"
+	"net/url"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+	"golang.org/x/net/http2"
+	"golang.org/x/net/http2/h2c"
+)
+
+// (see https://github.com/caddyserver/caddy/issues/3556 for use case)
+func TestH2ToH2CStream(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
+  {
+    "apps": {
+      "http": {
+        "http_port": 9080,
+        "https_port": 9443,
+        "servers": {
+          "srv0": {
+            "listen": [
+              ":9443"
+            ],
+            "routes": [
+              {
+                "handle": [
+                  {
+                    "handler": "reverse_proxy",
+                    "transport": {
+                      "protocol": "http",
+                      "compression": false,
+                      "versions": [
+                        "h2c",
+                        "2"
+                      ]
+                    },
+                    "upstreams": [
+                      {
+                        "dial": "localhost:54321"
+                      }
+                    ]
+                  }
+                ],
+                "match": [
+                  {
+                    "path": [
+                      "/tov2ray"
+                    ]
+                  }
+                ]
+              }
+            ],
+            "tls_connection_policies": [
+              {
+                "certificate_selection": {
+                  "any_tag": ["cert0"]
+                },
+                "default_sni": "a.caddy.localhost"
+              }
+            ]
+          }
+        }
+      },
+      "tls": {
+        "certificates": {
+          "load_files": [
+            {
+              "certificate": "/a.caddy.localhost.crt",
+              "key": "/a.caddy.localhost.key",
+              "tags": [
+                "cert0"
+              ]
+            }
+          ]
+        }
+      },
+      "pki": {
+        "certificate_authorities" : {
+          "local" : {
+            "install_trust": false
+          }
+        }
+      }
+    }
+  }
+  `, "json")
+
+	expectedBody := "some data to be echoed"
+	// start the server
+	server := testH2ToH2CStreamServeH2C(t)
+	go server.ListenAndServe()
+	defer func() {
+		ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
+		defer cancel()
+		server.Shutdown(ctx)
+	}()
+
+	r, w := io.Pipe()
+	req := &http.Request{
+		Method: "PUT",
+		Body:   ioutil.NopCloser(r),
+		URL: &url.URL{
+			Scheme: "https",
+			Host:   "127.0.0.1:9443",
+			Path:   "/tov2ray",
+		},
+		Proto:      "HTTP/2",
+		ProtoMajor: 2,
+		ProtoMinor: 0,
+		Header:     make(http.Header),
+	}
+	// Disable any compression method from server.
+	req.Header.Set("Accept-Encoding", "identity")
+
+	resp := tester.AssertResponseCode(req, 200)
+	if 200 != resp.StatusCode {
+		return
+	}
+	go func() {
+		fmt.Fprint(w, expectedBody)
+		w.Close()
+	}()
+
+	defer resp.Body.Close()
+	bytes, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		t.Fatalf("unable to read the response body %s", err)
+	}
+
+	body := string(bytes)
+
+	if !strings.Contains(body, expectedBody) {
+		t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", req.RequestURI, expectedBody, body)
+	}
+	return
+}
+
+func testH2ToH2CStreamServeH2C(t *testing.T) *http.Server {
+	h2s := &http2.Server{}
+	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		rstring, err := httputil.DumpRequest(r, false)
+		if err == nil {
+			t.Logf("h2c server received req: %s", rstring)
+		}
+		// We only accept HTTP/2!
+		if r.ProtoMajor != 2 {
+			t.Error("Not a HTTP/2 request, rejected!")
+			w.WriteHeader(http.StatusInternalServerError)
+			return
+		}
+
+		if r.Host != "127.0.0.1:9443" {
+			t.Errorf("r.Host doesn't match, %v!", r.Host)
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+
+		if !strings.HasPrefix(r.URL.Path, "/tov2ray") {
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+
+		w.Header().Set("Cache-Control", "no-store")
+		w.WriteHeader(200)
+		if f, ok := w.(http.Flusher); ok {
+			f.Flush()
+		}
+
+		buf := make([]byte, 4*1024)
+
+		for {
+			n, err := r.Body.Read(buf)
+			if n > 0 {
+				w.Write(buf[:n])
+			}
+
+			if err != nil {
+				if err == io.EOF {
+					r.Body.Close()
+				}
+				break
+			}
+		}
+	})
+
+	server := &http.Server{
+		Addr:    "127.0.0.1:54321",
+		Handler: h2c.NewHandler(handler, h2s),
+	}
+	return server
+}
+
+// (see https://github.com/caddyserver/caddy/issues/3606 for use case)
+func TestH2ToH1ChunkedResponse(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
+{
+  "logging": {
+    "logs": {
+      "default": {
+        "level": "DEBUG"
+      }
+    }
+  },
+  "apps": {
+    "http": {
+      "http_port": 9080,
+      "https_port": 9443,
+      "servers": {
+        "srv0": {
+          "listen": [
+            ":9443"
+          ],
+          "routes": [
+            {
+              "handle": [
+                {
+                  "handler": "subroute",
+                  "routes": [
+                    {
+                      "handle": [
+                        {
+                          "encodings": {
+                            "gzip": {}
+                          },
+                          "handler": "encode"
+                        }
+                      ]
+                    },
+                    {
+                      "handle": [
+                        {
+                          "handler": "reverse_proxy",
+                          "upstreams": [
+                            {
+                              "dial": "localhost:54321"
+                            }
+                          ]
+                        }
+                      ],
+                      "match": [
+                        {
+                          "path": [
+                            "/tov2ray"
+                          ]
+                        }
+                      ]
+                    }
+                  ]
+                }
+              ],
+              "terminal": true
+            }
+          ],
+          "tls_connection_policies": [
+            {
+              "certificate_selection": {
+                "any_tag": [
+                  "cert0"
+                ]
+              },
+              "default_sni": "a.caddy.localhost"
+            }
+          ]
+        }
+      }
+    },
+    "tls": {
+      "certificates": {
+        "load_files": [
+          {
+            "certificate": "/a.caddy.localhost.crt",
+            "key": "/a.caddy.localhost.key",
+            "tags": [
+              "cert0"
+            ]
+          }
+        ]
+      }
+    },
+    "pki": {
+      "certificate_authorities": {
+        "local": {
+          "install_trust": false
+        }
+      }
+    }
+  }
+}
+  `, "json")
+
+	// need a large body here to trigger caddy's compression, larger than gzip.miniLength
+	expectedBody, err := GenerateRandomString(1024)
+	if err != nil {
+		t.Fatalf("generate expected body failed, err: %s", err)
+	}
+
+	// start the server
+	server := testH2ToH1ChunkedResponseServeH1(t)
+	go server.ListenAndServe()
+	defer func() {
+		ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
+		defer cancel()
+		server.Shutdown(ctx)
+	}()
+
+	r, w := io.Pipe()
+	req := &http.Request{
+		Method: "PUT",
+		Body:   ioutil.NopCloser(r),
+		URL: &url.URL{
+			Scheme: "https",
+			Host:   "127.0.0.1:9443",
+			Path:   "/tov2ray",
+		},
+		Proto:      "HTTP/2",
+		ProtoMajor: 2,
+		ProtoMinor: 0,
+		Header:     make(http.Header),
+	}
+	// underlying transport will automaticlly add gzip
+	// req.Header.Set("Accept-Encoding", "gzip")
+	go func() {
+		fmt.Fprint(w, expectedBody)
+		w.Close()
+	}()
+	resp := tester.AssertResponseCode(req, 200)
+	if 200 != resp.StatusCode {
+		return
+	}
+
+	defer resp.Body.Close()
+	bytes, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		t.Fatalf("unable to read the response body %s", err)
+	}
+
+	body := string(bytes)
+
+	if body != expectedBody {
+		t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", req.RequestURI, expectedBody, body)
+	}
+	return
+}
+
+func testH2ToH1ChunkedResponseServeH1(t *testing.T) *http.Server {
+	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+
+		if r.Host != "127.0.0.1:9443" {
+			t.Errorf("r.Host doesn't match, %v!", r.Host)
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+
+		if !strings.HasPrefix(r.URL.Path, "/tov2ray") {
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+
+		defer r.Body.Close()
+		bytes, err := ioutil.ReadAll(r.Body)
+		if err != nil {
+			t.Fatalf("unable to read the response body %s", err)
+		}
+
+		n := len(bytes)
+
+		var writer io.Writer
+		if strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
+			gw, err := gzip.NewWriterLevel(w, 5)
+			if err != nil {
+				t.Error("can't return gzip data")
+				w.WriteHeader(http.StatusInternalServerError)
+				return
+			}
+			defer gw.Close()
+			writer = gw
+			w.Header().Set("Content-Encoding", "gzip")
+			w.Header().Del("Content-Length")
+			w.WriteHeader(200)
+		} else {
+			writer = w
+		}
+		if n > 0 {
+			writer.Write(bytes[:])
+		}
+	})
+
+	server := &http.Server{
+		Addr:    "127.0.0.1:54321",
+		Handler: handler,
+	}
+	return server
+}
+
+// GenerateRandomBytes returns securely generated random bytes.
+// It will return an error if the system's secure random
+// number generator fails to function correctly, in which
+// case the caller should not continue.
+func GenerateRandomBytes(n int) ([]byte, error) {
+	b := make([]byte, n)
+	_, err := rand.Read(b)
+	// Note that err == nil only if we read len(b) bytes.
+	if err != nil {
+		return nil, err
+	}
+
+	return b, nil
+}
+
+// GenerateRandomString returns a securely generated random string.
+// It will return an error if the system's secure random
+// number generator fails to function correctly, in which
+// case the caller should not continue.
+func GenerateRandomString(n int) (string, error) {
+	const letters = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-"
+	bytes, err := GenerateRandomBytes(n)
+	if err != nil {
+		return "", err
+	}
+	for i, b := range bytes {
+		bytes[i] = letters[b%byte(len(letters))]
+	}
+	return string(bytes), nil
+}

caddytest/integration/testdata/cookie.html

@@ -0,0 +1 @@
+<h2>Cookie.ClientName {{.Cookie "clientname"}}</h2>

caddytest/integration/testdata/import_respond.txt

@@ -0,0 +1 @@
+respond "'I am {args.0}', hears {args.1}"

cmd/commandfuncs.go

@@ -16,6 +16,7 @@ package caddycmd
 
 import (
 	"bytes"
+	"context"
 	"crypto/rand"
 	"encoding/json"
 	"fmt"
@@ -41,6 +42,7 @@ import (
 func cmdStart(fl Flags) (int, error) {
 	startCmdConfigFlag := fl.String("config")
 	startCmdConfigAdapterFlag := fl.String("adapter")
+	startCmdPidfileFlag := fl.String("pidfile")
 	startCmdWatchFlag := fl.Bool("watch")
 
 	// open a listener to which the child process will connect when
@@ -71,6 +73,9 @@ func cmdStart(fl Flags) (int, error) {
 	if startCmdWatchFlag {
 		cmd.Args = append(cmd.Args, "--watch")
 	}
+	if startCmdPidfileFlag != "" {
+		cmd.Args = append(cmd.Args, "--pidfile", startCmdPidfileFlag)
+	}
 	stdinpipe, err := cmd.StdinPipe()
 	if err != nil {
 		return caddy.ExitCodeFailedStartup,
@@ -144,13 +149,25 @@ func cmdStart(fl Flags) (int, error) {
 }
 
 func cmdRun(fl Flags) (int, error) {
+	caddy.TrapSignals()
+
 	runCmdConfigFlag := fl.String("config")
 	runCmdConfigAdapterFlag := fl.String("adapter")
 	runCmdResumeFlag := fl.Bool("resume")
+	runCmdLoadEnvfileFlag := fl.String("envfile")
 	runCmdPrintEnvFlag := fl.Bool("environ")
 	runCmdWatchFlag := fl.Bool("watch")
+	runCmdPidfileFlag := fl.String("pidfile")
 	runCmdPingbackFlag := fl.String("pingback")
 
+	// load all additional envs as soon as possible
+	if runCmdLoadEnvfileFlag != "" {
+		if err := loadEnvFromFile(runCmdLoadEnvfileFlag); err != nil {
+			return caddy.ExitCodeFailedStartup,
+				fmt.Errorf("loading additional environment variables: %v", err)
+		}
+	}
+
 	// if we are supposed to print the environment, do that first
 	if runCmdPrintEnvFlag {
 		printEnvironment()
@@ -225,6 +242,16 @@ func cmdRun(fl Flags) (int, error) {
 		go watchConfigFile(configFile, runCmdConfigAdapterFlag)
 	}
 
+	// create pidfile
+	if runCmdPidfileFlag != "" {
+		err := caddy.PIDFile(runCmdPidfileFlag)
+		if err != nil {
+			caddy.Log().Error("unable to write PID file",
+				zap.String("pidfile", runCmdPidfileFlag),
+				zap.Error(err))
+		}
+	}
+
 	// warn if the environment does not provide enough information about the disk
 	hasXDG := os.Getenv("XDG_DATA_HOME") != "" &&
 		os.Getenv("XDG_CONFIG_HOME") != "" &&
@@ -250,24 +277,9 @@ func cmdRun(fl Flags) (int, error) {
 func cmdStop(fl Flags) (int, error) {
 	stopCmdAddrFlag := fl.String("address")
 
-	adminAddr := caddy.DefaultAdminListen
-	if stopCmdAddrFlag != "" {
-		adminAddr = stopCmdAddrFlag
-	}
-	stopEndpoint := fmt.Sprintf("http://%s/stop", adminAddr)
-
-	req, err := http.NewRequest(http.MethodPost, stopEndpoint, nil)
+	err := apiRequest(stopCmdAddrFlag, http.MethodPost, "/stop", nil)
 	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("making request: %v", err)
-	}
-	req.Header.Set("Origin", adminAddr)
-
-	err = apiRequest(req)
-	if err != nil {
-		caddy.Log().Warn("failed using API to stop instance",
-			zap.String("endpoint", stopEndpoint),
-			zap.Error(err),
-		)
+		caddy.Log().Warn("failed using API to stop instance", zap.Error(err))
 		return caddy.ExitCodeFailedStartup, err
 	}
 
@@ -288,7 +300,7 @@ func cmdReload(fl Flags) (int, error) {
 		return caddy.ExitCodeFailedStartup, fmt.Errorf("no config file to load")
 	}
 
-	// get the address of the admin listener and craft endpoint URL
+	// get the address of the admin listener; use flag if specified
 	adminAddr := reloadCmdAddrFlag
 	if adminAddr == "" && len(config) > 0 {
 		var tmpStruct struct {
@@ -301,20 +313,8 @@ func cmdReload(fl Flags) (int, error) {
 		}
 		adminAddr = tmpStruct.Admin.Listen
 	}
-	if adminAddr == "" {
-		adminAddr = caddy.DefaultAdminListen
-	}
-	loadEndpoint := fmt.Sprintf("http://%s/load", adminAddr)
 
-	// prepare the request to update the configuration
-	req, err := http.NewRequest(http.MethodPost, loadEndpoint, bytes.NewReader(config))
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("making request: %v", err)
-	}
-	req.Header.Set("Content-Type", "application/json")
-	req.Header.Set("Origin", adminAddr)
-
-	err = apiRequest(req)
+	err = apiRequest(adminAddr, http.MethodPost, "/load", bytes.NewReader(config))
 	if err != nil {
 		return caddy.ExitCodeFailedStartup, fmt.Errorf("sending configuration to instance: %v", err)
 	}
@@ -533,7 +533,17 @@ func cmdFmt(fl Flags) (int, error) {
 	if formatCmdConfigFile == "" {
 		formatCmdConfigFile = "Caddyfile"
 	}
-	overwrite := fl.Bool("overwrite")
+
+	// as a special case, read from stdin if the file name is "-"
+	if formatCmdConfigFile == "-" {
+		input, err := ioutil.ReadAll(os.Stdin)
+		if err != nil {
+			return caddy.ExitCodeFailedStartup,
+				fmt.Errorf("reading stdin: %v", err)
+		}
+		fmt.Print(string(caddyfile.Format(input)))
+		return caddy.ExitCodeSuccess, nil
+	}
 
 	input, err := ioutil.ReadFile(formatCmdConfigFile)
 	if err != nil {
@@ -543,9 +553,8 @@ func cmdFmt(fl Flags) (int, error) {
 
 	output := caddyfile.Format(input)
 
-	if overwrite {
-		err = ioutil.WriteFile(formatCmdConfigFile, output, 0644)
-		if err != nil {
+	if fl.Bool("overwrite") {
+		if err := ioutil.WriteFile(formatCmdConfigFile, output, 0600); err != nil {
 			return caddy.ExitCodeFailedStartup, nil
 		}
 	} else {
@@ -619,8 +628,62 @@ commands:
 	return caddy.ExitCodeSuccess, nil
 }
 
-func apiRequest(req *http.Request) error {
-	resp, err := http.DefaultClient.Do(req)
+// apiRequest makes an API request to the endpoint adminAddr with the
+// given HTTP method and request URI. If body is non-nil, it will be
+// assumed to be Content-Type application/json.
+func apiRequest(adminAddr, method, uri string, body io.Reader) error {
+	// parse the admin address
+	if adminAddr == "" {
+		adminAddr = caddy.DefaultAdminListen
+	}
+	parsedAddr, err := caddy.ParseNetworkAddress(adminAddr)
+	if err != nil || parsedAddr.PortRangeSize() > 1 {
+		return fmt.Errorf("invalid admin address %s: %v", adminAddr, err)
+	}
+	origin := parsedAddr.JoinHostPort(0)
+	if parsedAddr.IsUnixNetwork() {
+		origin = "unixsocket" // hack so that http.NewRequest() is happy
+	}
+
+	// form the request
+	req, err := http.NewRequest(method, "http://"+origin+uri, body)
+	if err != nil {
+		return fmt.Errorf("making request: %v", err)
+	}
+	if parsedAddr.IsUnixNetwork() {
+		// When listening on a unix socket, the admin endpoint doesn't
+		// accept any Host header because there is no host:port for
+		// a unix socket's address. The server's host check is fairly
+		// strict for security reasons, so we don't allow just any
+		// Host header. For unix sockets, the Host header must be
+		// empty. Unfortunately, Go makes it impossible to make HTTP
+		// requests with an empty Host header... except with this one
+		// weird trick. (Hopefully they don't fix it. It's already
+		// hard enough to use HTTP over unix sockets.)
+		//
+		// An equivalent curl command would be something like:
+		// $ curl --unix-socket caddy.sock http:/:$REQUEST_URI
+		req.URL.Host = " "
+		req.Host = ""
+	} else {
+		req.Header.Set("Origin", origin)
+	}
+	if body != nil {
+		req.Header.Set("Content-Type", "application/json")
+	}
+
+	// make an HTTP client that dials our network type, since admin
+	// endpoints aren't always TCP, which is what the default transport
+	// expects; reuse is not of particular concern here
+	client := http.Client{
+		Transport: &http.Transport{
+			DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
+				return net.Dial(parsedAddr.Network, parsedAddr.JoinHostPort(0))
+			},
+		},
+	}
+
+	resp, err := client.Do(req)
 	if err != nil {
 		return fmt.Errorf("performing request: %v", err)
 	}

cmd/commands.go

@@ -27,7 +27,7 @@ type Command struct {
 	// Required.
 	Name string
 
-	// Run is a function that executes a subcommand using
+	// Func is a function that executes a subcommand using
 	// the parsed flags. It returns an exit code and any
 	// associated error.
 	// Required.
@@ -74,7 +74,7 @@ func init() {
 	RegisterCommand(Command{
 		Name:  "start",
 		Func:  cmdStart,
-		Usage: "[--config <path> [--adapter <name>]] [--watch]",
+		Usage: "[--config <path> [--adapter <name>]] [--watch] [--pidfile <file>]",
 		Short: "Starts the Caddy process in the background and then returns",
 		Long: `
 Starts the Caddy process, optionally bootstrapped with an initial config file.
@@ -87,6 +87,7 @@ using 'caddy run' instead to keep it in the foreground.`,
 			fs := flag.NewFlagSet("start", flag.ExitOnError)
 			fs.String("config", "", "Configuration file")
 			fs.String("adapter", "", "Name of config adapter to apply")
+			fs.String("pidfile", "", "Path of file to which to write process ID")
 			fs.Bool("watch", false, "Reload changed config file automatically")
 			return fs
 		}(),
@@ -95,7 +96,7 @@ using 'caddy run' instead to keep it in the foreground.`,
 	RegisterCommand(Command{
 		Name:  "run",
 		Func:  cmdRun,
-		Usage: "[--config <path> [--adapter <name>]] [--environ] [--watch]",
+		Usage: "[--config <path> [--adapter <name>]] [--envfile <path>] [--environ] [--resume] [--watch] [--pidfile <fil>]",
 		Short: `Starts the Caddy process and blocks indefinitely`,
 		Long: `
 Starts the Caddy process, optionally bootstrapped with an initial config file,
@@ -115,6 +116,9 @@ As a special case, if the current working directory has a file called
 that file will be loaded and used to configure Caddy, even without any command
 line flags.
 
+If --envfile is specified, an environment file with environment variables in
+the KEY=VALUE format will be loaded into the Caddy process.
+
 If --environ is specified, the environment as seen by the Caddy process will
 be printed before starting. This is the same as the environ command but does
 not quit after printing, and can be useful for troubleshooting.
@@ -129,9 +133,11 @@ development environment.`,
 			fs := flag.NewFlagSet("run", flag.ExitOnError)
 			fs.String("config", "", "Configuration file")
 			fs.String("adapter", "", "Name of config adapter to apply")
+			fs.String("envfile", "", "Environment file to load")
 			fs.Bool("environ", false, "Print environment")
 			fs.Bool("resume", false, "Use saved config, if any (and prefer over --config file)")
 			fs.Bool("watch", false, "Watch config file for changes and reload it automatically")
+			fs.String("pidfile", "", "Path of file to which to write process ID")
 			fs.String("pingback", "", "Echo confirmation bytes to this address on success")
 			return fs
 		}(),
@@ -257,8 +263,12 @@ provisioning stages.`,
 Formats the Caddyfile by adding proper indentation and spaces to improve
 human readability. It prints the result to stdout.
 
-If --write is specified, the output will be written to the config file
-directly instead of printing it.`,
+If --overwrite is specified, the output will be written to the config file
+directly instead of printing it.
+
+If you wish you use stdin instead of a regular file, use - as the path.
+When reading from stdin, the --overwrite flag has no effect: the result
+is always printed to stdout.`,
 		Flags: func() *flag.FlagSet {
 			fs := flag.NewFlagSet("format", flag.ExitOnError)
 			fs.Bool("overwrite", false, "Overwrite the input file with the results")

cmd/main.go

@@ -15,15 +15,18 @@
 package caddycmd
 
 import (
+	"bufio"
 	"bytes"
 	"flag"
 	"fmt"
 	"io"
 	"io/ioutil"
+	"log"
 	"net"
 	"os"
 	"path/filepath"
 	"runtime"
+	"runtime/debug"
 	"strconv"
 	"strings"
 	"time"
@@ -48,8 +51,6 @@ func init() {
 // Main implements the main function of the caddy command.
 // Call this if Caddy is to be the main() if your program.
 func Main() {
-	caddy.TrapSignals()
-
 	switch len(os.Args) {
 	case 0:
 		fmt.Printf("[FATAL] no arguments provided by OS; args[0] must be command\n")
@@ -194,6 +195,12 @@ func loadConfig(configFile, adapterName string) ([]byte, string, error) {
 // long enough time. The filename passed in must be the actual
 // config file used, not one to be discovered.
 func watchConfigFile(filename, adapterName string) {
+	defer func() {
+		if err := recover(); err != nil {
+			log.Printf("[PANIC] watching config file: %v\n%s", err, debug.Stack())
+		}
+	}()
+
 	// make our logger; since config reloads can change the
 	// default logger, we need to get it dynamically each time
 	logger := func() *zap.Logger {
@@ -311,7 +318,7 @@ func (f Flags) Float64(name string) float64 {
 // is not a duration type. It panics if the flag is
 // not in the flag set.
 func (f Flags) Duration(name string) time.Duration {
-	val, _ := time.ParseDuration(f.String(name))
+	val, _ := caddy.ParseDuration(f.String(name))
 	return val
 }
 
@@ -331,11 +338,79 @@ func flagHelp(fs *flag.FlagSet) string {
 	return buf.String()
 }
 
+func loadEnvFromFile(envFile string) error {
+	file, err := os.Open(envFile)
+	if err != nil {
+		return fmt.Errorf("reading environment file: %v", err)
+	}
+	defer file.Close()
+
+	envMap, err := parseEnvFile(file)
+	if err != nil {
+		return fmt.Errorf("parsing environment file: %v", err)
+	}
+
+	for k, v := range envMap {
+		if err := os.Setenv(k, v); err != nil {
+			return fmt.Errorf("setting environment variables: %v", err)
+		}
+	}
+
+	return nil
+}
+
+func parseEnvFile(envInput io.Reader) (map[string]string, error) {
+	envMap := make(map[string]string)
+
+	scanner := bufio.NewScanner(envInput)
+	var line string
+	lineNumber := 0
+
+	for scanner.Scan() {
+		line = strings.TrimSpace(scanner.Text())
+		lineNumber++
+
+		// skip lines starting with comment
+		if strings.HasPrefix(line, "#") {
+			continue
+		}
+
+		// skip empty line
+		if len(line) == 0 {
+			continue
+		}
+
+		fields := strings.SplitN(line, "=", 2)
+		if len(fields) != 2 {
+			return nil, fmt.Errorf("can't parse line %d; line should be in KEY=VALUE format", lineNumber)
+		}
+
+		if strings.Contains(fields[0], " ") {
+			return nil, fmt.Errorf("bad key on line %d: contains whitespace", lineNumber)
+		}
+
+		key := fields[0]
+		val := fields[1]
+
+		if key == "" {
+			return nil, fmt.Errorf("missing or empty key on line %d", lineNumber)
+		}
+		envMap[key] = val
+	}
+
+	if err := scanner.Err(); err != nil {
+		return nil, err
+	}
+
+	return envMap, nil
+}
+
 func printEnvironment() {
 	fmt.Printf("caddy.HomeDir=%s\n", caddy.HomeDir())
 	fmt.Printf("caddy.AppDataDir=%s\n", caddy.AppDataDir())
 	fmt.Printf("caddy.AppConfigDir=%s\n", caddy.AppConfigDir())
 	fmt.Printf("caddy.ConfigAutosavePath=%s\n", caddy.ConfigAutosavePath)
+	fmt.Printf("caddy.Version=%s\n", caddy.GoModule().Version)
 	fmt.Printf("runtime.GOOS=%s\n", runtime.GOOS)
 	fmt.Printf("runtime.GOARCH=%s\n", runtime.GOARCH)
 	fmt.Printf("runtime.Compiler=%s\n", runtime.Compiler)

context.go

@@ -92,6 +92,7 @@ func (ctx *Context) OnCancel(f func()) {
 //
 //    json.RawMessage              => interface{}
 //    []json.RawMessage            => []interface{}
+//    [][]json.RawMessage          => [][]interface{}
 //    map[string]json.RawMessage   => map[string]interface{}
 //    []map[string]json.RawMessage => []map[string]interface{}
 //
@@ -179,6 +180,27 @@ func (ctx Context) LoadModule(structPointer interface{}, fieldName string) (inte
 			}
 			result = all
 
+		} else if typ.Elem().Kind() == reflect.Slice && isJSONRawMessage(typ.Elem().Elem()) {
+			// val is `[][]json.RawMessage`
+
+			if inlineModuleKey == "" {
+				panic("unable to determine module name without inline_key because type is not a ModuleMap")
+			}
+			var all [][]interface{}
+			for i := 0; i < val.Len(); i++ {
+				innerVal := val.Index(i)
+				var allInner []interface{}
+				for j := 0; j < innerVal.Len(); j++ {
+					innerInnerVal, err := ctx.loadModuleInline(inlineModuleKey, moduleNamespace, innerVal.Index(j).Interface().(json.RawMessage))
+					if err != nil {
+						return nil, fmt.Errorf("position %d: %v", j, err)
+					}
+					allInner = append(allInner, innerInnerVal)
+				}
+				all = append(all, allInner)
+			}
+			result = all
+
 		} else if isModuleMapType(typ.Elem()) {
 			// val is `[]map[string]json.RawMessage`
 

go.mod

@@ -3,36 +3,32 @@ module github.com/caddyserver/caddy/v2
 go 1.14
 
 require (
-	github.com/Masterminds/sprig/v3 v3.0.2
-	github.com/alecthomas/chroma v0.7.2-0.20200305040604-4f3623dce67a
+	github.com/Masterminds/sprig/v3 v3.1.0
+	github.com/alecthomas/chroma v0.8.0
 	github.com/aryann/difflib v0.0.0-20170710044230-e206f873d14a
-	github.com/caddyserver/certmagic v0.10.11
-	github.com/cenkalti/backoff/v4 v4.0.2 // indirect
+	github.com/caddyserver/certmagic v0.12.0
 	github.com/dustin/go-humanize v1.0.1-0.20200219035652-afde56e7acac
-	github.com/go-acme/lego/v3 v3.5.0
-	github.com/gogo/protobuf v1.3.1
-	github.com/google/cel-go v0.4.1
-	github.com/imdario/mergo v0.3.9 // indirect
+	github.com/go-chi/chi v4.1.2+incompatible
+	github.com/google/cel-go v0.5.1
 	github.com/jsternberg/zap-logfmt v1.2.0
-	github.com/klauspost/compress v1.10.4
-	github.com/klauspost/cpuid v1.2.3
-	github.com/lucas-clemente/quic-go v0.15.3
-	github.com/manifoldco/promptui v0.7.0 // indirect
-	github.com/miekg/dns v1.1.29 // indirect
+	github.com/klauspost/compress v1.11.0
+	github.com/klauspost/cpuid v1.2.5 // cannot upgrade until arm is fixed: https://github.com/klauspost/cpuid/issues/52
+	github.com/lucas-clemente/quic-go v0.18.0
+	github.com/mholt/acmez v0.1.1
 	github.com/naoina/go-stringutil v0.1.0 // indirect
 	github.com/naoina/toml v0.1.1
-	github.com/smallstep/certificates v0.14.1
-	github.com/smallstep/cli v0.14.1
-	github.com/smallstep/truststore v0.9.5
-	github.com/vulcand/oxy v1.1.0
-	github.com/yuin/goldmark v1.1.27
+	github.com/prometheus/client_golang v1.7.1
+	github.com/smallstep/certificates v0.15.4
+	github.com/smallstep/cli v0.15.2
+	github.com/smallstep/nosql v0.3.0 // cannot upgrade until protobuf warning is fixed
+	github.com/smallstep/truststore v0.9.6
+	github.com/yuin/goldmark v1.2.1
 	github.com/yuin/goldmark-highlighting v0.0.0-20200307114337-60d527fdb691
-	go.uber.org/zap v1.14.1
-	golang.org/x/crypto v0.0.0-20200406173513-056763e48d71
-	golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e
-	golang.org/x/sys v0.0.0-20200409092240-59c9f1ba88fa // indirect
-	google.golang.org/genproto v0.0.0-20200409111301-baae70f3302d
+	go.uber.org/zap v1.15.0
+	golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de
+	golang.org/x/net v0.0.0-20200707034311-ab3426394381
+	google.golang.org/genproto v0.0.0-20200806141610-86f49bd18e98
+	google.golang.org/protobuf v1.24.0 // cannot upgrade until warning is fixed
 	gopkg.in/natefinch/lumberjack.v2 v2.0.0
-	gopkg.in/square/go-jose.v2 v2.4.1 // indirect
-	gopkg.in/yaml.v2 v2.2.8
+	gopkg.in/yaml.v2 v2.3.0
 )

listeners.go

@@ -289,14 +289,41 @@ func (na NetworkAddress) PortRangeSize() uint {
 	return (na.EndPort - na.StartPort) + 1
 }
 
-// String reconstructs the address string to the form expected
-// by ParseNetworkAddress().
-func (na NetworkAddress) String() string {
-	port := strconv.FormatUint(uint64(na.StartPort), 10)
-	if na.StartPort != na.EndPort {
-		port += "-" + strconv.FormatUint(uint64(na.EndPort), 10)
+func (na NetworkAddress) isLoopback() bool {
+	if na.IsUnixNetwork() {
+		return true
 	}
-	return JoinNetworkAddress(na.Network, na.Host, port)
+	if na.Host == "localhost" {
+		return true
+	}
+	if ip := net.ParseIP(na.Host); ip != nil {
+		return ip.IsLoopback()
+	}
+	return false
+}
+
+func (na NetworkAddress) isWildcardInterface() bool {
+	if na.Host == "" {
+		return true
+	}
+	if ip := net.ParseIP(na.Host); ip != nil {
+		return ip.IsUnspecified()
+	}
+	return false
+}
+
+func (na NetworkAddress) port() string {
+	if na.StartPort == na.EndPort {
+		return strconv.FormatUint(uint64(na.StartPort), 10)
+	}
+	return fmt.Sprintf("%d-%d", na.StartPort, na.EndPort)
+}
+
+// String reconstructs the address string to the form expected
+// by ParseNetworkAddress(). If the address is a unix socket,
+// any non-zero port will be dropped.
+func (na NetworkAddress) String() string {
+	return JoinNetworkAddress(na.Network, na.Host, na.port())
 }
 
 func isUnixNetwork(netw string) bool {
@@ -378,7 +405,7 @@ func JoinNetworkAddress(network, host, port string) string {
 	if network != "" {
 		a = network + "/"
 	}
-	if host != "" && port == "" {
+	if (host != "" && port == "") || isUnixNetwork(network) {
 		a += host
 	} else if port != "" {
 		a += net.JoinHostPort(host, port)

listeners_test.go

@@ -138,6 +138,14 @@ func TestJoinNetworkAddress(t *testing.T) {
 			network: "unix", host: "/foo/bar", port: "",
 			expect: "unix//foo/bar",
 		},
+		{
+			network: "unix", host: "/foo/bar", port: "0",
+			expect: "unix//foo/bar",
+		},
+		{
+			network: "unix", host: "/foo/bar", port: "1234",
+			expect: "unix//foo/bar",
+		},
 		{
 			network: "", host: "::1", port: "1234",
 			expect: "[::1]:1234",

metrics.go

@@ -0,0 +1,77 @@
+package caddy
+
+import (
+	"net/http"
+	"strconv"
+	"strings"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promauto"
+)
+
+// define and register the metrics used in this package.
+func init() {
+	prometheus.MustRegister(prometheus.NewBuildInfoCollector())
+
+	const ns, sub = "caddy", "admin"
+
+	adminMetrics.requestCount = promauto.NewCounterVec(prometheus.CounterOpts{
+		Namespace: ns,
+		Subsystem: sub,
+		Name:      "http_requests_total",
+		Help:      "Counter of requests made to the Admin API's HTTP endpoints.",
+	}, []string{"handler", "path", "code", "method"})
+	adminMetrics.requestErrors = promauto.NewCounterVec(prometheus.CounterOpts{
+		Namespace: ns,
+		Subsystem: sub,
+		Name:      "http_request_errors_total",
+		Help:      "Number of requests resulting in middleware errors.",
+	}, []string{"handler", "path", "method"})
+}
+
+// adminMetrics is a collection of metrics that can be tracked for the admin API.
+var adminMetrics = struct {
+	requestCount  *prometheus.CounterVec
+	requestErrors *prometheus.CounterVec
+}{}
+
+// Similar to promhttp.InstrumentHandlerCounter, but upper-cases method names
+// instead of lower-casing them.
+//
+// Unlike promhttp.InstrumentHandlerCounter, this assumes a "code" and "method"
+// label is present, and will panic otherwise.
+func instrumentHandlerCounter(counter *prometheus.CounterVec, next http.Handler) http.HandlerFunc {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		d := newDelegator(w)
+		next.ServeHTTP(d, r)
+		counter.With(prometheus.Labels{
+			"code":   sanitizeCode(d.status),
+			"method": strings.ToUpper(r.Method),
+		}).Inc()
+	})
+}
+
+func newDelegator(w http.ResponseWriter) *delegator {
+	return &delegator{
+		ResponseWriter: w,
+	}
+}
+
+type delegator struct {
+	http.ResponseWriter
+	status int
+}
+
+func (d *delegator) WriteHeader(code int) {
+	d.status = code
+	d.ResponseWriter.WriteHeader(code)
+}
+
+func sanitizeCode(s int) string {
+	switch s {
+	case 0, 200:
+		return "200"
+	default:
+		return strconv.Itoa(s)
+	}
+}

modules.go

@@ -65,7 +65,12 @@ type ModuleInfo struct {
 
 	// New returns a pointer to a new, empty
 	// instance of the module's type. This
-	// function must not have any side-effects.
+	// method must not have any side-effects,
+	// and no other initialization should
+	// occur within it. Any initialization
+	// of the returned value should be done
+	// in a Provision() method (see the
+	// Provisioner interface).
 	New func() Module
 }
 
@@ -125,33 +130,32 @@ type ModuleMap map[string]json.RawMessage
 // be properly recorded, this should be called in the
 // init phase of runtime. Typically, the module package
 // will do this as a side-effect of being imported.
-// This function returns an error if the module's info
-// is incomplete or invalid, or if the module is
-// already registered.
-func RegisterModule(instance Module) error {
+// This function panics if the module's info is
+// incomplete or invalid, or if the module is already
+// registered.
+func RegisterModule(instance Module) {
 	mod := instance.CaddyModule()
 
 	if mod.ID == "" {
-		return fmt.Errorf("module ID missing")
+		panic("module ID missing")
 	}
 	if mod.ID == "caddy" || mod.ID == "admin" {
-		return fmt.Errorf("module ID '%s' is reserved", mod.ID)
+		panic(fmt.Sprintf("module ID '%s' is reserved", mod.ID))
 	}
 	if mod.New == nil {
-		return fmt.Errorf("missing ModuleInfo.New")
+		panic("missing ModuleInfo.New")
 	}
 	if val := mod.New(); val == nil {
-		return fmt.Errorf("ModuleInfo.New must return a non-nil module instance")
+		panic("ModuleInfo.New must return a non-nil module instance")
 	}
 
 	modulesMu.Lock()
 	defer modulesMu.Unlock()
 
 	if _, ok := modules[string(mod.ID)]; ok {
-		return fmt.Errorf("module already registered: %s", mod.ID)
+		panic(fmt.Sprintf("module already registered: %s", mod.ID))
 	}
 	modules[string(mod.ID)] = mod
-	return nil
 }
 
 // GetModule returns module information from its ID (full name).

modules/caddyhttp/app.go

@@ -27,13 +27,12 @@ import (
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
 	"github.com/lucas-clemente/quic-go/http3"
 	"go.uber.org/zap"
+	"golang.org/x/net/http2"
+	"golang.org/x/net/http2/h2c"
 )
 
 func init() {
-	err := caddy.RegisterModule(App{})
-	if err != nil {
-		caddy.Log().Fatal(err.Error())
-	}
+	caddy.RegisterModule(App{})
 }
 
 // App is a robust, production-ready HTTP server.
@@ -48,6 +47,7 @@ func init() {
 //
 // Placeholder | Description
 // ------------|---------------
+// `{http.request.body}` | The request body (⚠️ inefficient; use only for debugging)
 // `{http.request.cookie.*}` | HTTP request cookie
 // `{http.request.header.*}` | Specific request header field
 // `{http.request.host.labels.*}` | Request host labels (0-based from right); e.g. for foo.example.com: 0=com, 1=example, 2=foo
@@ -73,9 +73,15 @@ func init() {
 // `{http.request.tls.proto_mutual}` | The negotiated next protocol was advertised by the server
 // `{http.request.tls.server_name}` | The server name requested by the client, if any
 // `{http.request.tls.client.fingerprint}` | The SHA256 checksum of the client certificate
+// `{http.request.tls.client.public_key}` | The public key of the client certificate.
+// `{http.request.tls.client.public_key_sha256}` | The SHA256 checksum of the client's public key.
 // `{http.request.tls.client.issuer}` | The issuer DN of the client certificate
 // `{http.request.tls.client.serial}` | The serial number of the client certificate
 // `{http.request.tls.client.subject}` | The subject DN of the client certificate
+// `{http.request.tls.client.san.dns_names.*}` | SAN DNS names(index optional)
+// `{http.request.tls.client.san.emails.*}` | SAN email addresses (index optional)
+// `{http.request.tls.client.san.ips.*}` | SAN IP addresses (index optional)
+// `{http.request.tls.client.san.uris.*}` | SAN URIs (index optional)
 // `{http.request.uri.path.*}` | Parts of the path, split by `/` (0-based from left)
 // `{http.request.uri.path.dir}` | The directory, excluding leaf filename
 // `{http.request.uri.path.file}` | The filename of the path, excluding directory
@@ -149,6 +155,7 @@ func (app *App) Provision(ctx caddy.Context) error {
 
 	// prepare each server
 	for srvName, srv := range app.Servers {
+		srv.name = srvName
 		srv.tlsApp = app.tlsApp
 		srv.logger = app.logger.Named("log")
 		srv.errorLogger = app.logger.Named("log.error")
@@ -275,6 +282,12 @@ func (app *App) Validate() error {
 // Start runs the app. It finishes automatic HTTPS if enabled,
 // including management of certificates.
 func (app *App) Start() error {
+	// get a logger compatible with http.Server
+	serverLogger, err := zap.NewStdLogAt(app.logger.Named("stdlib"), zap.DebugLevel)
+	if err != nil {
+		return fmt.Errorf("failed to set up server logger: %v", err)
+	}
+
 	for srvName, srv := range app.Servers {
 		s := &http.Server{
 			ReadTimeout:       time.Duration(srv.ReadTimeout),
@@ -283,6 +296,15 @@ func (app *App) Start() error {
 			IdleTimeout:       time.Duration(srv.IdleTimeout),
 			MaxHeaderBytes:    srv.MaxHeaderBytes,
 			Handler:           srv,
+			ErrorLog:          serverLogger,
+		}
+
+		// enable h2c if configured
+		if srv.AllowH2C {
+			h2server := &http2.Server{
+				IdleTimeout: time.Duration(srv.IdleTimeout),
+			}
+			s.Handler = h2c.NewHandler(srv, h2server)
 		}
 
 		for _, lnAddr := range srv.Listen {
@@ -330,6 +352,7 @@ func (app *App) Start() error {
 								Addr:      hostport,
 								Handler:   srv,
 								TLSConfig: tlsCfg,
+								ErrorLog:  serverLogger,
 							},
 						}
 						go h3srv.Serve(h3ln)
@@ -368,7 +391,7 @@ func (app *App) Start() error {
 
 	// finish automatic HTTPS by finally beginning
 	// certificate management
-	err := app.automaticHTTPSPhase2()
+	err = app.automaticHTTPSPhase2()
 	if err != nil {
 		return fmt.Errorf("finalizing automatic HTTPS: %v", err)
 	}

modules/caddyhttp/autohttps.go

@@ -81,8 +81,10 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 	uniqueDomainsForCerts := make(map[string]struct{})
 
 	// this maps domain names for automatic HTTP->HTTPS
-	// redirects to their destination server address
-	redirDomains := make(map[string]caddy.NetworkAddress)
+	// redirects to their destination server addresses
+	// (there might be more than 1 if bind is used; see
+	// https://github.com/caddyserver/caddy/issues/3443)
+	redirDomains := make(map[string][]caddy.NetworkAddress)
 
 	for srvName, srv := range app.Servers {
 		// as a prerequisite, provision route matchers; this is
@@ -129,6 +131,7 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 		}
 
 		// find all qualifying domain names (deduplicated) in this server
+		// (this is where we need the provisioned, decoded request matchers)
 		serverDomainSet := make(map[string]struct{})
 		for routeIdx, route := range srv.Routes {
 			for matcherSetIdx, matcherSet := range route.MatcherSets {
@@ -150,9 +153,14 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 			}
 		}
 
-		// nothing more to do here if there are no
-		// domains that qualify for automatic HTTPS
-		if len(serverDomainSet) == 0 {
+		// nothing more to do here if there are no domains that qualify for
+		// automatic HTTPS and there are no explicit TLS connection policies:
+		// if there is at least one domain but no TLS conn policy (F&&T), we'll
+		// add one below; if there are no domains but at least one TLS conn
+		// policy (meaning TLS is enabled) (T&&F), it could be a catch-all with
+		// on-demand TLS -- and in that case we would still need HTTP->HTTPS
+		// redirects, which we set up below; hence these two conditions
+		if len(serverDomainSet) == 0 && len(srv.TLSConnPolicies) == 0 {
 			continue
 		}
 
@@ -207,13 +215,24 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 				return fmt.Errorf("%s: invalid listener address: %v", srvName, addr)
 			}
 
+			// this address might not have a hostname, i.e. might be a
+			// catch-all address for a particular port; we need to keep
+			// track if it is, so we can set up redirects for it anyway
+			// (e.g. the user might have enabled on-demand TLS); we use
+			// an empty string to indicate a catch-all, which we have to
+			// treat special later
+			if len(serverDomainSet) == 0 {
+				redirDomains[""] = append(redirDomains[""], addr)
+				continue
+			}
+
 			// ...and associate it with each domain in this server
 			for d := range serverDomainSet {
 				// if this domain is used on more than one HTTPS-enabled
 				// port, we'll have to choose one, so prefer the HTTPS port
 				if _, ok := redirDomains[d]; !ok ||
 					addr.StartPort == uint(app.httpsPort()) {
-					redirDomains[d] = addr
+					redirDomains[d] = append(redirDomains[d], addr)
 				}
 			}
 		}
@@ -258,34 +277,32 @@ uniqueDomainsLoop:
 		return err
 	}
 
-	// we're done if there are no HTTP->HTTPS redirects to add
-	if len(redirDomains) == 0 {
-		return nil
-	}
-
 	// we need to reduce the mapping, i.e. group domains by address
 	// since new routes are appended to servers by their address
 	domainsByAddr := make(map[string][]string)
-	for domain, addr := range redirDomains {
-		addrStr := addr.String()
-		domainsByAddr[addrStr] = append(domainsByAddr[addrStr], domain)
+	for domain, addrs := range redirDomains {
+		for _, addr := range addrs {
+			addrStr := addr.String()
+			domainsByAddr[addrStr] = append(domainsByAddr[addrStr], domain)
+		}
 	}
 
 	// these keep track of the redirect server address(es)
 	// and the routes for those servers which actually
 	// respond with the redirects
 	redirServerAddrs := make(map[string]struct{})
+	redirServers := make(map[string][]Route)
 	var redirRoutes RouteList
 
-	redirServers := make(map[string][]Route)
-
 	for addrStr, domains := range domainsByAddr {
-		// build the matcher set for this redirect route
-		// (note that we happen to bypass Provision and
-		// Validate steps for these matcher modules)
-		matcherSet := MatcherSet{
-			MatchProtocol("http"),
-			MatchHost(domains),
+		// build the matcher set for this redirect route; (note that we happen
+		// to bypass Provision and Validate steps for these matcher modules)
+		matcherSet := MatcherSet{MatchProtocol("http")}
+		// match on known domain names, unless it's our special case of a
+		// catch-all which is an empty string (common among catch-all sites
+		// that enable on-demand TLS for yet-unknown domain names)
+		if !(len(domains) == 1 && domains[0] == "") {
+			matcherSet = append(matcherSet, MatchHost(domains))
 		}
 
 		// build the address to which to redirect
@@ -332,6 +349,13 @@ uniqueDomainsLoop:
 	// not entirely clear what the redirect destination should be,
 	// so I'm going to just hard-code the app's HTTPS port and call
 	// it good for now...
+	// TODO: This implies that all plaintext requests will be blindly
+	// redirected to their HTTPS equivalent, even if this server
+	// doesn't handle that hostname at all; I don't think this is a
+	// bad thing, and it also obscures the actual hostnames that this
+	// server is configured to match on, which may be desirable, but
+	// it's not something that should be relied on. We can change this
+	// if we want to.
 	appendCatchAll := func(routes []Route) []Route {
 		redirTo := "https://{http.request.host}"
 		if app.httpsPort() != DefaultHTTPSPort {
@@ -427,8 +451,8 @@ func (app *App) createAutomationPolicies(ctx caddy.Context, publicNames, interna
 		if ap.Issuer == nil {
 			ap.Issuer = new(caddytls.ACMEIssuer)
 		}
-		if acmeIssuer, ok := ap.Issuer.(*caddytls.ACMEIssuer); ok {
-			err := app.fillInACMEIssuer(acmeIssuer)
+		if acmeIssuer, ok := ap.Issuer.(acmeCapable); ok {
+			err := app.fillInACMEIssuer(acmeIssuer.GetACMEIssuer())
 			if err != nil {
 				return err
 			}
@@ -446,9 +470,13 @@ func (app *App) createAutomationPolicies(ctx caddy.Context, publicNames, interna
 		basePolicy = new(caddytls.AutomationPolicy)
 	}
 
-	// if the basePolicy has an existing ACMEIssuer, let's
-	// use it, otherwise we'll make one
-	baseACMEIssuer, _ := basePolicy.Issuer.(*caddytls.ACMEIssuer)
+	// if the basePolicy has an existing ACMEIssuer (particularly to
+	// include any type that embeds/wraps an ACMEIssuer), let's use it,
+	// otherwise we'll make one
+	var baseACMEIssuer *caddytls.ACMEIssuer
+	if acmeWrapper, ok := basePolicy.Issuer.(acmeCapable); ok {
+		baseACMEIssuer = acmeWrapper.GetACMEIssuer()
+	}
 	if baseACMEIssuer == nil {
 		// note that this happens if basePolicy.Issuer is nil
 		// OR if it is not nil but is not an ACMEIssuer
@@ -606,3 +634,5 @@ func (app *App) automaticHTTPSPhase2() error {
 	app.allCertDomains = nil // no longer needed; allow GC to deallocate
 	return nil
 }
+
+type acmeCapable interface{ GetACMEIssuer() *caddytls.ACMEIssuer }

modules/caddyhttp/caddyauth/basicauth.go

@@ -16,15 +16,21 @@ package caddyauth
 
 import (
 	"encoding/base64"
+	"encoding/hex"
 	"encoding/json"
 	"fmt"
+	weakrand "math/rand"
 	"net/http"
+	"sync"
+	"time"
 
 	"github.com/caddyserver/caddy/v2"
 )
 
 func init() {
 	caddy.RegisterModule(HTTPBasicAuth{})
+
+	weakrand.Seed(time.Now().UnixNano())
 }
 
 // HTTPBasicAuth facilitates HTTP basic authentication.
@@ -38,6 +44,17 @@ type HTTPBasicAuth struct {
 	// The name of the realm. Default: restricted
 	Realm string `json:"realm,omitempty"`
 
+	// If non-nil, a mapping of plaintext passwords to their
+	// hashes will be cached in memory (with random eviction).
+	// This can greatly improve the performance of traffic-heavy
+	// servers that use secure password hashing algorithms, with
+	// the downside that plaintext passwords will be stored in
+	// memory for a longer time (this should not be a problem
+	// as long as your machine is not compromised, at which point
+	// all bets are off, since basicauth necessitates plaintext
+	// passwords being received over the wire anyway).
+	HashCache *Cache `json:"hash_cache,omitempty"`
+
 	Accounts map[string]Account `json:"-"`
 	Hash     Comparer           `json:"-"`
 }
@@ -99,6 +116,11 @@ func (hba *HTTPBasicAuth) Provision(ctx caddy.Context) error {
 	}
 	hba.AccountList = nil // allow GC to deallocate
 
+	if hba.HashCache != nil {
+		hba.HashCache.cache = make(map[string]bool)
+		hba.HashCache.mu = new(sync.Mutex)
+	}
+
 	return nil
 }
 
@@ -109,13 +131,11 @@ func (hba HTTPBasicAuth) Authenticate(w http.ResponseWriter, req *http.Request)
 		return hba.promptForCredentials(w, nil)
 	}
 
-	plaintextPassword := []byte(plaintextPasswordStr)
-
 	account, accountExists := hba.Accounts[username]
 	// don't return early if account does not exist; we want
 	// to try to avoid side-channels that leak existence
 
-	same, err := hba.Hash.Compare(account.password, plaintextPassword, account.salt)
+	same, err := hba.correctPassword(account, []byte(plaintextPasswordStr))
 	if err != nil {
 		return hba.promptForCredentials(w, err)
 	}
@@ -126,6 +146,43 @@ func (hba HTTPBasicAuth) Authenticate(w http.ResponseWriter, req *http.Request)
 	return User{ID: username}, true, nil
 }
 
+func (hba HTTPBasicAuth) correctPassword(account Account, plaintextPassword []byte) (bool, error) {
+	compare := func() (bool, error) {
+		return hba.Hash.Compare(account.password, plaintextPassword, account.salt)
+	}
+
+	// if no caching is enabled, simply return the result of hashing + comparing
+	if hba.HashCache == nil {
+		return compare()
+	}
+
+	// compute a cache key that is unique for these input parameters
+	cacheKey := hex.EncodeToString(append(append(account.password, account.salt...), plaintextPassword...))
+
+	// fast track: if the result of the input is already cached, use it
+	hba.HashCache.mu.Lock()
+	same, ok := hba.HashCache.cache[cacheKey]
+	if ok {
+		hba.HashCache.mu.Unlock()
+		return same, nil
+	}
+	hba.HashCache.mu.Unlock()
+
+	// slow track: do the expensive op, then add it to the cache
+	same, err := compare()
+	if err != nil {
+		return false, err
+	}
+	hba.HashCache.mu.Lock()
+	if len(hba.HashCache.cache) >= 1000 {
+		hba.HashCache.makeRoom() // keep cache size under control
+	}
+	hba.HashCache.cache[cacheKey] = same
+	hba.HashCache.mu.Unlock()
+
+	return same, nil
+}
+
 func (hba HTTPBasicAuth) promptForCredentials(w http.ResponseWriter, err error) (User, bool, error) {
 	// browsers show a message that says something like:
 	// "The website says: <realm>"
@@ -138,6 +195,47 @@ func (hba HTTPBasicAuth) promptForCredentials(w http.ResponseWriter, err error)
 	return User{}, false, err
 }
 
+// Cache enables caching of basic auth results. This is especially
+// helpful for secure password hashes which can be expensive to
+// compute on every HTTP request.
+type Cache struct {
+	mu *sync.Mutex
+
+	// map of concatenated hashed password + plaintext password + salt, to result
+	cache map[string]bool
+}
+
+// makeRoom deletes about 1/10 of the items in the cache
+// in order to keep its size under control. It must not be
+// called without a lock on c.mu.
+func (c *Cache) makeRoom() {
+	// we delete more than just 1 entry so that we don't have
+	// to do this on every request; assuming the capacity of
+	// the cache is on a long tail, we can save a lot of CPU
+	// time by doing a whole bunch of deletions now and then
+	// we won't have to do them again for a while
+	numToDelete := len(c.cache) / 10
+	if numToDelete < 1 {
+		numToDelete = 1
+	}
+	for deleted := 0; deleted <= numToDelete; deleted++ {
+		// Go maps are "nondeterministic" not actually random,
+		// so although we could just chop off the "front" of the
+		// map with less code, this is a heavily skewed eviction
+		// strategy; generating random numbers is cheap and
+		// ensures a much better distribution.
+		rnd := weakrand.Intn(len(c.cache))
+		i := 0
+		for key := range c.cache {
+			if i == rnd {
+				delete(c.cache, key)
+				break
+			}
+			i++
+		}
+	}
+}
+
 // Comparer is a type that can securely compare
 // a plaintext password with a hashed password
 // in constant-time. Comparers should hash the

modules/caddyhttp/caddyauth/caddyauth.go

@@ -29,6 +29,8 @@ func init() {
 
 // Authentication is a middleware which provides user authentication.
 // Rejects requests with HTTP 401 if the request is not authenticated.
+//
+// Its API is still experimental and may be subject to change.
 type Authentication struct {
 	// A set of authentication providers. If none are specified,
 	// all requests will always be unauthenticated.

modules/caddyhttp/caddyauth/caddyfile.go

@@ -27,7 +27,7 @@ func init() {
 
 // parseCaddyfile sets up the handler from Caddyfile tokens. Syntax:
 //
-//     basicauth [<matcher>] [<hash_algorithm>] {
+//     basicauth [<matcher>] [<hash_algorithm> [<realm>]] {
 //         <username> <hashed_password_base64> [<salt_base64>]
 //         ...
 //     }
@@ -35,6 +35,7 @@ func init() {
 // If no hash algorithm is supplied, bcrypt will be assumed.
 func parseCaddyfile(h httpcaddyfile.Helper) (caddyhttp.MiddlewareHandler, error) {
 	var ba HTTPBasicAuth
+	ba.HashCache = new(Cache)
 
 	for h.Next() {
 		var cmp Comparer
@@ -46,6 +47,9 @@ func parseCaddyfile(h httpcaddyfile.Helper) (caddyhttp.MiddlewareHandler, error)
 			hashName = "bcrypt"
 		case 1:
 			hashName = args[0]
+		case 2:
+			hashName = args[0]
+			ba.Realm = args[1]
 		default:
 			return nil, h.ArgErr()
 		}

modules/caddyhttp/caddyauth/command.go

@@ -15,26 +15,35 @@
 package caddyauth
 
 import (
+	"bufio"
+	"bytes"
 	"encoding/base64"
 	"flag"
 	"fmt"
+	"os"
+	"os/signal"
 
 	"github.com/caddyserver/caddy/v2"
 	caddycmd "github.com/caddyserver/caddy/v2/cmd"
 	"golang.org/x/crypto/bcrypt"
 	"golang.org/x/crypto/scrypt"
+	"golang.org/x/crypto/ssh/terminal"
 )
 
 func init() {
 	caddycmd.RegisterCommand(caddycmd.Command{
 		Name:  "hash-password",
 		Func:  cmdHashPassword,
-		Usage: "--plaintext <password> [--salt <string>] [--algorithm <name>]",
+		Usage: "[--algorithm <name>] [--salt <string>] [--plaintext <password>]",
 		Short: "Hashes a password and writes base64",
 		Long: `
 Convenient way to hash a plaintext password. The resulting
 hash is written to stdout as a base64 string.
 
+--plaintext, when omitted, will be read from stdin. If
+Caddy is attached to a controlling tty, the plaintext will
+not be echoed.
+
 --algorithm may be bcrypt or scrypt. If script, the default
 parameters are used.
 
@@ -52,19 +61,62 @@ be provided (scrypt).
 }
 
 func cmdHashPassword(fs caddycmd.Flags) (int, error) {
+	var err error
+
 	algorithm := fs.String("algorithm")
 	plaintext := []byte(fs.String("plaintext"))
 	salt := []byte(fs.String("salt"))
 
 	if len(plaintext) == 0 {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("password is required")
+		fd := int(os.Stdin.Fd())
+		if terminal.IsTerminal(fd) {
+			// ensure the terminal state is restored on SIGINT
+			state, _ := terminal.GetState(fd)
+			c := make(chan os.Signal)
+			signal.Notify(c, os.Interrupt)
+			go func() {
+				<-c
+				_ = terminal.Restore(fd, state)
+				os.Exit(caddy.ExitCodeFailedStartup)
+			}()
+			defer signal.Stop(c)
+
+			fmt.Fprint(os.Stderr, "Enter password: ")
+			plaintext, err = terminal.ReadPassword(fd)
+			fmt.Fprintln(os.Stderr)
+			if err != nil {
+				return caddy.ExitCodeFailedStartup, err
+			}
+
+			fmt.Fprint(os.Stderr, "Confirm password: ")
+			confirmation, err := terminal.ReadPassword(fd)
+			fmt.Fprintln(os.Stderr)
+			if err != nil {
+				return caddy.ExitCodeFailedStartup, err
+			}
+
+			if !bytes.Equal(plaintext, confirmation) {
+				return caddy.ExitCodeFailedStartup, fmt.Errorf("password does not match")
+			}
+		} else {
+			rd := bufio.NewReader(os.Stdin)
+			plaintext, err = rd.ReadBytes('\n')
+			if err != nil {
+				return caddy.ExitCodeFailedStartup, err
+			}
+
+			plaintext = plaintext[:len(plaintext)-1] // Trailing newline
+		}
+
+		if len(plaintext) == 0 {
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("plaintext is required")
+		}
 	}
 
 	var hash []byte
-	var err error
 	switch algorithm {
 	case "bcrypt":
-		hash, err = bcrypt.GenerateFromPassword(plaintext, bcrypt.DefaultCost)
+		hash, err = bcrypt.GenerateFromPassword(plaintext, 14)
 	case "scrypt":
 		def := ScryptHash{}
 		def.SetDefaults()

modules/caddyhttp/caddyhttp.go

@@ -30,10 +30,7 @@ import (
 func init() {
 	weakrand.Seed(time.Now().UnixNano())
 
-	err := caddy.RegisterModule(tlsPlaceholderWrapper{})
-	if err != nil {
-		caddy.Log().Fatal(err.Error())
-	}
+	caddy.RegisterModule(tlsPlaceholderWrapper{})
 }
 
 // RequestMatcher is a type that can match to a request.
@@ -95,6 +92,45 @@ var errorEmptyHandler Handler = HandlerFunc(func(w http.ResponseWriter, r *http.
 	return nil
 })
 
+// ResponseHandler pairs a response matcher with custom handling
+// logic. Either the status code can be changed to something else
+// while using the original response body, or, if a status code
+// is not set, it can execute a custom route list; this is useful
+// for executing handler routes based on the properties of an HTTP
+// response that has not been written out to the client yet.
+//
+// To use this type, provision it at module load time, then when
+// ready to use, match the response against its matcher; if it
+// matches (or doesn't have a matcher), change the status code on
+// the response if configured; otherwise invoke the routes by
+// calling `rh.Routes.Compile(next).ServeHTTP(rw, req)` (or similar).
+type ResponseHandler struct {
+	// The response matcher for this handler. If empty/nil,
+	// it always matches.
+	Match *ResponseMatcher `json:"match,omitempty"`
+
+	// To write the original response body but with a different
+	// status code, set this field to the desired status code.
+	// If set, this takes priority over routes.
+	StatusCode WeakString `json:"status_code,omitempty"`
+
+	// The list of HTTP routes to execute if no status code is
+	// specified. If evaluated, the original response body
+	// will not be written.
+	Routes RouteList `json:"routes,omitempty"`
+}
+
+// Provision sets up the routse in rh.
+func (rh *ResponseHandler) Provision(ctx caddy.Context) error {
+	if rh.Routes != nil {
+		err := rh.Routes.Provision(ctx)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
 // WeakString is a type that unmarshals any JSON value
 // as a string literal, with the following exceptions:
 //
@@ -168,19 +204,6 @@ func (ws WeakString) String() string {
 	return string(ws)
 }
 
-// CopyHeader copies HTTP headers by completely
-// replacing dest with src. (This allows deletions
-// to be propagated, assuming src started as a
-// consistent copy of dest.)
-func CopyHeader(dest, src http.Header) {
-	for field := range dest {
-		delete(dest, field)
-	}
-	for field, val := range src {
-		dest[field] = val
-	}
-}
-
 // StatusCodeMatches returns true if a real HTTP status code matches
 // the configured status code, which may be either a real HTTP status
 // code or an integer representing a class of codes (e.g. 4 for all

modules/caddyhttp/celmatcher.go

@@ -15,16 +15,17 @@
 package caddyhttp
 
 import (
+	"crypto/x509/pkix"
 	"encoding/json"
 	"fmt"
 	"net/http"
 	"reflect"
 	"regexp"
 	"strings"
+	"time"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
-	"github.com/gogo/protobuf/proto"
 	"github.com/google/cel-go/cel"
 	"github.com/google/cel-go/checker/decls"
 	"github.com/google/cel-go/common/types"
@@ -33,6 +34,8 @@ import (
 	"github.com/google/cel-go/ext"
 	"github.com/google/cel-go/interpreter/functions"
 	exprpb "google.golang.org/genproto/googleapis/api/expr/v1alpha1"
+	"google.golang.org/protobuf/proto"
+	timestamp "google.golang.org/protobuf/types/known/timestamppb"
 )
 
 func init() {
@@ -42,7 +45,12 @@ func init() {
 // MatchExpression matches requests by evaluating a
 // [CEL](https://github.com/google/cel-spec) expression.
 // This enables complex logic to be expressed using a comfortable,
-// familiar syntax.
+// familiar syntax. Please refer to
+// [the standard definitions of CEL functions and operators](https://github.com/google/cel-spec/blob/master/doc/langdef.md#standard-definitions).
+//
+// This matcher's JSON interface is actually a string, not a struct.
+// The generated docs are not correct because this type has custom
+// marshaling logic.
 //
 // COMPATIBILITY NOTE: This module is still experimental and is not
 // subject to Caddy's compatibility guarantee.
@@ -87,7 +95,7 @@ func (m *MatchExpression) Provision(_ caddy.Context) error {
 	// create the CEL environment
 	env, err := cel.NewEnv(
 		cel.Declarations(
-			decls.NewIdent("request", httpRequestObjectType, nil),
+			decls.NewVar("request", httpRequestObjectType),
 			decls.NewFunction(placeholderFuncName,
 				decls.NewOverload(placeholderFuncName+"_httpRequest_string",
 					[]*exprpb.Type{httpRequestObjectType, decls.String},
@@ -192,6 +200,27 @@ func (cr celHTTPRequest) Equal(other ref.Val) ref.Val {
 func (celHTTPRequest) Type() ref.Type        { return httpRequestCELType }
 func (cr celHTTPRequest) Value() interface{} { return cr }
 
+var pkixNameCELType = types.NewTypeValue("pkix.Name", traits.ReceiverType)
+
+// celPkixName wraps an pkix.Name with
+// methods to satisfy the ref.Val interface.
+type celPkixName struct{ *pkix.Name }
+
+func (pn celPkixName) ConvertToNative(typeDesc reflect.Type) (interface{}, error) {
+	return pn.Name, nil
+}
+func (celPkixName) ConvertToType(typeVal ref.Type) ref.Val {
+	panic("not implemented")
+}
+func (pn celPkixName) Equal(other ref.Val) ref.Val {
+	if o, ok := other.Value().(string); ok {
+		return types.Bool(pn.Name.String() == o)
+	}
+	return types.ValOrErr(other, "%v is not comparable type", other)
+}
+func (celPkixName) Type() ref.Type        { return pkixNameCELType }
+func (pn celPkixName) Value() interface{} { return pn }
+
 // celTypeAdapter can adapt our custom types to a CEL value.
 type celTypeAdapter struct{}
 
@@ -199,6 +228,11 @@ func (celTypeAdapter) NativeToValue(value interface{}) ref.Val {
 	switch v := value.(type) {
 	case celHTTPRequest:
 		return v
+	case pkix.Name:
+		return celPkixName{&v}
+	case time.Time:
+		// TODO: eliminate direct protobuf dependency, sigh -- just wrap stdlib time.Time instead...
+		return types.Timestamp{Timestamp: &timestamp.Timestamp{Seconds: v.Unix(), Nanos: int32(v.Nanosecond())}}
 	case error:
 		types.NewErr(v.Error())
 	}

modules/caddyhttp/celmatcher_test.go

@@ -0,0 +1,124 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyhttp
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/pem"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func TestMatchExpressionProvision(t *testing.T) {
+	tests := []struct {
+		name       string
+		expression *MatchExpression
+		wantErr    bool
+	}{
+		{
+			name: "boolean matches succeed",
+			expression: &MatchExpression{
+				Expr: "{http.request.uri.query} != ''",
+			},
+			wantErr: false,
+		},
+		{
+			name: "reject expressions with non-boolean results",
+			expression: &MatchExpression{
+				Expr: "{http.request.uri.query}",
+			},
+			wantErr: true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if err := tt.expression.Provision(caddy.Context{}); (err != nil) != tt.wantErr {
+				t.Errorf("MatchExpression.Provision() error = %v, wantErr %v", err, tt.wantErr)
+			}
+		})
+	}
+}
+
+func TestMatchExpressionMatch(t *testing.T) {
+
+	clientCert := []byte(`-----BEGIN CERTIFICATE-----
+MIIB9jCCAV+gAwIBAgIBAjANBgkqhkiG9w0BAQsFADAYMRYwFAYDVQQDDA1DYWRk
+eSBUZXN0IENBMB4XDTE4MDcyNDIxMzUwNVoXDTI4MDcyMTIxMzUwNVowHTEbMBkG
+A1UEAwwSY2xpZW50LmxvY2FsZG9tYWluMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCB
+iQKBgQDFDEpzF0ew68teT3xDzcUxVFaTII+jXH1ftHXxxP4BEYBU4q90qzeKFneF
+z83I0nC0WAQ45ZwHfhLMYHFzHPdxr6+jkvKPASf0J2v2HDJuTM1bHBbik5Ls5eq+
+fVZDP8o/VHKSBKxNs8Goc2NTsr5b07QTIpkRStQK+RJALk4x9QIDAQABo0swSTAJ
+BgNVHRMEAjAAMAsGA1UdDwQEAwIHgDAaBgNVHREEEzARgglsb2NhbGhvc3SHBH8A
+AAEwEwYDVR0lBAwwCgYIKwYBBQUHAwIwDQYJKoZIhvcNAQELBQADgYEANSjz2Sk+
+eqp31wM9il1n+guTNyxJd+FzVAH+hCZE5K+tCgVDdVFUlDEHHbS/wqb2PSIoouLV
+3Q9fgDkiUod+uIK0IynzIKvw+Cjg+3nx6NQ0IM0zo8c7v398RzB4apbXKZyeeqUH
+9fNwfEi+OoXR6s+upSKobCmLGLGi9Na5s5g=
+-----END CERTIFICATE-----`)
+
+	tests := []struct {
+		name              string
+		expression        *MatchExpression
+		wantErr           bool
+		wantResult        bool
+		clientCertificate []byte
+	}{
+		{
+			name: "boolean matches succeed for placeholder http.request.tls.client.subject",
+			expression: &MatchExpression{
+				Expr: "{http.request.tls.client.subject} == 'CN=client.localdomain'",
+			},
+			clientCertificate: clientCert,
+			wantResult:        true,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if err := tt.expression.Provision(caddy.Context{}); (err != nil) != tt.wantErr {
+				t.Errorf("MatchExpression.Provision() error = %v, wantErr %v", err, tt.wantErr)
+			}
+
+			req := httptest.NewRequest("GET", "https://example.com/foo", nil)
+			repl := caddy.NewReplacer()
+			ctx := context.WithValue(req.Context(), caddy.ReplacerCtxKey, repl)
+			req = req.WithContext(ctx)
+			addHTTPVarsToReplacer(repl, req, httptest.NewRecorder())
+
+			if tt.clientCertificate != nil {
+				block, _ := pem.Decode(clientCert)
+				if block == nil {
+					t.Fatalf("failed to decode PEM certificate")
+				}
+
+				cert, err := x509.ParseCertificate(block.Bytes)
+				if err != nil {
+					t.Fatalf("failed to decode PEM certificate: %v", err)
+				}
+
+				req.TLS = &tls.ConnectionState{
+					PeerCertificates: []*x509.Certificate{cert},
+				}
+			}
+
+			if tt.expression.Match(req) != tt.wantResult {
+				t.Errorf("MatchExpression.Match() expected to return '%t', for expression : '%s'", tt.wantResult, tt.expression)
+			}
+
+		})
+	}
+}

modules/caddyhttp/fileserver/browse.go

@@ -106,6 +106,7 @@ func (fsrv *FileServer) browseApplyQueryParams(w http.ResponseWriter, r *http.Re
 	sortParam := r.URL.Query().Get("sort")
 	orderParam := r.URL.Query().Get("order")
 	limitParam := r.URL.Query().Get("limit")
+	offsetParam := r.URL.Query().Get("offset")
 
 	// first figure out what to sort by
 	switch sortParam {
@@ -130,7 +131,7 @@ func (fsrv *FileServer) browseApplyQueryParams(w http.ResponseWriter, r *http.Re
 	}
 
 	// finally, apply the sorting and limiting
-	listing.applySortAndLimit(sortParam, orderParam, limitParam)
+	listing.applySortAndLimit(sortParam, orderParam, limitParam, offsetParam)
 }
 
 func (fsrv *FileServer) browseWriteJSON(listing browseListing) (*bytes.Buffer, error) {

modules/caddyhttp/fileserver/browse_test.go

@@ -11,15 +11,15 @@ func BenchmarkBrowseWriteJSON(b *testing.B) {
 	fsrv := new(FileServer)
 	fsrv.Provision(caddy.Context{})
 	listing := browseListing{
-		Name:           "test",
-		Path:           "test",
-		CanGoUp:        false,
-		Items:          make([]fileInfo, 100),
-		NumDirs:        42,
-		NumFiles:       420,
-		Sort:           "",
-		Order:          "",
-		ItemsLimitedTo: 42,
+		Name:     "test",
+		Path:     "test",
+		CanGoUp:  false,
+		Items:    make([]fileInfo, 100),
+		NumDirs:  42,
+		NumFiles: 420,
+		Sort:     "",
+		Order:    "",
+		Limit:    42,
 	}
 	b.ResetTimer()
 
@@ -36,15 +36,15 @@ func BenchmarkBrowseWriteHTML(b *testing.B) {
 		template:     template.New("test"),
 	}
 	listing := browseListing{
-		Name:           "test",
-		Path:           "test",
-		CanGoUp:        false,
-		Items:          make([]fileInfo, 100),
-		NumDirs:        42,
-		NumFiles:       420,
-		Sort:           "",
-		Order:          "",
-		ItemsLimitedTo: 42,
+		Name:     "test",
+		Path:     "test",
+		CanGoUp:  false,
+		Items:    make([]fileInfo, 100),
+		NumDirs:  42,
+		NumFiles: 420,
+		Sort:     "",
+		Order:    "",
+		Limit:    42,
 	}
 	b.ResetTimer()
 

modules/caddyhttp/fileserver/browselisting.go

@@ -76,31 +76,34 @@ func (fsrv *FileServer) directoryListing(files []os.FileInfo, canGoUp bool, urlP
 
 type browseListing struct {
 	// The name of the directory (the last element of the path).
-	Name string
+	Name string `json:"name"`
 
 	// The full path of the request.
-	Path string
+	Path string `json:"path"`
 
 	// Whether the parent directory is browseable.
-	CanGoUp bool
+	CanGoUp bool `json:"can_go_up"`
 
 	// The items (files and folders) in the path.
-	Items []fileInfo
+	Items []fileInfo `json:"items,omitempty"`
 
-	// The number of directories in the listing.
-	NumDirs int
-
-	// The number of files (items that aren't directories) in the listing.
-	NumFiles int
-
-	// Sort column used
-	Sort string
-
-	// Sorting order
-	Order string
+	// If ≠0 then Items starting from that many elements.
+	Offset int `json:"offset,omitempty"`
 
 	// If ≠0 then Items have been limited to that many elements.
-	ItemsLimitedTo int
+	Limit int `json:"limit,omitempty"`
+
+	// The number of directories in the listing.
+	NumDirs int `json:"num_dirs"`
+
+	// The number of files (items that aren't directories) in the listing.
+	NumFiles int `json:"num_files"`
+
+	// Sort column used
+	Sort string `json:"sort,omitempty"`
+
+	// Sorting order
+	Order string `json:"order,omitempty"`
 }
 
 // Breadcrumbs returns l.Path where every element maps
@@ -131,7 +134,7 @@ func (l browseListing) Breadcrumbs() []crumb {
 	return result
 }
 
-func (l *browseListing) applySortAndLimit(sortParam, orderParam, limitParam string) {
+func (l *browseListing) applySortAndLimit(sortParam, orderParam, limitParam string, offsetParam string) {
 	l.Sort = sortParam
 	l.Order = orderParam
 
@@ -159,11 +162,20 @@ func (l *browseListing) applySortAndLimit(sortParam, orderParam, limitParam stri
 		}
 	}
 
+	if offsetParam != "" {
+		offset, _ := strconv.Atoi(offsetParam)
+		if offset > 0 && offset <= len(l.Items) {
+			l.Items = l.Items[offset:]
+			l.Offset = offset
+		}
+	}
+
 	if limitParam != "" {
 		limit, _ := strconv.Atoi(limitParam)
+
 		if limit > 0 && limit <= len(l.Items) {
 			l.Items = l.Items[:limit]
-			l.ItemsLimitedTo = limit
+			l.Limit = limit
 		}
 	}
 }

modules/caddyhttp/fileserver/browsetpl.go

@@ -283,8 +283,8 @@ footer {
 				<div id="summary">
 					<span class="meta-item"><b>{{.NumDirs}}</b> director{{if eq 1 .NumDirs}}y{{else}}ies{{end}}</span>
 					<span class="meta-item"><b>{{.NumFiles}}</b> file{{if ne 1 .NumFiles}}s{{end}}</span>
-					{{- if ne 0 .ItemsLimitedTo}}
-					<span class="meta-item">(of which only <b>{{.ItemsLimitedTo}}</b> are displayed)</span>
+					{{- if ne 0 .Limit}}
+					<span class="meta-item">(of which only <b>{{.Limit}}</b> are displayed)</span>
 					{{- end}}
 					<span class="meta-item"><input type="text" placeholder="filter" id="filter" onkeyup='filter()'></span>
 				</div>
@@ -296,37 +296,37 @@ footer {
 						<th></th>
 						<th>
 							{{- if and (eq .Sort "namedirfirst") (ne .Order "desc")}}
-							<a href="?sort=namedirfirst&order=desc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}" class="icon"><svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
+							<a href="?sort=namedirfirst&order=desc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}" class="icon"><svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
 							{{- else if and (eq .Sort "namedirfirst") (ne .Order "asc")}}
-							<a href="?sort=namedirfirst&order=asc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}" class="icon"><svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
+							<a href="?sort=namedirfirst&order=asc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}" class="icon"><svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
 							{{- else}}
-							<a href="?sort=namedirfirst&order=asc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}" class="icon sort"><svg class="top" width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg><svg class="bottom" width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
+							<a href="?sort=namedirfirst&order=asc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}" class="icon sort"><svg class="top" width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg><svg class="bottom" width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
 							{{- end}}
 							
 							{{- if and (eq .Sort "name") (ne .Order "desc")}}
-							<a href="?sort=name&order=desc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Name <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
+							<a href="?sort=name&order=desc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Name <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
 							{{- else if and (eq .Sort "name") (ne .Order "asc")}}
-							<a href="?sort=name&order=asc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Name <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
+							<a href="?sort=name&order=asc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Name <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
 							{{- else}}
-							<a href="?sort=name&order=asc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Name</a>
+							<a href="?sort=name&order=asc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Name</a>
 							{{- end}}
 						</th>
 						<th>
 							{{- if and (eq .Sort "size") (ne .Order "desc")}}
-							<a href="?sort=size&order=desc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Size <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
+							<a href="?sort=size&order=desc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Size <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
 							{{- else if and (eq .Sort "size") (ne .Order "asc")}}
-							<a href="?sort=size&order=asc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Size <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
+							<a href="?sort=size&order=asc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Size <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
 							{{- else}}
-							<a href="?sort=size&order=asc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Size</a>
+							<a href="?sort=size&order=asc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Size</a>
 							{{- end}}
 						</th>
 						<th class="hideable">
 							{{- if and (eq .Sort "time") (ne .Order "desc")}}
-							<a href="?sort=time&order=desc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Modified <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
+							<a href="?sort=time&order=desc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Modified <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#up-arrow"></use></svg></a>
 							{{- else if and (eq .Sort "time") (ne .Order "asc")}}
-							<a href="?sort=time&order=asc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Modified <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
+							<a href="?sort=time&order=asc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Modified <svg width="1em" height=".5em" version="1.1" viewBox="0 0 12.922194 6.0358899"><use xlink:href="#down-arrow"></use></svg></a>
 							{{- else}}
-							<a href="?sort=time&order=asc{{if ne 0 .ItemsLimitedTo}}&limit={{.ItemsLimitedTo}}{{end}}">Modified</a>
+							<a href="?sort=time&order=asc{{if ne 0 .Limit}}&limit={{.Limit}}{{end}}{{if ne 0 .Offset}}&offset={{.Offset}}{{end}}">Modified</a>
 							{{- end}}
 						</th>
 						<th class="hideable"></th>

modules/caddyhttp/fileserver/command.go

@@ -33,7 +33,7 @@ func init() {
 	caddycmd.RegisterCommand(caddycmd.Command{
 		Name:  "file-server",
 		Func:  cmdFileServer,
-		Usage: "[--domain <example.com>] [--root <path>] [--listen <addr>] [--browse]",
+		Usage: "[--domain <example.com>] [--root <path>] [--listen <addr>] [--browse] [--access-log]",
 		Short: "Spins up a production-ready file server",
 		Long: `
 A simple but production-ready file server. Useful for quick deployments,
@@ -55,17 +55,21 @@ respond with a file listing.`,
 			fs.String("listen", "", "The address to which to bind the listener")
 			fs.Bool("browse", false, "Enable directory browsing")
 			fs.Bool("templates", false, "Enable template rendering")
+			fs.Bool("access-log", false, "Enable the access log")
 			return fs
 		}(),
 	})
 }
 
 func cmdFileServer(fs caddycmd.Flags) (int, error) {
+	caddy.TrapSignals()
+
 	domain := fs.String("domain")
 	root := fs.String("root")
 	listen := fs.String("listen")
 	browse := fs.Bool("browse")
 	templates := fs.Bool("templates")
+	accessLog := fs.Bool("access-log")
 
 	var handlers []json.RawMessage
 
@@ -85,7 +89,7 @@ func cmdFileServer(fs caddycmd.Flags) (int, error) {
 
 	if domain != "" {
 		route.MatcherSetsRaw = []caddy.ModuleMap{
-			caddy.ModuleMap{
+			{
 				"host": caddyconfig.JSON(caddyhttp.MatchHost{domain}, nil),
 			},
 		}
@@ -105,6 +109,9 @@ func cmdFileServer(fs caddycmd.Flags) (int, error) {
 		}
 	}
 	server.Listen = []string{listen}
+	if accessLog {
+		server.Logs = &caddyhttp.ServerLogConfig{}
+	}
 
 	httpApp := caddyhttp.App{
 		Servers: map[string]*caddyhttp.Server{"static": server},

modules/caddyhttp/fileserver/matcher.go

@@ -19,6 +19,7 @@ import (
 	"net/http"
 	"os"
 	"path"
+	"path/filepath"
 	"strings"
 	"time"
 
@@ -34,7 +35,7 @@ func init() {
 // MatchFile is an HTTP request matcher that can match
 // requests based upon file existence.
 //
-// Upon matching, two new placeholders will be made
+// Upon matching, three new placeholders will be made
 // available:
 //
 // - `{http.matchers.file.relative}` The root-relative
@@ -42,6 +43,8 @@ func init() {
 // requests.
 // - `{http.matchers.file.absolute}` The absolute path
 // of the matched file.
+// - `{http.matchers.file.type}` Set to "directory" if
+// the matched file is a directory, "file" otherwise.
 type MatchFile struct {
 	// The root directory, used for creating absolute
 	// file paths, and required when working with
@@ -68,6 +71,17 @@ type MatchFile struct {
 	//
 	// Default is first_exist.
 	TryPolicy string `json:"try_policy,omitempty"`
+
+	// A list of delimiters to use to split the path in two
+	// when trying files. If empty, no splitting will
+	// occur, and the path will be tried as-is. For each
+	// split value, the left-hand side of the split,
+	// including the split value, will be the path tried.
+	// For example, the path `/remote.php/dav/` using the
+	// split value `.php` would try the file `/remote.php`.
+	// Each delimiter must appear at the end of a URI path
+	// component in order to be used as a split delimiter.
+	SplitPath []string `json:"split_path,omitempty"`
 }
 
 // CaddyModule returns the Caddy module information.
@@ -80,7 +94,7 @@ func (MatchFile) CaddyModule() caddy.ModuleInfo {
 
 // UnmarshalCaddyfile sets up the matcher from Caddyfile tokens. Syntax:
 //
-//     file {
+//     file <files...> {
 //         root <path>
 //         try_files <files...>
 //         try_policy first_exist|smallest_size|largest_size|most_recently_modified
@@ -88,6 +102,7 @@ func (MatchFile) CaddyModule() caddy.ModuleInfo {
 //
 func (m *MatchFile) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 	for d.Next() {
+		m.TryFiles = append(m.TryFiles, d.RemainingArgs()...)
 		for d.NextBlock(0) {
 			switch d.Val() {
 			case "root":
@@ -96,7 +111,7 @@ func (m *MatchFile) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 				}
 				m.Root = d.Val()
 			case "try_files":
-				m.TryFiles = d.RemainingArgs()
+				m.TryFiles = append(m.TryFiles, d.RemainingArgs()...)
 				if len(m.TryFiles) == 0 {
 					return d.ArgErr()
 				}
@@ -105,6 +120,13 @@ func (m *MatchFile) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 					return d.ArgErr()
 				}
 				m.TryPolicy = d.Val()
+			case "split_path":
+				m.SplitPath = d.RemainingArgs()
+				if len(m.SplitPath) == 0 {
+					return d.ArgErr()
+				}
+			default:
+				return d.Errf("unrecognized subdirective: %s", d.Val())
 			}
 		}
 	}
@@ -134,25 +156,18 @@ func (m MatchFile) Validate() error {
 }
 
 // Match returns true if r matches m. Returns true
-// if a file was matched. If so, two placeholders
+// if a file was matched. If so, three placeholders
 // will be available:
 //    - http.matchers.file.relative
 //    - http.matchers.file.absolute
+//    - http.matchers.file.type
 func (m MatchFile) Match(r *http.Request) bool {
-	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)
-	rel, abs, matched := m.selectFile(r)
-	if matched {
-		repl.Set("http.matchers.file.relative", rel)
-		repl.Set("http.matchers.file.absolute", abs)
-	}
-	return matched
+	return m.selectFile(r)
 }
 
 // selectFile chooses a file according to m.TryPolicy by appending
 // the paths in m.TryFiles to m.Root, with placeholder replacements.
-// It returns the root-relative path to the matched file, the full
-// or absolute path, and whether a match was made.
-func (m MatchFile) selectFile(r *http.Request) (rel, abs string, matched bool) {
+func (m MatchFile) selectFile(r *http.Request) (matched bool) {
 	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)
 
 	root := repl.ReplaceAll(m.Root, ".")
@@ -164,13 +179,35 @@ func (m MatchFile) selectFile(r *http.Request) (rel, abs string, matched bool) {
 		m.TryFiles = []string{r.URL.Path}
 	}
 
+	// common preparation of the file into parts
+	prepareFilePath := func(file string) (string, string) {
+		suffix := m.firstSplit(path.Clean(repl.ReplaceAll(file, "")))
+		if strings.HasSuffix(file, "/") {
+			suffix += "/"
+		}
+		fullpath := sanitizedPathJoin(root, suffix)
+		return suffix, fullpath
+	}
+
+	// sets up the placeholders for the matched file
+	setPlaceholders := func(info os.FileInfo, rel string, abs string) {
+		repl.Set("http.matchers.file.relative", rel)
+		repl.Set("http.matchers.file.absolute", abs)
+
+		fileType := "file"
+		if info.IsDir() {
+			fileType = "directory"
+		}
+		repl.Set("http.matchers.file.type", fileType)
+	}
+
 	switch m.TryPolicy {
 	case "", tryPolicyFirstExist:
 		for _, f := range m.TryFiles {
-			suffix := path.Clean(repl.ReplaceAll(f, ""))
-			fullpath := sanitizedPathJoin(root, suffix)
-			if strictFileExists(fullpath) {
-				return suffix, fullpath, true
+			suffix, fullpath := prepareFilePath(f)
+			if info, exists := strictFileExists(fullpath); exists {
+				setPlaceholders(info, suffix, fullpath)
+				return true
 			}
 		}
 
@@ -178,9 +215,9 @@ func (m MatchFile) selectFile(r *http.Request) (rel, abs string, matched bool) {
 		var largestSize int64
 		var largestFilename string
 		var largestSuffix string
+		var info os.FileInfo
 		for _, f := range m.TryFiles {
-			suffix := path.Clean(repl.ReplaceAll(f, ""))
-			fullpath := sanitizedPathJoin(root, suffix)
+			suffix, fullpath := prepareFilePath(f)
 			info, err := os.Stat(fullpath)
 			if err == nil && info.Size() > largestSize {
 				largestSize = info.Size()
@@ -188,15 +225,16 @@ func (m MatchFile) selectFile(r *http.Request) (rel, abs string, matched bool) {
 				largestSuffix = suffix
 			}
 		}
-		return largestSuffix, largestFilename, true
+		setPlaceholders(info, largestSuffix, largestFilename)
+		return true
 
 	case tryPolicySmallestSize:
 		var smallestSize int64
 		var smallestFilename string
 		var smallestSuffix string
+		var info os.FileInfo
 		for _, f := range m.TryFiles {
-			suffix := path.Clean(repl.ReplaceAll(f, ""))
-			fullpath := sanitizedPathJoin(root, suffix)
+			suffix, fullpath := prepareFilePath(f)
 			info, err := os.Stat(fullpath)
 			if err == nil && (smallestSize == 0 || info.Size() < smallestSize) {
 				smallestSize = info.Size()
@@ -204,15 +242,16 @@ func (m MatchFile) selectFile(r *http.Request) (rel, abs string, matched bool) {
 				smallestSuffix = suffix
 			}
 		}
-		return smallestSuffix, smallestFilename, true
+		setPlaceholders(info, smallestSuffix, smallestFilename)
+		return true
 
 	case tryPolicyMostRecentlyMod:
 		var recentDate time.Time
 		var recentFilename string
 		var recentSuffix string
+		var info os.FileInfo
 		for _, f := range m.TryFiles {
-			suffix := path.Clean(repl.ReplaceAll(f, ""))
-			fullpath := sanitizedPathJoin(root, suffix)
+			suffix, fullpath := prepareFilePath(f)
 			info, err := os.Stat(fullpath)
 			if err == nil &&
 				(recentDate.IsZero() || info.ModTime().After(recentDate)) {
@@ -221,7 +260,8 @@ func (m MatchFile) selectFile(r *http.Request) (rel, abs string, matched bool) {
 				recentSuffix = suffix
 			}
 		}
-		return recentSuffix, recentFilename, true
+		setPlaceholders(info, recentSuffix, recentFilename)
+		return true
 	}
 
 	return
@@ -233,7 +273,7 @@ func (m MatchFile) selectFile(r *http.Request) (rel, abs string, matched bool) {
 // the file must also be a directory; if it does
 // NOT end in a forward slash, the file must NOT
 // be a directory.
-func strictFileExists(file string) bool {
+func strictFileExists(file string) (os.FileInfo, bool) {
 	stat, err := os.Stat(file)
 	if err != nil {
 		// in reality, this can be any error
@@ -244,16 +284,48 @@ func strictFileExists(file string) bool {
 		// the file exists, so we just treat any
 		// error as if it does not exist; see
 		// https://stackoverflow.com/a/12518877/1048862
-		return false
+		return nil, false
 	}
-	if strings.HasSuffix(file, "/") {
+	if strings.HasSuffix(file, string(filepath.Separator)) {
 		// by convention, file paths ending
-		// in a slash must be a directory
-		return stat.IsDir()
+		// in a path separator must be a directory
+		return stat, stat.IsDir()
 	}
 	// by convention, file paths NOT ending
-	// in a slash must NOT be a directory
-	return !stat.IsDir()
+	// in a path separator must NOT be a directory
+	return stat, !stat.IsDir()
+}
+
+// firstSplit returns the first result where the path
+// can be split in two by a value in m.SplitPath. The
+// result is the first piece of the path that ends with
+// in the split value. Returns the path as-is if the
+// path cannot be split.
+func (m MatchFile) firstSplit(path string) string {
+	for _, split := range m.SplitPath {
+		if idx := indexFold(path, split); idx > -1 {
+			pos := idx + len(split)
+			// skip the split if it's not the final part of the filename
+			if pos != len(path) && !strings.HasPrefix(path[pos:], "/") {
+				continue
+			}
+			return path[:pos]
+		}
+	}
+	return path
+}
+
+// There is no strings.IndexFold() function like there is strings.EqualFold(),
+// but we can use strings.EqualFold() to build our own case-insensitive
+// substring search (as of Go 1.14).
+func indexFold(haystack, needle string) int {
+	nlen := len(needle)
+	for i := 0; i+nlen < len(haystack); i++ {
+		if strings.EqualFold(haystack[i:i+nlen], needle) {
+			return i
+		}
+	}
+	return -1
 }
 
 const (