caddyfile: Fix caddy fmt nesting not decrementing (#4157)

* caddyfile(formatter): fix nesting not decrementing

This is an extremely weird edge-case where if you had a environment variable {}
on one line, a comment on the next line, and the closing of the block on the
following line; the rest of the Caddyfile would be indented further than it
should've been.

ref; https://github.com/matthewpi/vscode-caddyfile-support/issues/13

* run gofmt

* fmt: better way of handling edge case

.github/CONTRIBUTING.md

@@ -0,0 +1,187 @@
+Contributing to Caddy
+=====================
+
+Welcome! Thank you for choosing to be a part of our community. Caddy wouldn't be great without your involvement!
+
+For starters, we invite you to join [the Caddy forum](https://caddy.community) where you can hang out with other Caddy users and developers.
+
+## Common Tasks
+
+- [Contributing code](#contributing-code)
+- [Writing a Caddy module](#writing-a-caddy-module)
+- [Asking or answering questions for help using Caddy](#getting-help-using-caddy)
+- [Reporting a bug](#reporting-bugs)
+- [Suggesting an enhancement or a new feature](#suggesting-features)
+- [Improving documentation](#improving-documentation)
+
+Other menu items:
+
+- [Values](#values)
+- [Coordinated Disclosure](#coordinated-disclosure)
+- [Thank You](#thank-you)
+
+
+### Contributing code
+
+You can have a huge impact on the project by helping with its code. To contribute code to Caddy, first submit or comment in an issue to discuss your contribution, then open a [pull request](https://github.com/caddyserver/caddy/pulls) (PR). If you're new to our community, that's okay: **we gladly welcome pull requests from anyone, regardless of your native language or coding experience.** You can get familiar with Caddy's code base by using [code search at Sourcegraph](https://sourcegraph.com/github.com/caddyserver/caddy).
+
+We hold contributions to a high standard for quality :bowtie:, so don't be surprised if we ask for revisions—even if it seems small or insignificant. Please don't take it personally. :blue_heart: If your change is on the right track, we can guide you to make it mergable.
+
+Here are some of the expectations we have of contributors:
+
+- **Open an issue to propose your change first.** This way we can avoid confusion, coordinate what everyone is working on, and ensure that any changes are in-line with the project's goals and the best interests of its users. We can also discuss the best possible implementation. If there's already an issue about it, comment on the existing issue to claim it. A lot of valuable time can be saved by discussing a proposal first.
+
+- **Keep pull requests small.** Smaller PRs are more likely to be merged because they are easier to review! We might ask you to break up large PRs into smaller ones. [An example of what we want to avoid.](https://twitter.com/iamdevloper/status/397664295875805184)
+
+- **Keep related commits together in a PR.** We do want pull requests to be small, but you should also keep multiple related commits in the same PR if they rely on each other.
+
+- **Write tests.** Tests are essential! Written properly, they ensure your change works, and that other changes in the future won't break your change. CI checks should pass.
+
+- **Benchmarks should be included for optimizations.** Optimizations sometimes make code harder to read or have changes that are less than obvious. They should be proven with benchmarks or profiling.
+
+- **[Squash](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html) insignificant commits.** Every commit should be significant. Commits which merely rewrite a comment or fix a typo can be combined into another commit that has more substance. Interactive rebase can do this, or a simpler way is `git reset --soft <diverging-commit>` then `git commit -s`.
+
+- **Own your contributions.** Caddy is a growing project, and it's much better when individual contributors help maintain their change after it is merged.
+
+- **Use comments properly.** We expect good godoc comments for package-level functions, types, and values. Comments are also useful whenever the purpose for a line of code is not obvious.
+
+- **Pull requests may still get closed.** The longer a PR stays open and idle, the more likely it is to be closed. If we haven't reviewed it in a while, it probably means the change is not a priority. Please don't take this personally, we're trying to balance a lot of tasks! If nobody else has commented or reacted to the PR, it likely means your change is useful only to you. The reality is this happens quite a bit. We don't tend to accept PRs that aren't generally helpful. For these reasons or others, the PR may get closed even after a review. We are not obligated to accept all proposed changes, even if the best justification we can give is something vague like, "It doesn't sit right." Sometimes PRs are just the wrong thing or the wrong time. Because it is open source, you can always build your own modified version of Caddy with a change you need, even if we reject it in the official repo.
+
+We often grant [collaborator status](#collaborator-instructions) to contributors who author one or more significant, high-quality PRs that are merged into the code base!
+
+
+#### HOW TO MAKE A PULL REQUEST TO CADDY
+
+Contributing to Go projects on GitHub is fun and easy. After you have proposed your change in an issue, we recommend the following workflow:
+
+1. [Fork this repo](https://github.com/caddyserver/caddy). This makes a copy of the code you can write to.
+
+2. If you don't already have this repo (caddyserver/caddy.git) repo on your computer, clone it down: `git clone https://github.com/caddyserver/caddy.git`
+
+3. Tell git that it can push the caddyserver/caddy.git repo to your fork by adding a remote: `git remote add myfork https://github.com/<your-username>/caddy.git`
+
+4. Make your changes in the caddyserver/caddy.git repo on your computer.
+
+5. Push your changes to your fork: `git push myfork`
+
+6. [Create a pull request](https://github.com/caddyserver/caddy/pull/new/master) to merge your changes into caddyserver/caddy @ master. (Click "compare across forks" and change the head fork.)
+
+This workflow is nice because you don't have to change import paths. You can get fancier by using different branches if you want.
+
+
+### Writing a Caddy module
+
+Caddy can do more with modules! Anyone can write one. Caddy modules are Go libraries that get compiled into Caddy, extending its feature set. They can add directives to the Caddyfile, add new configuration adapters, and even implement new server types (e.g. HTTP, DNS).
+
+[Learn how to write a module here](https://caddyserver.com/docs/extending-caddy). You should also share and discuss your module idea [on the forums](https://caddy.community) to have people test it out. We don't use the Caddy issue tracker for third-party modules.
+
+
+### Getting help using Caddy
+
+If you have a question about using Caddy, [ask on our forum](https://caddy.community)! There will be more people there who can help you than just the Caddy developers who follow our issue tracker. Issues are not the place for usage questions.
+
+Many people on the forums could benefit from your experience and expertise, too. Once you've been helped, consider giving back by answering other people's questions and participating in other discussions.
+
+
+### Reporting bugs
+
+Like every software, Caddy has its flaws. If you find one, [search the issues](https://github.com/caddyserver/caddy/issues) to see if it has already been reported. If not, [open a new issue](https://github.com/caddyserver/caddy/issues/new) and describe the bug, and somebody will look into it! (This repository is only for Caddy and its standard modules.)
+
+**You can help us fix bugs!** Speed up the patch by identifying the bug in the code. This can sometimes be done by adding `fmt.Println()` statements (or similar) in relevant code paths to narrow down where the problem may be. It's a good way to [introduce yourself to the Go language](https://tour.golang.org), too.
+
+We may reply with an issue template. Please follow the template so we have all the needed information. Unredacted&mdash;yes, actual values matter. We need to be able to repeat the bug using your instructions. Please simplify the issue as much as possible. If you don't, we might close your report. The burden is on you to make it easily reproducible and to convince us that it is actually a bug in Caddy. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you!
+
+We suggest reading [How to Report Bugs Effectively](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html).
+
+Please be kind. :smile: Remember that Caddy comes at no cost to you, and you're getting free support when we fix your issues. If we helped you, please consider helping someone else!
+
+#### Bug reporting expectations
+
+Maintainers---or more generally, developers---need three things to act on bugs:
+
+1. To agree or be convinced that it's a bug (reporter's responsibility).
+	- A bug is unintentional, undesired, or surprising behavior which violates documentation or relevant spec. It might be either a mistake in the documentation or a bug in the code.
+	- This project usually does not work around bugs in other software, systems, and dependencies; instead, we recommend that those bugs are fixed at their source. This sometimes means we close issues or reject PRs that attempt to fix, workaround, or hide bugs in other projects.
+
+2. To be able to understand what is happening (mostly reporter's responsibility).
+	- If the reporter can provide satisfactory instructions such that a developer can reproduce the bug, the developer will likely be able to understand the bug, write a test case, and implement a fix. This is the least amount of work for everyone and path to the fastest resolution.
+	- Otherwise, the burden is on the reporter to test possible solutions. This is less preferable because it loosens the feedback loop, slows down debugging efforts, obscures the true nature of the problem from the developers, and is unlikely to result in new test cases.
+
+3. A solution, or ideas toward a solution (mostly maintainer's responsibility).
+	- Sometimes the best solution is a documentation change.
+	- Usually the developers have the best domain knowledge for inventing a solution, but reporters may have ideas or preferences for how they would like the software to work.
+	- Security, correctness, and project goals/vision all take priority over a user's preferences.
+	- It's simply good business to yield a solution that satisfies the users, and it's even better business to leave them impressed.
+
+Thus, at the very least, the reporter is expected to:
+
+1. Convince the reader that it's a bug in Caddy (if it's not obvious).
+2. Reduce the problem down to the minimum specific steps required to reproduce it.
+
+The maintainer is usually able to do the rest; but of course the reporter may invest additional effort to speed up the process.
+
+
+
+### Suggesting features
+
+First, [search to see if your feature has already been requested](https://github.com/caddyserver/caddy/issues). If it has, you can add a :+1: reaction to vote for it. If your feature idea is new, open an issue to request the feature. Please describe your idea thoroughly so that we know how to implement it! Really vague requests may not be helpful or actionable and, without clarification, will have to be closed.
+
+While we really do value your requests and implement many of them, not all features are a good fit for Caddy. Most of those [make good modules](#writing-a-caddy-module), which can be made by anyone! But if a feature is not in the best interest of the Caddy project or its users in general, we may politely decline to implement it into Caddy core. Additionally, some features are bad ideas altogether (for either obvious or non-obvious reasons) which may be rejected. We'll try to explain why we reject a feature, but sometimes the best we can do is, "It's not a good fit for the project."
+
+
+### Improving documentation
+
+Caddy's documentation is available at [https://caddyserver.com/docs](https://caddyserver.com/docs) and its source is in the [website repo](https://github.com/caddyserver/website). If you would like to make a fix to the docs, please submit an issue there describing the change to make.
+
+Note that third-party module documentation is not hosted by the Caddy website, other than basic usage examples. They are managed by the individual module authors, and you will have to contact them to change their documentation.
+
+Our documentation is scoped to the Caddy project only: it is not for describing how other software or systems work, even if they relate to Caddy or web servers. That kind of content [can be found in our community wiki](https://caddy.community/c/wiki/13), however.
+
+## Collaborator Instructions
+
+Collaborators have push rights to the repository. We grant this permission after one or more successful, high-quality PRs are merged! We thank them for their help. The expectations we have of collaborators are:
+
+- **Help review pull requests.** Be meticulous, but also kind. We love our contributors, but we critique the contribution to make it better. Multiple, thorough reviews make for the best contributions! Here are some questions to consider:
+	- Can the change be made more elegant?
+	- Is this a maintenance burden?
+	- What assumptions does the code make?
+	- Is it well-tested?
+	- Is the change a good fit for the project?
+	- Does it actually fix the problem or is it creating a special case instead?
+	- Does the change incur any new dependencies? (Avoid these!)
+
+- **Answer issues.** If every collaborator helped out with issues, we could count the number of open issues on two hands. This means getting involved in the discussion, investigating the code, and yes, debugging it. It's fun. Really! :smile: Please, please help with open issues. Granted, some issues need to be done before others. And of course some are larger than others: you don't have to do it all yourself. Work with other collaborators as a team!
+
+- **Do not merge pull requests until they have been approved by one or two other collaborators.** If a project owner approves the PR, it can be merged (as long as the conversation has finished too).
+
+- **Prefer squashed commits over a messy merge.** If there are many little commits, please [squash the commits](https://stackoverflow.com/a/11732910/1048862) so we don't clutter the commit history.
+
+- **Don't accept new dependencies lightly.** Dependencies can make the world crash and burn, but they are sometimes necessary. Choose carefully. Extremely small dependencies (a few lines of code) can be inlined. The rest may not be needed. For those that are, Caddy uses [go modules](https://github.com/golang/go/wiki/Modules). All external dependencies must be installed as modules, and _Caddy must not export any types defined by those dependencies_. Check this diligently!
+
+- **Be extra careful in some areas of the code.** There are some critical areas in the Caddy code base that we review extra meticulously: the `caddyhttp` and `caddytls` packages especially.
+
+- **Make sure tests test the actual thing.** Double-check that the tests fail without the change, and pass with it. It's important that they assert what they're purported to assert.
+
+- **Recommended reading**
+	- [CodeReviewComments](https://github.com/golang/go/wiki/CodeReviewComments) for an idea of what we look for in good, clean Go code
+	- [Linus Torvalds describes a good commit message](https://gist.github.com/matthewhudson/1475276)
+	- [Best Practices for Maintainers](https://opensource.guide/best-practices/)
+	- [Shrinking Code Review](https://alexgaynor.net/2015/dec/29/shrinking-code-review/)
+
+
+
+## Values (WIP)
+
+- A person is always more important than code. People don't like being handled "efficiently". But we can still process issues and pull requests efficiently while being kind, patient, and considerate.
+
+- The ends justify the means, if the means are good. A good tree won't produce bad fruit. But if we cut corners or are hasty in our process, the end result will not be good.
+
+
+## Security Policy
+
+If you think you've found a security vulnerability, please refer to our [Security Policy](https://github.com/caddyserver/caddy/security/policy) document.
+
+
+## Thank you
+
+Thanks for your help! Caddy would not be what it is today without your contributions.

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: [mholt] # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.github/SECURITY.md

@@ -0,0 +1,27 @@
+# Security Policy
+
+The Caddy project would like to make sure that it stays on top of all practically-exploitable vulnerabilities.
+
+Some security problems are more the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please report only vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing or BGP hijacks a vulnerability in the Caddy web server).
+
+Please note that we consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems.
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x     | :white_check_mark: |
+| 1.x     | :x:                |
+| < 1.x   | :x:                |
+
+## Reporting a Vulnerability
+
+Please email Matt Holt (the author) directly: matt [at] lightcodelabs [dot com].
+
+We'll need enough information to verify the bug and make a patch. It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, resources permitting. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. Thank you for understanding.
+
+Please also understand that due to our nature as an open source project, we do not have a budget to award security bounties. We can only thank you.
+
+If your report is valid and a patch is released, we will not reveal your identity by default. If you wish to be credited, please give us the name to use and/or your GitHub username. If you don't provide this we can't credit you.
+
+Thanks for responsibly helping Caddy&mdash;and thousands of websites&mdash;be more secure!

.github/workflows/ci.yml

@@ -1,13 +1,13 @@
 # Used as inspiration: https://github.com/mvdan/github-actions-golang
 
-name: Cross-Platform
+name: Tests
 
 on:
   push:
-    branches: 
+    branches:
       - master
   pull_request:
-    branches: 
+    branches:
       - master
 
 jobs:
@@ -17,7 +17,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ ubuntu-latest, macos-latest, windows-latest ]
-        go-version: [ 1.14.x ]
+        go: [ '1.15', '1.16' ]
 
         # Set some variables per OS, usable via ${{ matrix.VAR }}
         # CADDY_BIN_PATH: the path to the compiled Caddy binary, for artifact publishing
@@ -39,9 +39,9 @@ jobs:
 
     steps:
     - name: Install Go
-      uses: actions/setup-go@v1
+      uses: actions/setup-go@v2
       with:
-        go-version: ${{ matrix.go-version }}
+        go-version: ${{ matrix.go }}
 
     - name: Checkout code
       uses: actions/checkout@v2
@@ -64,8 +64,18 @@ jobs:
         go env
         printf "\n\nSystem environment:\n\n"
         env
+        printf "Git version: $(git version)\n\n"
         # Calculate the short SHA1 hash of the git commit
         echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
+        echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+    - name: Cache the build cache
+      uses: actions/cache@v2
+      with:
+        path: ${{ steps.vars.outputs.go_cache }}
+        key: ${{ runner.os }}-${{ matrix.go }}-go-ci-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-${{ matrix.go }}-go-ci
 
     - name: Get dependencies
       run: |
@@ -77,12 +87,12 @@ jobs:
       env:
         CGO_ENABLED: 0
       run: |
-        go build -trimpath -a -ldflags="-w -s" -v
+        go build -trimpath -ldflags="-w -s" -v
 
     - name: Publish Build Artifact
       uses: actions/upload-artifact@v1
       with:
-        name: caddy_v2_${{ runner.os }}_${{ steps.vars.outputs.short_sha }}
+        name: caddy_${{ runner.os }}_go${{ matrix.go }}_${{ steps.vars.outputs.short_sha }}
         path: ${{ matrix.CADDY_BIN_PATH }}
 
     # Commented bits below were useful to allow the job to continue
@@ -111,16 +121,42 @@ jobs:
     #     echo "step_test ${{ steps.step_test.outputs.status }}\n"
     #     exit 1
 
-  # From https://github.com/reviewdog/action-golangci-lint
-  golangci-lint:
-    name: runner / golangci-lint
+  s390x-test:
+    name: test (s390x on IBM Z)
     runs-on: ubuntu-latest
+    if: github.event.pull_request.head.repo.full_name == github.repository
+    continue-on-error: true  # August 2020: s390x VM is down due to weather and power issues
     steps:
       - name: Checkout code into the Go module directory
         uses: actions/checkout@v2
+      - name: Run Tests
+        run: |
+          mkdir -p ~/.ssh && echo -e "${SSH_KEY//_/\\n}" > ~/.ssh/id_ecdsa && chmod og-rwx ~/.ssh/id_ecdsa
 
-      - name: Run golangci-lint
-        uses: reviewdog/action-golangci-lint@v1
-        # uses: docker://reviewdog/action-golangci-lint:v1 # pre-build docker image
+          # short sha is enough?
+          short_sha=$(git rev-parse --short HEAD)
+
+          # The environment is fresh, so there's no point in keeping accepting and adding the key.
+          rsync -arz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress --delete --exclude '.git' . caddy-ci@ci-s390x.caddyserver.com:/var/tmp/"$short_sha"
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -t caddy-ci@ci-s390x.caddyserver.com "cd /var/tmp/$short_sha; go version; go env; printf "\n\n";CGO_ENABLED=0 go test -v ./..."
+          test_result=$?
+
+          # There's no need leaving the files around
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null caddy-ci@ci-s390x.caddyserver.com "rm -rf /var/tmp/'$short_sha'"
+
+          echo "Test exit code: $test_result"
+          exit $test_result
+        env:
+          SSH_KEY: ${{ secrets.S390X_SSH_KEY }}
+
+  goreleaser-check:
+    runs-on: ubuntu-latest
+    steps:
+      - name: checkout
+        uses: actions/checkout@v2
+      - uses: goreleaser/goreleaser-action@v2
         with:
-          github_token: ${{ secrets.github_token }}
+          version: latest
+          args: check
+        env:
+          TAG: ${{ steps.vars.outputs.version_tag }}

.github/workflows/cross-build.yml

@@ -0,0 +1,60 @@
+name: Cross-Build
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  cross-build-test:
+    strategy:
+      fail-fast: false
+      matrix:
+        goos: ['android', 'linux', 'solaris', 'illumos', 'dragonfly', 'freebsd', 'openbsd', 'plan9', 'windows', 'darwin', 'netbsd']
+        go: [ '1.15', '1.16' ]
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: ${{ matrix.go }}
+
+      - name: Print Go version and environment
+        id: vars
+        run: |
+          printf "Using go at: $(which go)\n"
+          printf "Go version: $(go version)\n"
+          printf "\n\nGo environment:\n\n"
+          go env
+          printf "\n\nSystem environment:\n\n"
+          env
+          echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+      - name: Cache the build cache
+        uses: actions/cache@v2
+        with:
+          path: ${{ steps.vars.outputs.go_cache }}
+          key: cross-build-go${{ matrix.go }}-${{ matrix.goos }}-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            cross-build-go${{ matrix.go }}-${{ matrix.goos }}
+
+      - name: Checkout code into the Go module directory
+        uses: actions/checkout@v2
+
+      - name: Run Build
+        env:
+          CGO_ENABLED: 0
+          GOOS: ${{ matrix.goos }}
+        shell: bash
+        continue-on-error: true
+        working-directory: ./cmd/caddy
+        run: |
+          GOOS=$GOOS go build -trimpath -o caddy-"$GOOS"-amd64 2> /dev/null
+          if [ $? -ne 0 ]; then
+            echo "::warning ::$GOOS Build Failed"
+            exit 0
+          fi

.github/workflows/fuzzing.yml

@@ -1,75 +0,0 @@
-name: Fuzzing
-
-on:
-  # Daily midnight fuzzing
-  schedule:
-    - cron: '0 0 * * *'
-
-jobs:
-  fuzzing:
-    name: Fuzzing
-
-    strategy:
-      matrix:
-        os: [ ubuntu-latest ]
-        go-version: [ 1.14.x ]
-    runs-on: ${{ matrix.os }}
-
-    steps:
-    - name: Install Go
-      uses: actions/setup-go@v1
-      with:
-        go-version: ${{ matrix.go-version }}
-
-    - name: Checkout code
-      uses: actions/checkout@v2
-
-    - name: Download go-fuzz tools and the Fuzzit CLI, move Fuzzit CLI to GOBIN
-      # If we decide we need to prevent this from running on forks, we can use this line:
-      # if: github.repository == 'caddyserver/caddy'
-      run: |
-
-        go get -v github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build
-        wget -q -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v2.4.77/fuzzit_Linux_x86_64
-        chmod a+x fuzzit
-        mv fuzzit $(go env GOPATH)/bin
-        echo "::add-path::$(go env GOPATH)/bin"
-
-    - name: Generate fuzzers & submit them to Fuzzit
-      continue-on-error: true
-      env:
-        FUZZIT_API_KEY: ${{ secrets.FUZZIT_API_KEY }}
-        SYSTEM_PULLREQUEST_SOURCEBRANCH: ${{ github.ref }}
-        BUILD_SOURCEVERSION: ${{ github.sha }}
-      run: |
-        # debug
-        echo "PR Source Branch: $SYSTEM_PULLREQUEST_SOURCEBRANCH"
-        echo "Source version: $BUILD_SOURCEVERSION"
-
-        declare -A fuzzers_funcs=(\
-          ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="FuzzParseAddress" \
-          ["./caddyconfig/caddyfile/parse_fuzz.go"]="FuzzParseCaddyfile" \
-          ["./listeners_fuzz.go"]="FuzzParseNetworkAddress" \
-          ["./replacer_fuzz.go"]="FuzzReplacer" \
-        )
-
-        declare -A fuzzers_targets=(\
-          ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="parse-address" \
-          ["./caddyconfig/caddyfile/parse_fuzz.go"]="parse-caddyfile" \
-          ["./listeners_fuzz.go"]="parse-network-address" \
-          ["./replacer_fuzz.go"]="replacer" \
-        )
-
-        fuzz_type="fuzzing"
-
-        for f in $(find . -name \*_fuzz.go); do
-          FUZZER_DIRECTORY=$(dirname "$f")
-          
-          echo "go-fuzz-build func ${fuzzers_funcs[$f]} residing in $f"
-          
-          go-fuzz-build -func "${fuzzers_funcs[$f]}" -o "$FUZZER_DIRECTORY/${fuzzers_targets[$f]}.zip" "$FUZZER_DIRECTORY"
-          
-          fuzzit create job --engine go-fuzz caddyserver/"${fuzzers_targets[$f]}" "$FUZZER_DIRECTORY"/"${fuzzers_targets[$f]}.zip" --api-key "${FUZZIT_API_KEY}" --type "${fuzz_type}" --branch "${SYSTEM_PULLREQUEST_SOURCEBRANCH}" --revision "${BUILD_SOURCEVERSION}"
-          
-          echo "Completed $f"
-        done

.github/workflows/lint.yml

@@ -0,0 +1,23 @@
+name: Lint
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  # From https://github.com/golangci/golangci-lint-action
+  golangci:
+    name: lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@v2
+        with:
+          version: v1.31
+          # Optional: show only new issues if it's a pull request. The default value is `false`.
+          # only-new-issues: true

.github/workflows/release.yml

@@ -11,22 +11,32 @@ jobs:
     strategy:
       matrix:
         os: [ ubuntu-latest ]
-        go-version: [ 1.14.x ]
+        go: [ '1.16' ]
     runs-on: ${{ matrix.os }}
 
     steps:
     - name: Install Go
-      uses: actions/setup-go@v1
+      uses: actions/setup-go@v2
       with:
-        go-version: ${{ matrix.go-version }}
+        go-version: ${{ matrix.go }}
 
     - name: Checkout code
       uses: actions/checkout@v2
+      with:
+        fetch-depth: 0
 
-    # So GoReleaser can generate the changelog properly
-    - name: Unshallowify the repo clone
-      run: git fetch --prune --unshallow
+    # Force fetch upstream tags -- because 65 minutes
+    # tl;dr: actions/checkout@v2 runs this line:
+    #   git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +ebc278ec98bb24f2852b61fde2a9bf2e3d83818b:refs/tags/
+    # which makes its own local lightweight tag, losing all the annotations in the process. Our earlier script ran:
+    #   git fetch --prune --unshallow
+    # which doesn't overwrite that tag because that would be destructive.
+    # Credit to @francislavoie for the investigation.
+    # https://github.com/actions/checkout/issues/290#issuecomment-680260080
+    - name: Force fetch upstream tags
+      run: git fetch --tags --force
 
+    # https://github.community/t5/GitHub-Actions/How-to-get-just-the-tag-name/m-p/32167/highlight/true#M1027
     - name: Print Go version and environment
       id: vars
       run: |
@@ -36,18 +46,112 @@ jobs:
         go env
         printf "\n\nSystem environment:\n\n"
         env
-    
-    # https://github.community/t5/GitHub-Actions/How-to-get-just-the-tag-name/m-p/32167/highlight/true#M1027
-    - name: Get the version
-      id: get_version
-      run: echo "::set-output name=version_tag::${GITHUB_REF/refs\/tags\//}"
+        echo "::set-output name=version_tag::${GITHUB_REF/refs\/tags\//}"
+        echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
+        echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+        # Add "pip install" CLI tools to PATH
+        echo ~/.local/bin >> $GITHUB_PATH
+
+        # Parse semver
+        TAG=${GITHUB_REF/refs\/tags\//}
+        SEMVER_RE='[^0-9]*\([0-9]*\)[.]\([0-9]*\)[.]\([0-9]*\)\([0-9A-Za-z\.-]*\)'
+        TAG_MAJOR=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\1#"`
+        TAG_MINOR=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\2#"`
+        TAG_PATCH=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\3#"`
+        TAG_SPECIAL=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\4#"`
+        echo "::set-output name=tag_major::${TAG_MAJOR}"
+        echo "::set-output name=tag_minor::${TAG_MINOR}"
+        echo "::set-output name=tag_patch::${TAG_PATCH}"
+        echo "::set-output name=tag_special::${TAG_SPECIAL}"
+
+    # Cloudsmith CLI tooling for pushing releases
+    # See https://help.cloudsmith.io/docs/cli
+    - name: Install Cloudsmith CLI
+      run: pip install --upgrade cloudsmith-cli
+
+    - name: Validate commits and tag signatures
+      run: |
+        
+        # Import Matt Holt's key
+        curl 'https://github.com/mholt.gpg' | gpg --import
+
+        echo "Verifying the tag: ${{ steps.vars.outputs.version_tag }}"
+        # tags are only accepted if signed by Matt's key
+        git verify-tag "${{ steps.vars.outputs.version_tag }}" || exit 1
+
+    - name: Cache the build cache
+      uses: actions/cache@v2
+      with:
+        path: ${{ steps.vars.outputs.go_cache }}
+        key: ${{ runner.os }}-go${{ matrix.go }}-release-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-go${{ matrix.go }}-release
 
     # GoReleaser will take care of publishing those artifacts into the release
     - name: Run GoReleaser
-      uses: goreleaser/goreleaser-action@v1
+      uses: goreleaser/goreleaser-action@v2
       with:
         version: latest
         args: release --rm-dist
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        TAG: ${{ steps.get_version.outputs.version_tag }}
+        TAG: ${{ steps.vars.outputs.version_tag }}
+
+    # Only publish on non-special tags (e.g. non-beta)
+    # We will continue to push to Gemfury for the forseeable future, although
+    # Cloudsmith is probably better, to not break things for existing users of Gemfury.
+    # See https://gemfury.com/caddy/deb:caddy
+    - name: Publish .deb to Gemfury
+      if: ${{ steps.vars.outputs.tag_special == '' }}
+      env:
+        GEMFURY_PUSH_TOKEN: ${{ secrets.GEMFURY_PUSH_TOKEN }}
+      run: |
+        for filename in dist/*.deb; do
+          # armv6 and armv7 are both "armhf" so we can skip the duplicate
+          if [[ "$filename" == *"armv6"* ]]; then
+            echo "Skipping $filename"
+            continue
+          fi
+
+          curl -F package=@"$filename" https://${GEMFURY_PUSH_TOKEN}:@push.fury.io/caddy/
+        done
+
+    # Publish only special tags (unstable/beta/rc) to the "testing" repo
+    # See https://cloudsmith.io/~caddy/repos/testing/
+    - name: Publish .deb to Cloudsmith (special tags)
+      if: ${{ steps.vars.outputs.tag_special != '' }}
+      env:
+        CLOUDSMITH_API_KEY: ${{ secrets.CLOUDSMITH_API_KEY }}
+      run: |
+        for filename in dist/*.deb; do
+          # armv6 and armv7 are both "armhf" so we can skip the duplicate
+          if [[ "$filename" == *"armv6"* ]]; then
+            echo "Skipping $filename"
+            continue
+          fi
+
+          echo "Pushing $filename to 'testing'"
+          cloudsmith push deb caddy/testing/any-distro/any-version $filename
+        done
+
+    # Publish stable tags to Cloudsmith to both repos, "stable" and "testing"
+    # See https://cloudsmith.io/~caddy/repos/stable/
+    - name: Publish .deb to Cloudsmith (stable tags)
+      if: ${{ steps.vars.outputs.tag_special == '' }}
+      env:
+        CLOUDSMITH_API_KEY: ${{ secrets.CLOUDSMITH_API_KEY }}
+      run: |
+        for filename in dist/*.deb; do
+          # armv6 and armv7 are both "armhf" so we can skip the duplicate
+          if [[ "$filename" == *"armv6"* ]]; then
+            echo "Skipping $filename"
+            continue
+          fi
+
+          echo "Pushing $filename to 'stable'"
+          cloudsmith push deb caddy/stable/any-distro/any-version $filename
+
+          echo "Pushing $filename to 'testing'"
+          cloudsmith push deb caddy/testing/any-distro/any-version $filename
+        done

.github/workflows/release_published.yml

@@ -0,0 +1,34 @@
+name: Release Published
+
+# Event payload: https://developer.github.com/webhooks/event-payloads/#release
+on:
+  release:
+    types: [published]
+
+jobs:
+  release:
+    name: Release Published
+    strategy:
+      matrix:
+        os: [ ubuntu-latest ]
+    runs-on: ${{ matrix.os }}
+
+    steps:
+
+    # See https://github.com/peter-evans/repository-dispatch
+    - name: Trigger event on caddyserver/dist
+      uses: peter-evans/repository-dispatch@v1
+      with:
+        token: ${{ secrets.REPO_DISPATCH_TOKEN }}
+        repository: caddyserver/dist
+        event-type: release-tagged
+        client-payload: '{"tag": "${{ github.event.release.tag_name }}"}'
+
+    - name: Trigger event on caddyserver/caddy-docker
+      uses: peter-evans/repository-dispatch@v1
+      with:
+        token: ${{ secrets.REPO_DISPATCH_TOKEN }}
+        repository: caddyserver/caddy-docker
+        event-type: release-tagged
+        client-payload: '{"tag": "${{ github.event.release.tag_name }}"}'
+

.gitignore

@@ -7,7 +7,7 @@ Caddyfile
 *.prof
 *.test
 
-# build artifacts
+# build artifacts and helpers
 cmd/caddy/caddy
 cmd/caddy/caddy.exe
 
@@ -20,4 +20,8 @@ vendor
 # goreleaser artifacts
 dist
 caddy-build
-caddy-dist
+caddy-dist
+
+# IDE files
+.idea/
+.vscode/

.golangci.yml

@@ -1,19 +1,68 @@
 linters-settings:
   errcheck:
-    ignore: fmt:.*,io/ioutil:^Read.*,github.com/caddyserver/caddy/v2/caddyconfig:RegisterAdapter,github.com/caddyserver/caddy/v2:RegisterModule
+    ignore: fmt:.*,io/ioutil:^Read.*,go.uber.org/zap/zapcore:^Add.*
     ignoretests: true
-  misspell:
-    locale: US
 
 linters:
+  disable-all: true
   enable:
     - bodyclose
+    - deadcode
     - errcheck
     - gofmt
     - goimports
     - gosec
+    - gosimple
+    - govet
     - ineffassign
     - misspell
+    - prealloc
+    - staticcheck
+    - structcheck
+    - typecheck
+    - unconvert
+    - unused
+    - varcheck
+  # these are implicitly disabled:
+  # - asciicheck
+  # - depguard
+  # - dogsled
+  # - dupl
+  # - exhaustive
+  # - exportloopref
+  # - funlen
+  # - gci
+  # - gochecknoglobals
+  # - gochecknoinits
+  # - gocognit
+  # - goconst
+  # - gocritic
+  # - gocyclo
+  # - godot
+  # - godox
+  # - goerr113
+  # - gofumpt
+  # - goheader
+  # - golint
+  # - gomnd
+  # - gomodguard
+  # - goprintffuncname
+  # - interfacer
+  # - lll
+  # - maligned
+  # - nakedret
+  # - nestif
+  # - nlreturn
+  # - noctx
+  # - nolintlint
+  # - rowserrcheck
+  # - scopelint
+  # - sqlclosecheck
+  # - stylecheck
+  # - testpackage
+  # - unparam
+  # - whitespace
+  # - wsl
 
 run:
   # default concurrency is a available CPU number.

.goreleaser.yml

@@ -1,14 +1,22 @@
 before:
   hooks:
+    # The build is done in this particular way to build Caddy in a designated directory named in .gitignore.
+    # This is so we can run goreleaser on tag without Git complaining of being dirty. The main.go in cmd/caddy directory 
+    # cannot be built within that directory due to changes necessary for the build causing Git to be dirty, which
+    # subsequently causes gorleaser to refuse running.
     - mkdir -p caddy-build
     - cp cmd/caddy/main.go caddy-build/main.go
     - cp ./go.mod caddy-build/go.mod
-    - sed -i.bkp s/github.com\/caddyserver\/caddy\/v2/caddy/g ./caddy-build/go.mod
+    - sed -i.bkp 's|github.com/caddyserver/caddy/v2|caddy|g' ./caddy-build/go.mod
     # GoReleaser doesn't seem to offer {{.Tag}} at this stage, so we have to embed it into the env
     # so we run: TAG=$(git describe --abbrev=0) goreleaser release --rm-dist --skip-publish --skip-validate
     - go mod edit -require=github.com/caddyserver/caddy/v2@{{.Env.TAG}} ./caddy-build/go.mod
+    # as of Go 1.16, `go` commands no longer automatically change go.{mod,sum}. We now have to explicitly
+    # run `go mod tidy`. The `/bin/sh -c '...'` is because goreleaser can't find cd in PATH without shell invocation.
+    - /bin/sh -c 'cd ./caddy-build && go mod tidy'
     - git clone --depth 1 https://github.com/caddyserver/dist caddy-dist
     - go mod download
+
 builds:
 - env:
   - CGO_ENABLED=0
@@ -23,45 +31,99 @@ builds:
   - freebsd
   goarch:
   - amd64
-  - 386
   - arm
   - arm64
+  - s390x
+  - ppc64le
   goarm:
+  - 5
   - 6
   - 7
   ignore:
-    - goos: darwin
-      goarch: 386
     - goos: darwin
       goarch: arm
+    - goos: darwin
+      goarch: ppc64le
+    - goos: darwin
+      goarch: s390x
+    - goos: windows
+      goarch: ppc64le
+    - goos: windows
+      goarch: s390x
+    - goos: freebsd
+      goarch: ppc64le
+    - goos: freebsd
+      goarch: s390x
+    - goos: freebsd
+      goarch: arm
+      goarm: 5
   flags:
   - -trimpath
   ldflags:
   - -s -w
+
 archives:
   - format_overrides:
       - goos: windows
         format: zip
     replacements:
-      darwin: macOS
-      linux: Linux
-      windows: Windows
-      386: i386
-      amd64: x86_64
+      darwin: mac
 checksum:
   algorithm: sha512
+
+nfpms:
+  - id: default
+    package_name: caddy
+
+    vendor: Light Code Labs
+    homepage: https://caddyserver.com
+    maintainer: Matthew Holt <mholt@users.noreply.github.com>
+    description: |
+      Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go
+    license: Apache 2.0
+
+    formats:
+      - deb
+      # - rpm
+
+    bindir: /usr/bin
+    contents:
+      - src: ./caddy-dist/init/caddy.service
+        dst: /lib/systemd/system/caddy.service
+        
+      - src: ./caddy-dist/init/caddy-api.service
+        dst: /lib/systemd/system/caddy-api.service
+      
+      - src: ./caddy-dist/welcome/index.html
+        dst: /usr/share/caddy/index.html
+      
+      - src: ./caddy-dist/scripts/completions/bash-completion
+        dst: /etc/bash_completion.d/caddy
+    
+      - src: ./caddy-dist/config/Caddyfile
+        dst: /etc/caddy/Caddyfile
+        type: config
+
+    scripts:
+      postinstall: ./caddy-dist/scripts/postinstall.sh
+      preremove: ./caddy-dist/scripts/preremove.sh
+      postremove: ./caddy-dist/scripts/postremove.sh
+
+
 release:
   github:
     owner: caddyserver
     name: caddy
   draft: true
   prerelease: auto
+
 changelog:
   sort: asc
   filters:
     exclude:
-    - '^ci:'
-    - '^docs:'
-    - '^test:'
     - '^chore:'
+    - '^ci:'
+    - '^docs?:'
+    - '^readme:'
+    - '^tests?:'
     - '^\w+\s+' # a hack to remove commit messages without colons thus don't correspond to a package

README.md

@@ -1,21 +1,25 @@
 <p align="center">
 	<a href="https://caddyserver.com"><img src="https://user-images.githubusercontent.com/1128849/36338535-05fb646a-136f-11e8-987b-e6901e717d5a.png" alt="Caddy" width="450"></a>
+	<br>
+	<h3 align="center">a <a href="https://zerossl.com"><img src="https://caddyserver.com/resources/images/zerossl-logo.svg" height="28" valign="middle"></a> project</h3>
 </p>
+<hr>
 <h3 align="center">Every site on HTTPS</h3>
 <p align="center">Caddy is an extensible server platform that uses TLS by default.</p>
 <p align="center">
 	<a href="https://github.com/caddyserver/caddy/actions?query=workflow%3ACross-Platform"><img src="https://github.com/caddyserver/caddy/workflows/Cross-Platform/badge.svg"></a>
-	<a href="https://pkg.go.dev/github.com/caddyserver/caddy/v2"><img src="https://img.shields.io/badge/godoc-reference-blue.svg"></a>
-	<a href="https://app.fuzzit.dev/orgs/caddyserver-gh/dashboard"><img src="https://app.fuzzit.dev/badge?org_id=caddyserver-gh"></a>
+	<a href="https://pkg.go.dev/github.com/caddyserver/caddy/v2"><img src="https://img.shields.io/badge/godoc-reference-%23007d9c.svg"></a>
 	<br>
 	<a href="https://twitter.com/caddyserver" title="@caddyserver on Twitter"><img src="https://img.shields.io/badge/twitter-@caddyserver-55acee.svg" alt="@caddyserver on Twitter"></a>
 	<a href="https://caddy.community" title="Caddy Forum"><img src="https://img.shields.io/badge/community-forum-ff69b4.svg" alt="Caddy Forum"></a>
+	<br>
 	<a href="https://sourcegraph.com/github.com/caddyserver/caddy?badge" title="Caddy on Sourcegraph"><img src="https://sourcegraph.com/github.com/caddyserver/caddy/-/badge.svg" alt="Caddy on Sourcegraph"></a>
+	<a href="https://cloudsmith.io/~caddy/repos/"><img src="https://img.shields.io/badge/OSS%20hosting%20by-cloudsmith-blue?logo=cloudsmith" alt="Cloudsmith"></a>
 </p>
 <p align="center">
-	<a href="https://github.com/caddyserver/caddy/releases">Download</a> ·
+	<a href="https://github.com/caddyserver/caddy/releases">Releases</a> ·
 	<a href="https://caddyserver.com/docs/">Documentation</a> ·
-	<a href="https://caddy.community">Community</a>
+	<a href="https://caddy.community">Get Help</a>
 </p>
 
 
@@ -23,10 +27,11 @@
 ### Menu
 
 - [Features](#features)
+- [Install](#install)
 - [Build from source](#build-from-source)
 	- [For development](#for-development)
 	- [With version information and/or plugins](#with-version-information-andor-plugins)
-- [Getting started](#getting-started)
+- [Quick start](#quick-start)
 - [Overview](#overview)
 - [Full documentation](#full-documentation)
 - [Getting help](#getting-help)
@@ -39,49 +44,71 @@
 </p>
 
 
-## Features
+## [Features](https://caddyserver.com/v2)
 
 - **Easy configuration** with the [Caddyfile](https://caddyserver.com/docs/caddyfile)
 - **Powerful configuration** with its [native JSON config](https://caddyserver.com/docs/json/)
-- **Dynamic configuration** with the [JSON API](https://caddyserver.com/api)
+- **Dynamic configuration** with the [JSON API](https://caddyserver.com/docs/api)
 - [**Config adapters**](https://caddyserver.com/docs/config-adapters) if you don't like JSON
 - **Automatic HTTPS** by default
-	- [Let's Encrypt](https://letsencrypt.org) for public sites
+	- [ZeroSSL](https://zerossl.com) and [Let's Encrypt](https://letsencrypt.org) for public names
 	- Fully-managed local CA for internal names & IPs
 	- Can coordinate with other Caddy instances in a cluster
+	- Multi-issuer fallback
 - **Stays up when other servers go down** due to TLS/OCSP/certificate-related issues
+- **Production-ready** after serving trillions of requests and managing millions of TLS certificates
+- **Scales to tens of thousands of sites** ... and probably more
 - **HTTP/1.1, HTTP/2, and experimental HTTP/3** support
-- **Highly extensible** [modular architecture](https://caddyserver.com/docs/extending-caddy) lets Caddy do anything without bloat
+- **Highly extensible** [modular architecture](https://caddyserver.com/docs/architecture) lets Caddy do anything without bloat
 - **Runs anywhere** with **no external dependencies** (not even libc)
 - Written in Go, a language with higher **memory safety guarantees** than other servers
 - Actually **fun to use**
-- So, so much more to discover
+- So, so much more to [discover](https://caddyserver.com/v2)
 
+## Install
 
+The simplest, cross-platform way is to download from [GitHub Releases](https://github.com/caddyserver/caddy/releases) and place the executable file in your PATH.
+
+For other install options, see https://caddyserver.com/docs/install.
 
 ## Build from source
 
 Requirements:
 
-- [Go 1.14 or newer](https://golang.org/dl/)
-- Do NOT disable [Go modules](https://github.com/golang/go/wiki/Modules) (`export GO111MODULE=on`)
+- [Go 1.15 or newer](https://golang.org/dl/)
 
 ### For development
  
+_**Note:** These steps [will not embed proper version information](https://github.com/golang/go/issues/29228). For that, please follow the instructions in the next section._
+
 ```bash
 $ git clone "https://github.com/caddyserver/caddy.git"
 $ cd caddy/cmd/caddy/
 $ go build
 ```
 
-_**Note:** These steps [will not embed proper version information](https://github.com/golang/go/issues/29228). For that, please follow the instructions below._
+When you run Caddy, it may try to bind to low ports unless otherwise specified in your config. If your OS requires elevated privileges for this, you will need to give your new binary permission to do so. On Linux, this can be done easily with: `sudo setcap cap_net_bind_service=+ep ./caddy`
+
+If you prefer to use `go run` which only creates temporary binaries, you can still do this with the included `setcap.sh` like so:
+
+```bash
+$ go run -exec ./setcap.sh main.go
+```
+
+If you don't want to type your password for `setcap`, use `sudo visudo` to edit your sudoers file and allow your user account to run that command without a password, for example:
+
+```
+username ALL=(ALL:ALL) NOPASSWD: /usr/sbin/setcap
+```
+
+replacing `username` with your actual username. Please be careful and only do this if you know what you are doing! We are only qualified to document how to use Caddy, not Go tooling or your computer, and we are providing these instructions for convenience only; please learn how to use your own computer at your own risk and make any needful adjustments.
 
 ### With version information and/or plugins
 
-Using [our builder tool](https://github.com/caddyserver/xcaddy)...
+Using [our builder tool, `xcaddy`](https://github.com/caddyserver/xcaddy)...
 
 ```
-$ xcaddy --version CADDY_VERSION
+$ xcaddy build
 ```
 
 ...the following steps are automated:
@@ -90,8 +117,9 @@ $ xcaddy --version CADDY_VERSION
 2. Change into it: `cd caddy`
 3. Copy [Caddy's main.go](https://github.com/caddyserver/caddy/blob/master/cmd/caddy/main.go) into the empty folder. Add imports for any custom plugins you want to add.
 4. Initialize a Go module: `go mod init caddy`
-5. Pin Caddy version: `go get github.com/caddyserver/caddy/v2@TAG` replacing `TAG` with a git tag or commit. You can also pin any plugin versions similarly.
-6. Compile: `go build`
+5. (Optional) Pin Caddy version: `go get github.com/caddyserver/caddy/v2@version` replacing `version` with a git tag, commit, or branch name.
+6. (Optional) Add plugins by adding their import: `_ "import/path/here"`
+7. Compile: `go build`
 
 
 
@@ -100,7 +128,7 @@ $ xcaddy --version CADDY_VERSION
 
 The [Caddy website](https://caddyserver.com/docs/) has documentation that includes tutorials, quick-start guides, reference, and more.
 
-**We recommend that all users do our [Getting Started](https://caddyserver.com/docs/getting-started) guide to become familiar with using Caddy.**
+**We recommend that all users -- regardless of experience level -- do our [Getting Started](https://caddyserver.com/docs/getting-started) guide to become familiar with using Caddy.**
 
 If you've only got a minute, [the website has several quick-start tutorials](https://caddyserver.com/docs/quick-starts) to choose from! However, after finishing a quick-start tutorial, please read more documentation to understand how the software works. 🙂
 
@@ -119,7 +147,7 @@ The primary way to configure Caddy is through [its API](https://caddyserver.com/
 
 Caddy exposes an unprecedented level of control compared to any web server in existence. In Caddy, you are usually setting the actual values of the initialized types in memory that power everything from your HTTP handlers and TLS handshakes to your storage medium. Caddy is also ridiculously extensible, with a powerful plugin system that makes vast improvements over other web servers.
 
-To wield the power of this design, you need to know how the config document is structured. Please see the [our documentation site](https://caddyserver.com/docs/) for details about [Caddy's config structure](https://caddyserver.com/docs/json/).
+To wield the power of this design, you need to know how the config document is structured. Please see [our documentation site](https://caddyserver.com/docs/) for details about [Caddy's config structure](https://caddyserver.com/docs/json/).
 
 Nearly all of Caddy's configuration is contained in a single config document, rather than being scattered across CLI flags and env variables and a configuration file as with other web servers. This makes managing your server config more straightforward and reduces hidden variables/factors.
 
@@ -138,15 +166,21 @@ The docs are also open source. You can contribute to them here: https://github.c
 
 - We **strongly recommend** that all professionals or companies using Caddy get a support contract through [Ardan Labs](https://www.ardanlabs.com/my/contact-us?dd=caddy) before help is needed.
 
+- A [sponsorship](https://github.com/sponsors/mholt) goes a long way! If Caddy is benefitting your company, please consider a sponsorship! This not only helps fund full-time work to ensure the longevity of the project, it's also a great look for your company to your customers and potential customers!
+
 - Individuals can exchange help for free on our community forum at https://caddy.community. Remember that people give help out of their spare time and good will. The best way to get help is to give it first!
 
-Please use our [issue tracker](/caddyserver/caddy/issues) only for bug reports and feature requests, i.e. actionable development items (support questions will usually be referred to the forums).
+Please use our [issue tracker](https://github.com/caddyserver/caddy/issues) only for bug reports and feature requests, i.e. actionable development items (support questions will usually be referred to the forums).
 
 
 
 ## About
 
-**The name "Caddy" is trademarked.** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". Caddy is a registered trademark of Light Code Labs, LLC.
+**The name "Caddy" is trademarked.** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". Caddy is a registered trademark of Stack Holdings GmbH.
 
 - _Project on Twitter: [@caddyserver](https://twitter.com/caddyserver)_
 - _Author on Twitter: [@mholt6](https://twitter.com/mholt6)_
+
+Caddy is a project of [ZeroSSL](https://zerossl.com), a Stack Holdings company.
+
+Debian package repository hosting is graciously provided by [Cloudsmith](https://cloudsmith.com). Cloudsmith is the only fully hosted, cloud-native, universal package management solution, that enables your organization to create, store and share packages in any format, to any place, with total confidence.

admin.go

@@ -17,10 +17,17 @@ package caddy
 import (
 	"bytes"
 	"context"
+	"crypto"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/base64"
 	"encoding/json"
+	"errors"
 	"expvar"
 	"fmt"
 	"io"
+	"io/ioutil"
+	"net"
 	"net/http"
 	"net/http/pprof"
 	"net/url"
@@ -32,11 +39,12 @@ import (
 	"sync"
 	"time"
 
+	"github.com/caddyserver/caddy/v2/notify"
+	"github.com/caddyserver/certmagic"
+	"github.com/prometheus/client_golang/prometheus"
 	"go.uber.org/zap"
 )
 
-// TODO: is there a way to make the admin endpoint so that it can be plugged into the HTTP app? see issue #2833
-
 // AdminConfig configures Caddy's API endpoint, which is used
 // to manage Caddy while it is running.
 type AdminConfig struct {
@@ -54,82 +62,180 @@ type AdminConfig struct {
 	// If true, CORS headers will be emitted, and requests to the
 	// API will be rejected if their `Host` and `Origin` headers
 	// do not match the expected value(s). Use `origins` to
-	// customize which origins/hosts are allowed.If `origins` is
+	// customize which origins/hosts are allowed. If `origins` is
 	// not set, the listen address is the only value allowed by
-	// default.
+	// default. Enforced only on local (plaintext) endpoint.
 	EnforceOrigin bool `json:"enforce_origin,omitempty"`
 
-	// The list of allowed origins for API requests. Only used if
-	// `enforce_origin` is true. If not set, the listener address
-	// will be the default value. If set but empty, no origins will
-	// be allowed.
+	// The list of allowed origins/hosts for API requests. Only needed
+	// if accessing the admin endpoint from a host different from the
+	// socket's network interface or if `enforce_origin` is true. If not
+	// set, the listener address will be the default value. If set but
+	// empty, no origins will be allowed. Enforced only on local
+	// (plaintext) endpoint.
 	Origins []string `json:"origins,omitempty"`
 
-	// Options related to configuration management.
+	// Options pertaining to configuration management.
 	Config *ConfigSettings `json:"config,omitempty"`
+
+	// Options that establish this server's identity. Identity refers to
+	// credentials which can be used to uniquely identify and authenticate
+	// this server instance. This is required if remote administration is
+	// enabled (but does not require remote administration to be enabled).
+	// Default: no identity management.
+	Identity *IdentityConfig `json:"identity,omitempty"`
+
+	// Options pertaining to remote administration. By default, remote
+	// administration is disabled. If enabled, identity management must
+	// also be configured, as that is how the endpoint is secured.
+	// See the neighboring "identity" object.
+	//
+	// EXPERIMENTAL: This feature is subject to change.
+	Remote *RemoteAdmin `json:"remote,omitempty"`
 }
 
-// ConfigSettings configures the, uh, configuration... and
-// management thereof.
+// ConfigSettings configures the management of configuration.
 type ConfigSettings struct {
 	// Whether to keep a copy of the active config on disk. Default is true.
+	// Note that "pulled" dynamic configs (using the neighboring "load" module)
+	// are not persisted; only configs that are pushed to Caddy get persisted.
 	Persist *bool `json:"persist,omitempty"`
+
+	// Loads a configuration to use. This is helpful if your configs are
+	// managed elsewhere, and you want Caddy to pull its config dynamically
+	// when it starts. The pulled config completely replaces the current
+	// one, just like any other config load. It is an error if a pulled
+	// config is configured to pull another config.
+	//
+	// EXPERIMENTAL: Subject to change.
+	LoadRaw json.RawMessage `json:"load,omitempty" caddy:"namespace=caddy.config_loaders inline_key=module"`
 }
 
-// listenAddr extracts a singular listen address from ac.Listen,
-// returning the network and the address of the listener.
-func (admin AdminConfig) listenAddr() (string, string, error) {
-	input := admin.Listen
-	if input == "" {
-		input = DefaultAdminListen
-	}
-	listenAddr, err := ParseNetworkAddress(input)
-	if err != nil {
-		return "", "", fmt.Errorf("parsing admin listener address: %v", err)
-	}
-	if listenAddr.PortRangeSize() != 1 {
-		return "", "", fmt.Errorf("admin endpoint must have exactly one address; cannot listen on %v", listenAddr)
-	}
-	return listenAddr.Network, listenAddr.JoinHostPort(0), nil
+// IdentityConfig configures management of this server's identity. An identity
+// consists of credentials that uniquely verify this instance; for example,
+// TLS certificates (public + private key pairs).
+type IdentityConfig struct {
+	// List of names or IP addresses which refer to this server.
+	// Certificates will be obtained for these identifiers so
+	// secure TLS connections can be made using them.
+	Identifiers []string `json:"identifiers,omitempty"`
+
+	// Issuers that can provide this admin endpoint its identity
+	// certificate(s). Default: ACME issuers configured for
+	// ZeroSSL and Let's Encrypt. Be sure to change this if you
+	// require credentials for private identifiers.
+	IssuersRaw []json.RawMessage `json:"issuers,omitempty" caddy:"namespace=tls.issuance inline_key=module"`
+
+	issuers []certmagic.Issuer
+}
+
+// RemoteAdmin enables and configures remote administration. If enabled,
+// a secure listener enforcing mutual TLS authentication will be started
+// on a different port from the standard plaintext admin server.
+//
+// This endpoint is secured using identity management, which must be
+// configured separately (because identity management does not depend
+// on remote administration). See the admin/identity config struct.
+//
+// EXPERIMENTAL: Subject to change.
+type RemoteAdmin struct {
+	// The address on which to start the secure listener.
+	// Default: :2021
+	Listen string `json:"listen,omitempty"`
+
+	// List of access controls for this secure admin endpoint.
+	// This configures TLS mutual authentication (i.e. authorized
+	// client certificates), but also application-layer permissions
+	// like which paths and methods each identity is authorized for.
+	AccessControl []*AdminAccess `json:"access_control,omitempty"`
+}
+
+// AdminAccess specifies what permissions an identity or group
+// of identities are granted.
+type AdminAccess struct {
+	// Base64-encoded DER certificates containing public keys to accept.
+	// (The contents of PEM certificate blocks are base64-encoded DER.)
+	// Any of these public keys can appear in any part of a verified chain.
+	PublicKeys []string `json:"public_keys,omitempty"`
+
+	// Limits what the associated identities are allowed to do.
+	// If unspecified, all permissions are granted.
+	Permissions []AdminPermissions `json:"permissions,omitempty"`
+
+	publicKeys []crypto.PublicKey
+}
+
+// AdminPermissions specifies what kinds of requests are allowed
+// to be made to the admin endpoint.
+type AdminPermissions struct {
+	// The API paths allowed. Paths are simple prefix matches.
+	// Any subpath of the specified paths will be allowed.
+	Paths []string `json:"paths,omitempty"`
+
+	// The HTTP methods allowed for the given paths.
+	Methods []string `json:"methods,omitempty"`
 }
 
 // newAdminHandler reads admin's config and returns an http.Handler suitable
 // for use in an admin endpoint server, which will be listening on listenAddr.
-func (admin AdminConfig) newAdminHandler(listenAddr string) adminHandler {
-	muxWrap := adminHandler{
-		enforceOrigin:  admin.EnforceOrigin,
-		allowedOrigins: admin.allowedOrigins(listenAddr),
-		mux:            http.NewServeMux(),
+func (admin AdminConfig) newAdminHandler(addr NetworkAddress, remote bool) adminHandler {
+	muxWrap := adminHandler{mux: http.NewServeMux()}
+
+	// secure the local or remote endpoint respectively
+	if remote {
+		muxWrap.remoteControl = admin.Remote
+	} else {
+		muxWrap.enforceHost = !addr.isWildcardInterface()
+		muxWrap.allowedOrigins = admin.allowedOrigins(addr)
 	}
 
+	addRouteWithMetrics := func(pattern string, handlerLabel string, h http.Handler) {
+		labels := prometheus.Labels{"path": pattern, "handler": handlerLabel}
+		h = instrumentHandlerCounter(
+			adminMetrics.requestCount.MustCurryWith(labels),
+			h,
+		)
+		muxWrap.mux.Handle(pattern, h)
+	}
 	// addRoute just calls muxWrap.mux.Handle after
 	// wrapping the handler with error handling
-	addRoute := func(pattern string, h AdminHandler) {
+	addRoute := func(pattern string, handlerLabel string, h AdminHandler) {
 		wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 			err := h.ServeHTTP(w, r)
+			if err != nil {
+				labels := prometheus.Labels{
+					"path":    pattern,
+					"handler": handlerLabel,
+					"method":  strings.ToUpper(r.Method),
+				}
+				adminMetrics.requestErrors.With(labels).Inc()
+			}
 			muxWrap.handleError(w, r, err)
 		})
-		muxWrap.mux.Handle(pattern, wrapper)
+		addRouteWithMetrics(pattern, handlerLabel, wrapper)
 	}
 
+	const handlerLabel = "admin"
+
 	// register standard config control endpoints
-	addRoute("/"+rawConfigKey+"/", AdminHandlerFunc(handleConfig))
-	addRoute("/id/", AdminHandlerFunc(handleConfigID))
-	addRoute("/stop", AdminHandlerFunc(handleStop))
+	addRoute("/"+rawConfigKey+"/", handlerLabel, AdminHandlerFunc(handleConfig))
+	addRoute("/id/", handlerLabel, AdminHandlerFunc(handleConfigID))
+	addRoute("/stop", handlerLabel, AdminHandlerFunc(handleStop))
 
 	// register debugging endpoints
-	muxWrap.mux.HandleFunc("/debug/pprof/", pprof.Index)
-	muxWrap.mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
-	muxWrap.mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
-	muxWrap.mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
-	muxWrap.mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
-	muxWrap.mux.Handle("/debug/vars", expvar.Handler())
+	addRouteWithMetrics("/debug/pprof/", handlerLabel, http.HandlerFunc(pprof.Index))
+	addRouteWithMetrics("/debug/pprof/cmdline", handlerLabel, http.HandlerFunc(pprof.Cmdline))
+	addRouteWithMetrics("/debug/pprof/profile", handlerLabel, http.HandlerFunc(pprof.Profile))
+	addRouteWithMetrics("/debug/pprof/symbol", handlerLabel, http.HandlerFunc(pprof.Symbol))
+	addRouteWithMetrics("/debug/pprof/trace", handlerLabel, http.HandlerFunc(pprof.Trace))
+	addRouteWithMetrics("/debug/vars", handlerLabel, expvar.Handler())
 
 	// register third-party module endpoints
 	for _, m := range GetModules("admin.api") {
 		router := m.New().(AdminRouter)
+		handlerLabel := m.ID.Name()
 		for _, route := range router.Routes() {
-			addRoute(route.Pattern, route.Handler)
+			addRoute(route.Pattern, handlerLabel, route.Handler)
 		}
 	}
 
@@ -140,34 +246,50 @@ func (admin AdminConfig) newAdminHandler(listenAddr string) adminHandler {
 // If admin.Origins is nil (null), the provided listen address
 // will be used as the default origin. If admin.Origins is
 // empty, no origins will be allowed, effectively bricking the
-// endpoint, but whatever.
-func (admin AdminConfig) allowedOrigins(listen string) []string {
+// endpoint for non-unix-socket endpoints, but whatever.
+func (admin AdminConfig) allowedOrigins(addr NetworkAddress) []string {
 	uniqueOrigins := make(map[string]struct{})
 	for _, o := range admin.Origins {
 		uniqueOrigins[o] = struct{}{}
 	}
 	if admin.Origins == nil {
-		uniqueOrigins[listen] = struct{}{}
+		if addr.isLoopback() {
+			if addr.IsUnixNetwork() {
+				// RFC 2616, Section 14.26:
+				// "A client MUST include a Host header field in all HTTP/1.1 request
+				// messages. If the requested URI does not include an Internet host
+				// name for the service being requested, then the Host header field MUST
+				// be given with an empty value."
+				uniqueOrigins[""] = struct{}{}
+			} else {
+				uniqueOrigins[net.JoinHostPort("localhost", addr.port())] = struct{}{}
+				uniqueOrigins[net.JoinHostPort("::1", addr.port())] = struct{}{}
+				uniqueOrigins[net.JoinHostPort("127.0.0.1", addr.port())] = struct{}{}
+			}
+		}
+		if !addr.IsUnixNetwork() {
+			uniqueOrigins[addr.JoinHostPort(0)] = struct{}{}
+		}
 	}
-	var allowed []string
+	allowed := make([]string, 0, len(uniqueOrigins))
 	for origin := range uniqueOrigins {
 		allowed = append(allowed, origin)
 	}
 	return allowed
 }
 
-// replaceAdmin replaces the running admin server according
-// to the relevant configuration in cfg. If no configuration
-// for the admin endpoint exists in cfg, a default one is
-// used, so that there is always an admin server (unless it
-// is explicitly configured to be disabled).
-func replaceAdmin(cfg *Config) error {
+// replaceLocalAdminServer replaces the running local admin server
+// according to the relevant configuration in cfg. If no configuration
+// for the admin endpoint exists in cfg, a default one is used, so
+// that there is always an admin server (unless it is explicitly
+// configured to be disabled).
+func replaceLocalAdminServer(cfg *Config) error {
 	// always be sure to close down the old admin endpoint
 	// as gracefully as possible, even if the new one is
 	// disabled -- careful to use reference to the current
 	// (old) admin endpoint since it will be different
 	// when the function returns
-	oldAdminServer := adminServer
+	oldAdminServer := localAdminServer
 	defer func() {
 		// do the shutdown asynchronously so that any
 		// current API request gets a response; this
@@ -195,19 +317,20 @@ func replaceAdmin(cfg *Config) error {
 	}
 
 	// extract a singular listener address
-	netw, addr, err := adminConfig.listenAddr()
+	addr, err := parseAdminListenAddr(adminConfig.Listen, DefaultAdminListen)
 	if err != nil {
 		return err
 	}
 
-	handler := adminConfig.newAdminHandler(addr)
+	handler := adminConfig.newAdminHandler(addr, false)
 
-	ln, err := Listen(netw, addr)
+	ln, err := Listen(addr.Network, addr.JoinHostPort(0))
 	if err != nil {
 		return err
 	}
 
-	adminServer = &http.Server{
+	localAdminServer = &http.Server{
+		Addr:              addr.String(), // for logging purposes only
 		Handler:           handler,
 		ReadTimeout:       10 * time.Second,
 		ReadHeaderTimeout: 5 * time.Second,
@@ -215,18 +338,272 @@ func replaceAdmin(cfg *Config) error {
 		MaxHeaderBytes:    1024 * 64,
 	}
 
-	go adminServer.Serve(ln)
+	adminLogger := Log().Named("admin")
+	go func() {
+		if err := localAdminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+			adminLogger.Error("admin server shutdown for unknown reason", zap.Error(err))
+		}
+	}()
 
-	Log().Named("admin").Info(
-		"admin endpoint started",
-		zap.String("address", addr),
+	adminLogger.Info("admin endpoint started",
+		zap.String("address", addr.String()),
 		zap.Bool("enforce_origin", adminConfig.EnforceOrigin),
-		zap.Strings("origins", handler.allowedOrigins),
-	)
+		zap.Strings("origins", handler.allowedOrigins))
+
+	if !handler.enforceHost {
+		adminLogger.Warn("admin endpoint on open interface; host checking disabled",
+			zap.String("address", addr.String()))
+	}
 
 	return nil
 }
 
+// manageIdentity sets up automated identity management for this server.
+func manageIdentity(ctx Context, cfg *Config) error {
+	if cfg == nil || cfg.Admin == nil || cfg.Admin.Identity == nil {
+		return nil
+	}
+
+	oldIdentityCertCache := identityCertCache
+	if oldIdentityCertCache != nil {
+		defer oldIdentityCertCache.Stop()
+	}
+
+	// set default issuers; this is pretty hacky because we can't
+	// import the caddytls package -- but it works
+	if cfg.Admin.Identity.IssuersRaw == nil {
+		cfg.Admin.Identity.IssuersRaw = []json.RawMessage{
+			json.RawMessage(`{"module": "zerossl"}`),
+			json.RawMessage(`{"module": "acme"}`),
+		}
+	}
+
+	// load and provision issuer modules
+	if cfg.Admin.Identity.IssuersRaw != nil {
+		val, err := ctx.LoadModule(cfg.Admin.Identity, "IssuersRaw")
+		if err != nil {
+			return fmt.Errorf("loading identity issuer modules: %s", err)
+		}
+		for _, issVal := range val.([]interface{}) {
+			cfg.Admin.Identity.issuers = append(cfg.Admin.Identity.issuers, issVal.(certmagic.Issuer))
+		}
+	}
+
+	logger := Log().Named("admin.identity")
+	cmCfg := cfg.Admin.Identity.certmagicConfig(logger)
+
+	// issuers have circular dependencies with the configs because,
+	// as explained in the caddytls package, they need access to the
+	// correct storage and cache to solve ACME challenges
+	for _, issuer := range cfg.Admin.Identity.issuers {
+		// avoid import cycle with caddytls package, so manually duplicate the interface here, yuck
+		if annoying, ok := issuer.(interface{ SetConfig(cfg *certmagic.Config) }); ok {
+			annoying.SetConfig(cmCfg)
+		}
+	}
+
+	// obtain and renew server identity certificate(s)
+	return cmCfg.ManageAsync(ctx, cfg.Admin.Identity.Identifiers)
+}
+
+// replaceRemoteAdminServer replaces the running remote admin server
+// according to the relevant configuration in cfg. It stops any previous
+// remote admin server and only starts a new one if configured.
+func replaceRemoteAdminServer(ctx Context, cfg *Config) error {
+	if cfg == nil {
+		return nil
+	}
+
+	remoteLogger := Log().Named("admin.remote")
+
+	oldAdminServer := remoteAdminServer
+	defer func() {
+		if oldAdminServer != nil {
+			go func(oldAdminServer *http.Server) {
+				err := stopAdminServer(oldAdminServer)
+				if err != nil {
+					Log().Named("admin").Error("stopping current secure admin endpoint", zap.Error(err))
+				}
+			}(oldAdminServer)
+		}
+	}()
+
+	if cfg.Admin == nil || cfg.Admin.Remote == nil {
+		return nil
+	}
+
+	addr, err := parseAdminListenAddr(cfg.Admin.Remote.Listen, DefaultRemoteAdminListen)
+	if err != nil {
+		return err
+	}
+
+	// make the HTTP handler but disable Host/Origin enforcement
+	// because we are using TLS authentication instead
+	handler := cfg.Admin.newAdminHandler(addr, true)
+
+	// create client certificate pool for TLS mutual auth, and extract public keys
+	// so that we can enforce access controls at the application layer
+	clientCertPool := x509.NewCertPool()
+	for i, accessControl := range cfg.Admin.Remote.AccessControl {
+		for j, certBase64 := range accessControl.PublicKeys {
+			cert, err := decodeBase64DERCert(certBase64)
+			if err != nil {
+				return fmt.Errorf("access control %d public key %d: parsing base64 certificate DER: %v", i, j, err)
+			}
+			accessControl.publicKeys = append(accessControl.publicKeys, cert.PublicKey)
+			clientCertPool.AddCert(cert)
+		}
+	}
+
+	// create TLS config that will enforce mutual authentication
+	cmCfg := cfg.Admin.Identity.certmagicConfig(remoteLogger)
+	tlsConfig := cmCfg.TLSConfig()
+	tlsConfig.NextProtos = nil // this server does not solve ACME challenges
+	tlsConfig.ClientAuth = tls.RequireAndVerifyClientCert
+	tlsConfig.ClientCAs = clientCertPool
+
+	// convert logger to stdlib so it can be used by HTTP server
+	serverLogger, err := zap.NewStdLogAt(remoteLogger, zap.DebugLevel)
+	if err != nil {
+		return err
+	}
+
+	// create secure HTTP server
+	remoteAdminServer = &http.Server{
+		Addr:              addr.String(), // for logging purposes only
+		Handler:           handler,
+		TLSConfig:         tlsConfig,
+		ReadTimeout:       10 * time.Second,
+		ReadHeaderTimeout: 5 * time.Second,
+		IdleTimeout:       60 * time.Second,
+		MaxHeaderBytes:    1024 * 64,
+		ErrorLog:          serverLogger,
+	}
+
+	// start listener
+	ln, err := Listen(addr.Network, addr.JoinHostPort(0))
+	if err != nil {
+		return err
+	}
+	ln = tls.NewListener(ln, tlsConfig)
+
+	go func() {
+		if err := remoteAdminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+			remoteLogger.Error("admin remote server shutdown for unknown reason", zap.Error(err))
+		}
+	}()
+
+	remoteLogger.Info("secure admin remote control endpoint started",
+		zap.String("address", addr.String()))
+
+	return nil
+}
+
+func (ident *IdentityConfig) certmagicConfig(logger *zap.Logger) *certmagic.Config {
+	if ident == nil {
+		// user might not have configured identity; that's OK, we can still make a
+		// certmagic config, although it'll be mostly useless for remote management
+		ident = new(IdentityConfig)
+	}
+	cmCfg := &certmagic.Config{
+		Storage: DefaultStorage, // do not act as part of a cluster (this is for the server's local identity)
+		Logger:  logger,
+		Issuers: ident.issuers,
+	}
+	if identityCertCache == nil {
+		identityCertCache = certmagic.NewCache(certmagic.CacheOptions{
+			GetConfigForCert: func(certmagic.Certificate) (*certmagic.Config, error) {
+				return cmCfg, nil
+			},
+		})
+	}
+	return certmagic.New(identityCertCache, *cmCfg)
+}
+
+// IdentityCredentials returns this instance's configured, managed identity credentials
+// that can be used in TLS client authentication.
+func (ctx Context) IdentityCredentials(logger *zap.Logger) ([]tls.Certificate, error) {
+	if ctx.cfg == nil || ctx.cfg.Admin == nil || ctx.cfg.Admin.Identity == nil {
+		return nil, fmt.Errorf("no server identity configured")
+	}
+	ident := ctx.cfg.Admin.Identity
+	if len(ident.Identifiers) == 0 {
+		return nil, fmt.Errorf("no identifiers configured")
+	}
+	if logger == nil {
+		logger = Log()
+	}
+	magic := ident.certmagicConfig(logger)
+	return magic.ClientCredentials(ctx, ident.Identifiers)
+}
+
+// enforceAccessControls enforces application-layer access controls for r based on remote.
+// It expects that the TLS server has already established at least one verified chain of
+// trust, and then looks for a matching, authorized public key that is allowed to access
+// the defined path(s) using the defined method(s).
+func (remote RemoteAdmin) enforceAccessControls(r *http.Request) error {
+	for _, chain := range r.TLS.VerifiedChains {
+		for _, peerCert := range chain {
+			for _, adminAccess := range remote.AccessControl {
+				for _, allowedKey := range adminAccess.publicKeys {
+					// see if we found a matching public key; the TLS server already verified the chain
+					// so we know the client possesses the associated private key; this handy interface
+					// doesn't appear to be defined anywhere in the std lib, but was implemented here:
+					// https://github.com/golang/go/commit/b5f2c0f50297fa5cd14af668ddd7fd923626cf8c
+					comparer, ok := peerCert.PublicKey.(interface{ Equal(crypto.PublicKey) bool })
+					if !ok || !comparer.Equal(allowedKey) {
+						continue
+					}
+
+					// key recognized; make sure its HTTP request is permitted
+					for _, accessPerm := range adminAccess.Permissions {
+						// verify method
+						methodFound := accessPerm.Methods == nil
+						for _, method := range accessPerm.Methods {
+							if method == r.Method {
+								methodFound = true
+								break
+							}
+						}
+						if !methodFound {
+							return APIError{
+								HTTPStatus: http.StatusForbidden,
+								Message:    "not authorized to use this method",
+							}
+						}
+
+						// verify path
+						pathFound := accessPerm.Paths == nil
+						for _, allowedPath := range accessPerm.Paths {
+							if strings.HasPrefix(r.URL.Path, allowedPath) {
+								pathFound = true
+								break
+							}
+						}
+						if !pathFound {
+							return APIError{
+								HTTPStatus: http.StatusForbidden,
+								Message:    "not authorized to access this path",
+							}
+						}
+					}
+
+					// public key authorized, method and path allowed
+					return nil
+				}
+			}
+		}
+	}
+
+	// in theory, this should never happen; with an unverified chain, the TLS server
+	// should not accept the connection in the first place, and the acceptable cert
+	// pool is configured using the same list of public keys we verify against
+	return APIError{
+		HTTPStatus: http.StatusUnauthorized,
+		Message:    "client identity not authorized",
+	}
+}
+
 func stopAdminServer(srv *http.Server) error {
 	if srv == nil {
 		return fmt.Errorf("no admin server")
@@ -237,7 +614,7 @@ func stopAdminServer(srv *http.Server) error {
 	if err != nil {
 		return fmt.Errorf("shutting down admin server: %v", err)
 	}
-	Log().Named("admin").Info("stopped previous server")
+	Log().Named("admin").Info("stopped previous server", zap.String("address", srv.Addr))
 	return nil
 }
 
@@ -253,20 +630,38 @@ type AdminRoute struct {
 }
 
 type adminHandler struct {
+	mux *http.ServeMux
+
+	// security for local/plaintext) endpoint, on by default
 	enforceOrigin  bool
+	enforceHost    bool
 	allowedOrigins []string
-	mux            *http.ServeMux
+
+	// security for remote/encrypted endpoint
+	remoteControl *RemoteAdmin
 }
 
 // ServeHTTP is the external entry point for API requests.
 // It will only be called once per request.
 func (h adminHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
-	Log().Named("admin.api").Info("received request",
+	log := Log().Named("admin.api").With(
 		zap.String("method", r.Method),
+		zap.String("host", r.Host),
 		zap.String("uri", r.RequestURI),
 		zap.String("remote_addr", r.RemoteAddr),
 		zap.Reflect("headers", r.Header),
 	)
+	if r.TLS != nil {
+		log = log.With(
+			zap.Bool("secure", true),
+			zap.Int("verified_chains", len(r.TLS.VerifiedChains)),
+		)
+	}
+	if r.RequestURI == "/metrics" {
+		log.Debug("received request")
+	} else {
+		log.Info("received request")
+	}
 	h.serveHTTP(w, r)
 }
 
@@ -274,14 +669,32 @@ func (h adminHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 // be called more than once per request, for example if a request
 // is rewritten (i.e. internal redirect).
 func (h adminHandler) serveHTTP(w http.ResponseWriter, r *http.Request) {
-	if h.enforceOrigin {
+	if h.remoteControl != nil {
+		// enforce access controls on secure endpoint
+		if err := h.remoteControl.enforceAccessControls(r); err != nil {
+			h.handleError(w, r, err)
+			return
+		}
+	}
+
+	if strings.Contains(r.Header.Get("Upgrade"), "websocket") {
+		// I've never been able demonstrate a vulnerability myself, but apparently
+		// WebSocket connections originating from browsers aren't subject to CORS
+		// restrictions, so we'll just be on the safe side
+		h.handleError(w, r, fmt.Errorf("websocket connections aren't allowed"))
+		return
+	}
+
+	if h.enforceHost {
 		// DNS rebinding mitigation
 		err := h.checkHost(r)
 		if err != nil {
 			h.handleError(w, r, err)
 			return
 		}
+	}
 
+	if h.enforceOrigin {
 		// cross-site mitigation
 		origin, err := h.checkOrigin(r)
 		if err != nil {
@@ -297,8 +710,6 @@ func (h adminHandler) serveHTTP(w http.ResponseWriter, r *http.Request) {
 		w.Header().Set("Access-Control-Allow-Origin", origin)
 	}
 
-	// TODO: authentication & authorization, if configured
-
 	h.mux.ServeHTTP(w, r)
 }
 
@@ -306,20 +717,16 @@ func (h adminHandler) handleError(w http.ResponseWriter, r *http.Request, err er
 	if err == nil {
 		return
 	}
-	if err == ErrInternalRedir {
-		h.serveHTTP(w, r)
-		return
-	}
 
 	apiErr, ok := err.(APIError)
 	if !ok {
 		apiErr = APIError{
-			Code: http.StatusInternalServerError,
-			Err:  err,
+			HTTPStatus: http.StatusInternalServerError,
+			Err:        err,
 		}
 	}
-	if apiErr.Code == 0 {
-		apiErr.Code = http.StatusInternalServerError
+	if apiErr.HTTPStatus == 0 {
+		apiErr.HTTPStatus = http.StatusInternalServerError
 	}
 	if apiErr.Message == "" && apiErr.Err != nil {
 		apiErr.Message = apiErr.Err.Error()
@@ -327,12 +734,15 @@ func (h adminHandler) handleError(w http.ResponseWriter, r *http.Request, err er
 
 	Log().Named("admin.api").Error("request error",
 		zap.Error(err),
-		zap.Int("status_code", apiErr.Code),
+		zap.Int("status_code", apiErr.HTTPStatus),
 	)
 
 	w.Header().Set("Content-Type", "application/json")
-	w.WriteHeader(apiErr.Code)
-	json.NewEncoder(w).Encode(apiErr)
+	w.WriteHeader(apiErr.HTTPStatus)
+	encErr := json.NewEncoder(w).Encode(apiErr)
+	if encErr != nil {
+		Log().Named("admin.api").Error("failed to encode error response", zap.Error(encErr))
+	}
 }
 
 // checkHost returns a handler that wraps next such that
@@ -349,8 +759,8 @@ func (h adminHandler) checkHost(r *http.Request) error {
 	}
 	if !allowed {
 		return APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("host not allowed: %s", r.Host),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("host not allowed: %s", r.Host),
 		}
 	}
 	return nil
@@ -364,14 +774,14 @@ func (h adminHandler) checkOrigin(r *http.Request) (string, error) {
 	origin := h.getOriginHost(r)
 	if origin == "" {
 		return origin, APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("missing required Origin header"),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("missing required Origin header"),
 		}
 	}
 	if !h.originAllowed(origin) {
 		return origin, APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("client is not allowed to access from origin %s", origin),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("client is not allowed to access from origin %s", origin),
 		}
 	}
 	return origin, nil
@@ -411,7 +821,7 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 
 		err := readConfig(r.URL.Path, w)
 		if err != nil {
-			return APIError{Code: http.StatusBadRequest, Err: err}
+			return APIError{HTTPStatus: http.StatusBadRequest, Err: err}
 		}
 
 		return nil
@@ -426,8 +836,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 		if r.Method != http.MethodDelete {
 			if ct := r.Header.Get("Content-Type"); !strings.Contains(ct, "/json") {
 				return APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("unacceptable content-type: %v; 'application/json' required", ct),
+					HTTPStatus: http.StatusBadRequest,
+					Err:        fmt.Errorf("unacceptable content-type: %v; 'application/json' required", ct),
 				}
 			}
 
@@ -438,8 +848,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 			_, err := io.Copy(buf, r.Body)
 			if err != nil {
 				return APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("reading request body: %v", err),
+					HTTPStatus: http.StatusBadRequest,
+					Err:        fmt.Errorf("reading request body: %v", err),
 				}
 			}
 			body = buf.Bytes()
@@ -454,8 +864,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 
 	default:
 		return APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method %s not allowed", r.Method),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method %s not allowed", r.Method),
 		}
 	}
 
@@ -486,50 +896,22 @@ func handleConfigID(w http.ResponseWriter, r *http.Request) error {
 	parts = append([]string{expanded}, parts[3:]...)
 	r.URL.Path = path.Join(parts...)
 
-	return ErrInternalRedir
-}
-
-func handleStop(w http.ResponseWriter, r *http.Request) error {
-	err := handleUnload(w, r)
-	if err != nil {
-		Log().Named("admin.api").Error("unload error", zap.Error(err))
-	}
-	go func() {
-		err := stopAdminServer(adminServer)
-		var exitCode int
-		if err != nil {
-			exitCode = ExitCodeFailedQuit
-			Log().Named("admin.api").Error("failed to stop admin server gracefully", zap.Error(err))
-		}
-		Log().Named("admin.api").Info("stopping now, bye!! 👋")
-		os.Exit(exitCode)
-	}()
 	return nil
 }
 
-// handleUnload stops the current configuration that is running.
-// Note that doing this can also be accomplished with DELETE /config/
-// but we leave this function because handleStop uses it.
-func handleUnload(w http.ResponseWriter, r *http.Request) error {
+func handleStop(w http.ResponseWriter, r *http.Request) error {
 	if r.Method != http.MethodPost {
 		return APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method not allowed"),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method not allowed"),
 		}
 	}
-	currentCfgMu.RLock()
-	hasCfg := currentCfg != nil
-	currentCfgMu.RUnlock()
-	if !hasCfg {
-		Log().Named("admin.api").Info("nothing to unload")
-		return nil
-	}
-	Log().Named("admin.api").Info("unloading")
-	if err := stopAndCleanup(); err != nil {
-		Log().Named("admin.api").Error("error unloading", zap.Error(err))
-	} else {
-		Log().Named("admin.api").Info("unloading completed")
+
+	if err := notify.NotifyStopping(); err != nil {
+		Log().Error("unable to notify stopping to service manager", zap.Error(err))
 	}
+
+	exitProcess(Log().Named("admin.api"))
 	return nil
 }
 
@@ -741,9 +1123,9 @@ func (f AdminHandlerFunc) ServeHTTP(w http.ResponseWriter, r *http.Request) erro
 // and client responses. If Message is unset, then
 // Err.Error() will be serialized in its place.
 type APIError struct {
-	Code    int    `json:"-"`
-	Err     error  `json:"-"`
-	Message string `json:"error"`
+	HTTPStatus int    `json:"-"`
+	Err        error  `json:"-"`
+	Message    string `json:"error"`
 }
 
 func (e APIError) Error() string {
@@ -753,31 +1135,70 @@ func (e APIError) Error() string {
 	return e.Message
 }
 
+// parseAdminListenAddr extracts a singular listen address from either addr
+// or defaultAddr, returning the network and the address of the listener.
+func parseAdminListenAddr(addr string, defaultAddr string) (NetworkAddress, error) {
+	input := addr
+	if input == "" {
+		input = defaultAddr
+	}
+	listenAddr, err := ParseNetworkAddress(input)
+	if err != nil {
+		return NetworkAddress{}, fmt.Errorf("parsing listener address: %v", err)
+	}
+	if listenAddr.PortRangeSize() != 1 {
+		return NetworkAddress{}, fmt.Errorf("must be exactly one listener address; cannot listen on: %s", listenAddr)
+	}
+	return listenAddr, nil
+}
+
+// decodeBase64DERCert base64-decodes, then DER-decodes, certStr.
+func decodeBase64DERCert(certStr string) (*x509.Certificate, error) {
+	derBytes, err := base64.StdEncoding.DecodeString(certStr)
+	if err != nil {
+		return nil, err
+	}
+	return x509.ParseCertificate(derBytes)
+}
+
 var (
-	// DefaultAdminListen is the address for the admin
+	// DefaultAdminListen is the address for the local admin
 	// listener, if none is specified at startup.
 	DefaultAdminListen = "localhost:2019"
 
-	// ErrInternalRedir indicates an internal redirect
-	// and is useful when admin API handlers rewrite
-	// the request; in that case, authentication and
-	// authorization needs to happen again for the
-	// rewritten request.
-	ErrInternalRedir = fmt.Errorf("internal redirect; re-authorization required")
+	// DefaultRemoteAdminListen is the address for the remote
+	// (TLS-authenticated) admin listener, if enabled and not
+	// specified otherwise.
+	DefaultRemoteAdminListen = ":2021"
 
 	// DefaultAdminConfig is the default configuration
-	// for the administration endpoint.
+	// for the local administration endpoint.
 	DefaultAdminConfig = &AdminConfig{
 		Listen: DefaultAdminListen,
 	}
 )
 
+// PIDFile writes a pidfile to the file at filename. It
+// will get deleted before the process gracefully exits.
+func PIDFile(filename string) error {
+	pid := []byte(strconv.Itoa(os.Getpid()) + "\n")
+	err := ioutil.WriteFile(filename, pid, 0600)
+	if err != nil {
+		return err
+	}
+	pidfile = filename
+	return nil
+}
+
 // idRegexp is used to match ID fields and their associated values
 // in the config. It also matches adjacent commas so that syntax
 // can be preserved no matter where in the object the field appears.
 // It supports string and most numeric values.
 var idRegexp = regexp.MustCompile(`(?m),?\s*"` + idKey + `"\s*:\s*(-?[0-9]+(\.[0-9]+)?|(?U)".*")\s*,?`)
 
+// pidfile is the name of the pidfile, if any.
+var pidfile string
+
 const (
 	rawConfigKey = "config"
 	idKey        = "@id"
@@ -789,4 +1210,8 @@ var bufPool = sync.Pool{
 	},
 }
 
-var adminServer *http.Server
+// keep a reference to admin endpoint singletons while they're active
+var (
+	localAdminServer, remoteAdminServer *http.Server
+	identityCertCache                   *certmagic.Cache
+)

caddy.go

@@ -32,7 +32,9 @@ import (
 	"sync"
 	"time"
 
+	"github.com/caddyserver/caddy/v2/notify"
 	"github.com/caddyserver/certmagic"
+	"github.com/google/uuid"
 	"go.uber.org/zap"
 )
 
@@ -99,6 +101,16 @@ func Run(cfg *Config) error {
 // if it is different from the current config or
 // forceReload is true.
 func Load(cfgJSON []byte, forceReload bool) error {
+	if err := notify.NotifyReloading(); err != nil {
+		Log().Error("unable to notify reloading to service manager", zap.Error(err))
+	}
+
+	defer func() {
+		if err := notify.NotifyReadiness(); err != nil {
+			Log().Error("unable to notify readiness to service manager", zap.Error(err))
+		}
+	}()
+
 	return changeConfig(http.MethodPost, "/"+rawConfigKey, cfgJSON, forceReload)
 }
 
@@ -130,8 +142,8 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 	newCfg, err := json.Marshal(rawCfg[rawConfigKey])
 	if err != nil {
 		return APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("encoding new config: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("encoding new config: %v", err),
 		}
 	}
 
@@ -146,14 +158,14 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 	err = indexConfigObjects(rawCfg[rawConfigKey], "/"+rawConfigKey, idx)
 	if err != nil {
 		return APIError{
-			Code: http.StatusInternalServerError,
-			Err:  fmt.Errorf("indexing config: %v", err),
+			HTTPStatus: http.StatusInternalServerError,
+			Err:        fmt.Errorf("indexing config: %v", err),
 		}
 	}
 
 	// load this new config; if it fails, we need to revert to
 	// our old representation of caddy's actual config
-	err = unsyncedDecodeAndRun(newCfg)
+	err = unsyncedDecodeAndRun(newCfg, true)
 	if err != nil {
 		if len(rawCfgJSON) > 0 {
 			// restore old config state to keep it consistent
@@ -233,8 +245,10 @@ func indexConfigObjects(ptr interface{}, configPath string, index map[string]str
 // it as the new config, replacing any other current config.
 // It does NOT update the raw config state, as this is a
 // lower-level function; most callers will want to use Load
-// instead. A write lock on currentCfgMu is required!
-func unsyncedDecodeAndRun(cfgJSON []byte) error {
+// instead. A write lock on currentCfgMu is required! If
+// allowPersist is false, it will not be persisted to disk,
+// even if it is configured to.
+func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
 	// remove any @id fields from the JSON, which would cause
 	// loading to break since the field wouldn't be recognized
 	strippedCfgJSON := RemoveMetaFields(cfgJSON)
@@ -245,6 +259,19 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 		return err
 	}
 
+	// prevent recursive config loads; that is a user error, and
+	// although frequent config loads should be safe, we cannot
+	// guarantee that in the presence of third party plugins, nor
+	// do we want this error to go unnoticed (we assume it was a
+	// pulled config if we're not allowed to persist it)
+	if !allowPersist &&
+		newCfg != nil &&
+		newCfg.Admin != nil &&
+		newCfg.Admin.Config != nil &&
+		newCfg.Admin.Config.LoadRaw != nil {
+		return fmt.Errorf("recursive config loading detected: pulled configs cannot pull other configs")
+	}
+
 	// run the new config and start all its apps
 	err = run(newCfg, true)
 	if err != nil {
@@ -259,7 +286,8 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 	unsyncedStop(oldCfg)
 
 	// autosave a non-nil config, if not disabled
-	if newCfg != nil &&
+	if allowPersist &&
+		newCfg != nil &&
 		(newCfg.Admin == nil ||
 			newCfg.Admin.Config == nil ||
 			newCfg.Admin.Config.Persist == nil ||
@@ -273,7 +301,7 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 		} else {
 			err := ioutil.WriteFile(ConfigAutosavePath, cfgJSON, 0600)
 			if err == nil {
-				Log().Info("autosaved config", zap.String("file", ConfigAutosavePath))
+				Log().Info("autosaved config (load with --resume flag)", zap.String("file", ConfigAutosavePath))
 			} else {
 				Log().Error("unable to autosave config",
 					zap.String("file", ConfigAutosavePath),
@@ -309,21 +337,10 @@ func run(newCfg *Config, start bool) error {
 	// been set by a short assignment
 	var err error
 
-	// start the admin endpoint (and stop any prior one)
-	if start {
-		err = replaceAdmin(newCfg)
-		if err != nil {
-			return fmt.Errorf("starting caddy administration endpoint: %v", err)
-		}
-	}
-
 	if newCfg == nil {
-		return nil
+		newCfg = new(Config)
 	}
 
-	// prepare the new config for use
-	newCfg.apps = make(map[string]App)
-
 	// create a context within which to load
 	// modules - essentially our new config's
 	// execution environment; be sure that
@@ -357,6 +374,17 @@ func run(newCfg *Config, start bool) error {
 		return err
 	}
 
+	// start the admin endpoint (and stop any prior one)
+	if start {
+		err = replaceLocalAdminServer(newCfg)
+		if err != nil {
+			return fmt.Errorf("starting caddy administration endpoint: %v", err)
+		}
+	}
+
+	// prepare the new config for use
+	newCfg.apps = make(map[string]App)
+
 	// set up global storage and make it CertMagic's default storage, too
 	err = func() error {
 		if newCfg.StorageRaw != nil {
@@ -400,7 +428,7 @@ func run(newCfg *Config, start bool) error {
 	}
 
 	// Start
-	return func() error {
+	err = func() error {
 		var started []string
 		for name, a := range newCfg.apps {
 			err := a.Start()
@@ -420,6 +448,64 @@ func run(newCfg *Config, start bool) error {
 		}
 		return nil
 	}()
+	if err != nil {
+		return err
+	}
+
+	// now that the user's config is running, finish setting up anything else,
+	// such as remote admin endpoint, config loader, etc.
+	return finishSettingUp(ctx, newCfg)
+}
+
+// finishSettingUp should be run after all apps have successfully started.
+func finishSettingUp(ctx Context, cfg *Config) error {
+	// establish this server's identity (only after apps are loaded
+	// so that cert management of this endpoint doesn't prevent user's
+	// servers from starting which likely also use HTTP/HTTPS ports;
+	// but before remote management which may depend on these creds)
+	err := manageIdentity(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("provisioning remote admin endpoint: %v", err)
+	}
+
+	// replace any remote admin endpoint
+	err = replaceRemoteAdminServer(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("provisioning remote admin endpoint: %v", err)
+	}
+
+	// if dynamic config is requested, set that up and run it
+	if cfg != nil && cfg.Admin != nil && cfg.Admin.Config != nil && cfg.Admin.Config.LoadRaw != nil {
+		val, err := ctx.LoadModule(cfg.Admin.Config, "LoadRaw")
+		if err != nil {
+			return fmt.Errorf("loading config loader module: %s", err)
+		}
+		loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
+		if err != nil {
+			return fmt.Errorf("loading dynamic config from %T: %v", val, err)
+		}
+
+		// do this in a goroutine so current config can finish being loaded; otherwise deadlock
+		go func() {
+			Log().Info("applying dynamically-loaded config", zap.String("loader_module", val.(Module).CaddyModule().ID.Name()))
+			currentCfgMu.Lock()
+			err := unsyncedDecodeAndRun(loadedConfig, false)
+			currentCfgMu.Unlock()
+			if err == nil {
+				Log().Info("dynamically-loaded config applied successfully")
+			} else {
+				Log().Error("running dynamically-loaded config failed", zap.Error(err))
+			}
+		}()
+	}
+
+	return nil
+}
+
+// ConfigLoader is a type that can load a Caddy config. The
+// returned config must be valid Caddy JSON.
+type ConfigLoader interface {
+	LoadConfig(Context) ([]byte, error)
 }
 
 // Stop stops running the current configuration.
@@ -462,17 +548,6 @@ func unsyncedStop(cfg *Config) {
 	cfg.cancelFunc()
 }
 
-// stopAndCleanup calls stop and cleans up anything
-// else that is expedient. This should only be used
-// when stopping and not replacing with a new config.
-func stopAndCleanup() error {
-	if err := Stop(); err != nil {
-		return err
-	}
-	certmagic.CleanUpOwnLocks()
-	return nil
-}
-
 // Validate loads, provisions, and validates
 // cfg, but does not start running it.
 func Validate(cfg *Config) error {
@@ -483,10 +558,76 @@ func Validate(cfg *Config) error {
 	return err
 }
 
+// exitProcess exits the process as gracefully as possible,
+// but it always exits, even if there are errors doing so.
+// It stops all apps, cleans up external locks, removes any
+// PID file, and shuts down admin endpoint(s) in a goroutine.
+// Errors are logged along the way, and an appropriate exit
+// code is emitted.
+func exitProcess(logger *zap.Logger) {
+	if logger == nil {
+		logger = Log()
+	}
+	logger.Warn("exiting; byeee!! 👋")
+
+	exitCode := ExitCodeSuccess
+
+	// stop all apps
+	if err := Stop(); err != nil {
+		logger.Error("failed to stop apps", zap.Error(err))
+		exitCode = ExitCodeFailedQuit
+	}
+
+	// clean up certmagic locks
+	certmagic.CleanUpOwnLocks(logger)
+
+	// remove pidfile
+	if pidfile != "" {
+		err := os.Remove(pidfile)
+		if err != nil {
+			logger.Error("cleaning up PID file:",
+				zap.String("pidfile", pidfile),
+				zap.Error(err))
+			exitCode = ExitCodeFailedQuit
+		}
+	}
+
+	// shut down admin endpoint(s) in goroutines so that
+	// if this function was called from an admin handler,
+	// it has a chance to return gracefully
+	// use goroutine so that we can finish responding to API request
+	go func() {
+		defer func() {
+			logger = logger.With(zap.Int("exit_code", exitCode))
+			if exitCode == ExitCodeSuccess {
+				logger.Info("shutdown complete")
+			} else {
+				logger.Error("unclean shutdown")
+			}
+			os.Exit(exitCode)
+		}()
+
+		if remoteAdminServer != nil {
+			err := stopAdminServer(remoteAdminServer)
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				logger.Error("failed to stop remote admin server gracefully", zap.Error(err))
+			}
+		}
+		if localAdminServer != nil {
+			err := stopAdminServer(localAdminServer)
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				logger.Error("failed to stop local admin server gracefully", zap.Error(err))
+			}
+		}
+	}()
+}
+
 // Duration can be an integer or a string. An integer is
 // interpreted as nanoseconds. If a string, it is a Go
 // time.Duration value such as `300ms`, `1.5h`, or `2h45m`;
-// valid units are `ns`, `us`/`µs`, `ms`, `s`, `m`, and `h`.
+// valid units are `ns`, `us`/`µs`, `ms`, `s`, `m`, `h`, and `d`.
 type Duration time.Duration
 
 // UnmarshalJSON satisfies json.Unmarshaler.
@@ -497,7 +638,7 @@ func (d *Duration) UnmarshalJSON(b []byte) error {
 	var dur time.Duration
 	var err error
 	if b[0] == byte('"') && b[len(b)-1] == byte('"') {
-		dur, err = time.ParseDuration(strings.Trim(string(b), `"`))
+		dur, err = ParseDuration(strings.Trim(string(b), `"`))
 	} else {
 		err = json.Unmarshal(b, &dur)
 	}
@@ -505,6 +646,54 @@ func (d *Duration) UnmarshalJSON(b []byte) error {
 	return err
 }
 
+// ParseDuration parses a duration string, adding
+// support for the "d" unit meaning number of days,
+// where a day is assumed to be 24h.
+func ParseDuration(s string) (time.Duration, error) {
+	var inNumber bool
+	var numStart int
+	for i := 0; i < len(s); i++ {
+		ch := s[i]
+		if ch == 'd' {
+			daysStr := s[numStart:i]
+			days, err := strconv.ParseFloat(daysStr, 64)
+			if err != nil {
+				return 0, err
+			}
+			hours := days * 24.0
+			hoursStr := strconv.FormatFloat(hours, 'f', -1, 64)
+			s = s[:numStart] + hoursStr + "h" + s[i+1:]
+			i--
+			continue
+		}
+		if !inNumber {
+			numStart = i
+		}
+		inNumber = (ch >= '0' && ch <= '9') || ch == '.' || ch == '-' || ch == '+'
+	}
+	return time.ParseDuration(s)
+}
+
+// InstanceID returns the UUID for this instance, and generates one if it
+// does not already exist. The UUID is stored in the local data directory,
+// regardless of storage configuration, since each instance is intended to
+// have its own unique ID.
+func InstanceID() (uuid.UUID, error) {
+	uuidFilePath := filepath.Join(AppDataDir(), "instance.uuid")
+	uuidFileBytes, err := ioutil.ReadFile(uuidFilePath)
+	if os.IsNotExist(err) {
+		uuid, err := uuid.NewRandom()
+		if err != nil {
+			return uuid, err
+		}
+		err = ioutil.WriteFile(uuidFilePath, []byte(uuid.String()), 0600)
+		return uuid, err
+	} else if err != nil {
+		return [16]byte{}, err
+	}
+	return uuid.ParseBytes(uuidFileBytes)
+}
+
 // GoModule returns the build info of this Caddy
 // build from debug.BuildInfo (requires Go modules).
 // If no version information is available, a non-nil

caddy_test.go

@@ -0,0 +1,74 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddy
+
+import (
+	"testing"
+	"time"
+)
+
+func TestParseDuration(t *testing.T) {
+	const day = 24 * time.Hour
+	for i, tc := range []struct {
+		input  string
+		expect time.Duration
+	}{
+		{
+			input:  "3h",
+			expect: 3 * time.Hour,
+		},
+		{
+			input:  "1d",
+			expect: day,
+		},
+		{
+			input:  "1d30m",
+			expect: day + 30*time.Minute,
+		},
+		{
+			input:  "1m2d",
+			expect: time.Minute + day*2,
+		},
+		{
+			input:  "1m2d30s",
+			expect: time.Minute + day*2 + 30*time.Second,
+		},
+		{
+			input:  "1d2d",
+			expect: 3 * day,
+		},
+		{
+			input:  "1.5d",
+			expect: time.Duration(1.5 * float64(day)),
+		},
+		{
+			input:  "4m1.25d",
+			expect: 4*time.Minute + time.Duration(1.25*float64(day)),
+		},
+		{
+			input:  "-1.25d12h",
+			expect: time.Duration(-1.25*float64(day)) - 12*time.Hour,
+		},
+	} {
+		actual, err := ParseDuration(tc.input)
+		if err != nil {
+			t.Errorf("Test %d ('%s'): Got error: %v", i, tc.input, err)
+			continue
+		}
+		if actual != tc.expect {
+			t.Errorf("Test %d ('%s'): Expected=%s Actual=%s", i, tc.input, tc.expect, actual)
+		}
+	}
+}

caddyconfig/caddyfile/adapter.go

@@ -15,6 +15,7 @@
 package caddyfile
 
 import (
+	"bytes"
 	"encoding/json"
 	"fmt"
 
@@ -51,15 +52,46 @@ func (a Adapter) Adapt(body []byte, options map[string]interface{}) ([]byte, []c
 		return nil, warnings, err
 	}
 
-	marshalFunc := json.Marshal
-	if options["pretty"] == "true" {
-		marshalFunc = caddyconfig.JSONIndent
+	// lint check: see if input was properly formatted; sometimes messy files files parse
+	// successfully but result in logical errors (the Caddyfile is a bad format, I'm sorry)
+	if warning, different := formattingDifference(filename, body); different {
+		warnings = append(warnings, warning)
 	}
-	result, err := marshalFunc(cfg)
+
+	result, err := json.Marshal(cfg)
 
 	return result, warnings, err
 }
 
+// formattingDifference returns a warning and true if the formatted version
+// is any different from the input; empty warning and false otherwise.
+// TODO: also perform this check on imported files
+func formattingDifference(filename string, body []byte) (caddyconfig.Warning, bool) {
+	// replace windows-style newlines to normalize comparison
+	normalizedBody := bytes.Replace(body, []byte("\r\n"), []byte("\n"), -1)
+
+	formatted := Format(normalizedBody)
+	if bytes.Equal(formatted, normalizedBody) {
+		return caddyconfig.Warning{}, false
+	}
+
+	// find where the difference is
+	line := 1
+	for i, ch := range normalizedBody {
+		if i >= len(formatted) || ch != formatted[i] {
+			break
+		}
+		if ch == '\n' {
+			line++
+		}
+	}
+	return caddyconfig.Warning{
+		File:    filename,
+		Line:    line,
+		Message: "input is not formatted with 'caddy fmt'",
+	}, true
+}
+
 // Unmarshaler is a type that can unmarshal
 // Caddyfile tokens to set itself up for a
 // JSON encoding. The goal of an unmarshaler
@@ -87,5 +119,31 @@ type ServerType interface {
 	Setup([]ServerBlock, map[string]interface{}) (*caddy.Config, []caddyconfig.Warning, error)
 }
 
+// UnmarshalModule instantiates a module with the given ID and invokes
+// UnmarshalCaddyfile on the new value using the immediate next segment
+// of d as input. In other words, d's next token should be the first
+// token of the module's Caddyfile input.
+//
+// This function is used when the next segment of Caddyfile tokens
+// belongs to another Caddy module. The returned value is often
+// type-asserted to the module's associated type for practical use
+// when setting up a config.
+func UnmarshalModule(d *Dispenser, moduleID string) (Unmarshaler, error) {
+	mod, err := caddy.GetModule(moduleID)
+	if err != nil {
+		return nil, d.Errf("getting module named '%s': %v", moduleID, err)
+	}
+	inst := mod.New()
+	unm, ok := inst.(Unmarshaler)
+	if !ok {
+		return nil, d.Errf("module %s is not a Caddyfile unmarshaler; is %T", mod.ID, inst)
+	}
+	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
+	if err != nil {
+		return nil, err
+	}
+	return unm, nil
+}
+
 // Interface guard
 var _ caddyconfig.Adapter = (*Adapter)(nil)

caddyconfig/caddyfile/dispenser.go

@@ -17,6 +17,8 @@ package caddyfile
 import (
 	"errors"
 	"fmt"
+	"io"
+	"log"
 	"strings"
 )
 
@@ -37,6 +39,16 @@ func NewDispenser(tokens []Token) *Dispenser {
 	}
 }
 
+// NewTestDispenser parses input into tokens and creates a new
+// Dispenser for test purposes only; any errors are fatal.
+func NewTestDispenser(input string) *Dispenser {
+	tokens, err := allTokens("Testfile", []byte(input))
+	if err != nil && err != io.EOF {
+		log.Fatalf("getting all tokens from input: %v", err)
+	}
+	return NewDispenser(tokens)
+}
+
 // Next loads the next token. Returns true if a token
 // was loaded; false otherwise. If false, all tokens
 // have been consumed.

caddyconfig/caddyfile/dispenser_test.go

@@ -15,8 +15,6 @@
 package caddyfile
 
 import (
-	"io"
-	"log"
 	"reflect"
 	"strings"
 	"testing"
@@ -306,13 +304,3 @@ func TestDispenser_ArgErr_Err(t *testing.T) {
 		t.Errorf("Expected error message with custom message in it ('foobar'); got '%v'", err)
 	}
 }
-
-// NewTestDispenser parses input into tokens and creates a new
-// Disenser for test purposes only; any errors are fatal.
-func NewTestDispenser(input string) *Dispenser {
-	tokens, err := allTokens("Testfile", []byte(input))
-	if err != nil && err != io.EOF {
-		log.Fatalf("getting all tokens from input: %v", err)
-	}
-	return NewDispenser(tokens)
-}

caddyconfig/caddyfile/formatter.go

@@ -78,6 +78,9 @@ func Format(input []byte) []byte {
 		if comment {
 			if ch == '\n' {
 				comment = false
+				space = true
+				nextLine()
+				continue
 			} else {
 				write(ch)
 				continue
@@ -131,9 +134,6 @@ func Format(input []byte) []byte {
 		//////////////////////////////////////////////////////////
 
 		if ch == '#' {
-			if !spacePrior && !beginningOfLine {
-				write(' ')
-			}
 			comment = true
 		}
 

caddyconfig/caddyfile/formatter_fuzz.go

@@ -0,0 +1,27 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// +build gofuzz
+
+package caddyfile
+
+import "bytes"
+
+func FuzzFormat(input []byte) int {
+	formatted := Format(input)
+	if bytes.Equal(formatted, Format(formatted)) {
+		return 1
+	}
+	return 0
+}

caddyconfig/caddyfile/formatter_test.go

@@ -201,7 +201,7 @@ c
 }
 
 d {
-	e #f
+	e#f
 	# g
 }
 
@@ -229,7 +229,7 @@ bar"
 j {
 "\"k\" l m"
 }`,
-			expect: `"a \"b\" " #c
+			expect: `"a \"b\" "#c
 d
 
 e {
@@ -305,6 +305,60 @@ bar "{\"key\":34}"`,
 
 baz`,
 		},
+		{
+			description: "hash within string is not a comment",
+			input:       `redir / /some/#/path`,
+			expect:      `redir / /some/#/path`,
+		},
+		{
+			description: "brace does not fold into comment above",
+			input: `# comment
+{
+	foo
+}`,
+			expect: `# comment
+{
+	foo
+}`,
+		},
+		{
+			description: "matthewpi/vscode-caddyfile-support#13",
+			input: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+			expect: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+		},
+		{
+			description: "matthewpi/vscode-caddyfile-support#13 - bad formatting",
+			input: `{
+	email {$ACMEEMAIL}
+	#debug
+	}
+
+	block {
+	}
+`,
+			expect: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+		},
 	} {
 		// the formatter should output a trailing newline,
 		// even if the tests aren't written to expect that

caddyconfig/caddyfile/importgraph.go

@@ -0,0 +1,127 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"fmt"
+)
+
+type adjacency map[string][]string
+
+type importGraph struct {
+	nodes map[string]bool
+	edges adjacency
+}
+
+func (i *importGraph) addNode(name string) {
+	if i.nodes == nil {
+		i.nodes = make(map[string]bool)
+	}
+	if _, exists := i.nodes[name]; exists {
+		return
+	}
+	i.nodes[name] = true
+}
+func (i *importGraph) addNodes(names []string) {
+	for _, name := range names {
+		i.addNode(name)
+	}
+}
+
+func (i *importGraph) removeNode(name string) {
+	delete(i.nodes, name)
+}
+func (i *importGraph) removeNodes(names []string) {
+	for _, name := range names {
+		i.removeNode(name)
+	}
+}
+
+func (i *importGraph) addEdge(from, to string) error {
+	if !i.exists(from) || !i.exists(to) {
+		return fmt.Errorf("one of the nodes does not exist")
+	}
+
+	if i.willCycle(to, from) {
+		return fmt.Errorf("a cycle of imports exists between %s and %s", from, to)
+	}
+
+	if i.areConnected(from, to) {
+		// if connected, there's nothing to do
+		return nil
+	}
+
+	if i.nodes == nil {
+		i.nodes = make(map[string]bool)
+	}
+	if i.edges == nil {
+		i.edges = make(adjacency)
+	}
+
+	i.edges[from] = append(i.edges[from], to)
+	return nil
+}
+func (i *importGraph) addEdges(from string, tos []string) error {
+	for _, to := range tos {
+		err := i.addEdge(from, to)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (i *importGraph) areConnected(from, to string) bool {
+	al, ok := i.edges[from]
+	if !ok {
+		return false
+	}
+	for _, v := range al {
+		if v == to {
+			return true
+		}
+	}
+	return false
+}
+
+func (i *importGraph) willCycle(from, to string) bool {
+	collector := make(map[string]bool)
+
+	var visit func(string)
+	visit = func(start string) {
+		if !collector[start] {
+			collector[start] = true
+			for _, v := range i.edges[start] {
+				visit(v)
+			}
+		}
+	}
+
+	for _, v := range i.edges[from] {
+		visit(v)
+	}
+	for k := range collector {
+		if to == k {
+			return true
+		}
+	}
+
+	return false
+}
+
+func (i *importGraph) exists(key string) bool {
+	_, exists := i.nodes[key]
+	return exists
+}

caddyconfig/caddyfile/lexer.go

@@ -16,6 +16,7 @@ package caddyfile
 
 import (
 	"bufio"
+	"bytes"
 	"io"
 	"unicode"
 )
@@ -34,9 +35,11 @@ type (
 
 	// Token represents a single parsable unit.
 	Token struct {
-		File string
-		Line int
-		Text string
+		File        string
+		Line        int
+		Text        string
+		inSnippet   bool
+		snippetName string
 	}
 )
 
@@ -73,7 +76,7 @@ func (l *lexer) load(input io.Reader) error {
 // a token was loaded; false otherwise.
 func (l *lexer) next() bool {
 	var val []rune
-	var comment, quoted, escaped bool
+	var comment, quoted, btQuoted, escaped bool
 
 	makeToken := func() bool {
 		l.token.Text = string(val)
@@ -92,13 +95,13 @@ func (l *lexer) next() bool {
 			panic(err)
 		}
 
-		if !escaped && ch == '\\' {
+		if !escaped && !btQuoted && ch == '\\' {
 			escaped = true
 			continue
 		}
 
-		if quoted {
-			if escaped {
+		if quoted || btQuoted {
+			if quoted && escaped {
 				// all is literal in quoted area,
 				// so only escape quotes
 				if ch != '"' {
@@ -106,7 +109,10 @@ func (l *lexer) next() bool {
 				}
 				escaped = false
 			} else {
-				if ch == '"' {
+				if quoted && ch == '"' {
+					return makeToken()
+				}
+				if btQuoted && ch == '`' {
 					return makeToken()
 				}
 			}
@@ -138,7 +144,7 @@ func (l *lexer) next() bool {
 			continue
 		}
 
-		if ch == '#' {
+		if ch == '#' && len(val) == 0 {
 			comment = true
 		}
 		if comment {
@@ -151,6 +157,10 @@ func (l *lexer) next() bool {
 				quoted = true
 				continue
 			}
+			if ch == '`' {
+				btQuoted = true
+				continue
+			}
 		}
 
 		if escaped {
@@ -161,3 +171,21 @@ func (l *lexer) next() bool {
 		val = append(val, ch)
 	}
 }
+
+// Tokenize takes bytes as input and lexes it into
+// a list of tokens that can be parsed as a Caddyfile.
+// Also takes a filename to fill the token's File as
+// the source of the tokens, which is important to
+// determine relative paths for `import` directives.
+func Tokenize(input []byte, filename string) ([]Token, error) {
+	l := lexer{}
+	if err := l.load(bytes.NewReader(input)); err != nil {
+		return nil, err
+	}
+	var tokens []Token
+	for l.next() {
+		l.token.File = filename
+		tokens = append(tokens, l.token)
+	}
+	return tokens, nil
+}

caddyconfig/caddyfile/lexer_fuzz.go

@@ -12,26 +12,17 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// +build !windows
+// +build gofuzz
 
-package caddycmd
+package caddyfile
 
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"syscall"
-)
-
-func gracefullyStopProcess(pid int) error {
-	fmt.Print("Graceful stop... ")
-	err := syscall.Kill(pid, syscall.SIGINT)
+func FuzzTokenize(input []byte) int {
+	tokens, err := Tokenize(input, "Caddyfile")
 	if err != nil {
-		return fmt.Errorf("kill: %v", err)
+		return 0
 	}
-	return nil
-}
-
-func getProcessName() string {
-	return filepath.Base(os.Args[0])
+	if len(tokens) == 0 {
+		return -1
+	}
+	return 1
 }

caddyconfig/caddyfile/lexer_test.go

@@ -15,37 +15,35 @@
 package caddyfile
 
 import (
-	"log"
-	"strings"
 	"testing"
 )
 
 type lexerTestCase struct {
-	input    string
+	input    []byte
 	expected []Token
 }
 
 func TestLexer(t *testing.T) {
 	testCases := []lexerTestCase{
 		{
-			input: `host:123`,
+			input: []byte(`host:123`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 			},
 		},
 		{
-			input: `host:123
+			input: []byte(`host:123
 
-					directive`,
+					directive`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 3, Text: "directive"},
 			},
 		},
 		{
-			input: `host:123 {
+			input: []byte(`host:123 {
 						directive
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -54,7 +52,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `host:123 { directive }`,
+			input: []byte(`host:123 { directive }`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -63,12 +61,12 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `host:123 {
+			input: []byte(`host:123 {
 						#comment
 						directive
 						# comment
 						foobar # another comment
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -78,8 +76,28 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `a "quoted value" b
-					foobar`,
+			input: []byte(`host:123 {
+						# hash inside string is not a comment
+						redir / /some/#/path
+					}`),
+			expected: []Token{
+				{Line: 1, Text: "host:123"},
+				{Line: 1, Text: "{"},
+				{Line: 3, Text: "redir"},
+				{Line: 3, Text: "/"},
+				{Line: 3, Text: "/some/#/path"},
+				{Line: 4, Text: "}"},
+			},
+		},
+		{
+			input: []byte("# comment at beginning of file\n# comment at beginning of line\nhost:123"),
+			expected: []Token{
+				{Line: 3, Text: "host:123"},
+			},
+		},
+		{
+			input: []byte(`a "quoted value" b
+					foobar`),
 			expected: []Token{
 				{Line: 1, Text: "a"},
 				{Line: 1, Text: "quoted value"},
@@ -88,7 +106,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `A "quoted \"value\" inside" B`,
+			input: []byte(`A "quoted \"value\" inside" B`),
 			expected: []Token{
 				{Line: 1, Text: "A"},
 				{Line: 1, Text: `quoted "value" inside`},
@@ -96,7 +114,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "An escaped \"newline\\\ninside\" quotes",
+			input: []byte("An escaped \"newline\\\ninside\" quotes"),
 			expected: []Token{
 				{Line: 1, Text: "An"},
 				{Line: 1, Text: "escaped"},
@@ -105,7 +123,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "An escaped newline\\\noutside quotes",
+			input: []byte("An escaped newline\\\noutside quotes"),
 			expected: []Token{
 				{Line: 1, Text: "An"},
 				{Line: 1, Text: "escaped"},
@@ -115,7 +133,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "line1\\\nescaped\nline2\nline3",
+			input: []byte("line1\\\nescaped\nline2\nline3"),
 			expected: []Token{
 				{Line: 1, Text: "line1"},
 				{Line: 1, Text: "escaped"},
@@ -124,7 +142,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "line1\\\nescaped1\\\nescaped2\nline4\nline5",
+			input: []byte("line1\\\nescaped1\\\nescaped2\nline4\nline5"),
 			expected: []Token{
 				{Line: 1, Text: "line1"},
 				{Line: 1, Text: "escaped1"},
@@ -134,34 +152,34 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `"unescapable\ in quotes"`,
+			input: []byte(`"unescapable\ in quotes"`),
 			expected: []Token{
 				{Line: 1, Text: `unescapable\ in quotes`},
 			},
 		},
 		{
-			input: `"don't\escape"`,
+			input: []byte(`"don't\escape"`),
 			expected: []Token{
 				{Line: 1, Text: `don't\escape`},
 			},
 		},
 		{
-			input: `"don't\\escape"`,
+			input: []byte(`"don't\\escape"`),
 			expected: []Token{
 				{Line: 1, Text: `don't\\escape`},
 			},
 		},
 		{
-			input: `un\escapable`,
+			input: []byte(`un\escapable`),
 			expected: []Token{
 				{Line: 1, Text: `un\escapable`},
 			},
 		},
 		{
-			input: `A "quoted value with line
+			input: []byte(`A "quoted value with line
 					break inside" {
 						foobar
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "A"},
 				{Line: 1, Text: "quoted value with line\n\t\t\t\t\tbreak inside"},
@@ -171,13 +189,13 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `"C:\php\php-cgi.exe"`,
+			input: []byte(`"C:\php\php-cgi.exe"`),
 			expected: []Token{
 				{Line: 1, Text: `C:\php\php-cgi.exe`},
 			},
 		},
 		{
-			input: `empty "" string`,
+			input: []byte(`empty "" string`),
 			expected: []Token{
 				{Line: 1, Text: `empty`},
 				{Line: 1, Text: ``},
@@ -185,7 +203,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "skip those\r\nCR characters",
+			input: []byte("skip those\r\nCR characters"),
 			expected: []Token{
 				{Line: 1, Text: "skip"},
 				{Line: 1, Text: "those"},
@@ -194,30 +212,54 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "\xEF\xBB\xBF:8080", // test with leading byte order mark
+			input: []byte("\xEF\xBB\xBF:8080"), // test with leading byte order mark
 			expected: []Token{
 				{Line: 1, Text: ":8080"},
 			},
 		},
+		{
+			input: []byte("simple `backtick quoted` string"),
+			expected: []Token{
+				{Line: 1, Text: `simple`},
+				{Line: 1, Text: `backtick quoted`},
+				{Line: 1, Text: `string`},
+			},
+		},
+		{
+			input: []byte("multiline `backtick\nquoted\n` string"),
+			expected: []Token{
+				{Line: 1, Text: `multiline`},
+				{Line: 1, Text: "backtick\nquoted\n"},
+				{Line: 3, Text: `string`},
+			},
+		},
+		{
+			input: []byte("nested `\"quotes inside\" backticks` string"),
+			expected: []Token{
+				{Line: 1, Text: `nested`},
+				{Line: 1, Text: `"quotes inside" backticks`},
+				{Line: 1, Text: `string`},
+			},
+		},
+		{
+			input: []byte("reverse-nested \"`backticks` inside\" quotes"),
+			expected: []Token{
+				{Line: 1, Text: `reverse-nested`},
+				{Line: 1, Text: "`backticks` inside"},
+				{Line: 1, Text: `quotes`},
+			},
+		},
 	}
 
 	for i, testCase := range testCases {
-		actual := tokenize(testCase.input)
+		actual, err := Tokenize(testCase.input, "")
+		if err != nil {
+			t.Errorf("%v", err)
+		}
 		lexerCompare(t, i, testCase.expected, actual)
 	}
 }
 
-func tokenize(input string) (tokens []Token) {
-	l := lexer{}
-	if err := l.load(strings.NewReader(input)); err != nil {
-		log.Printf("[ERROR] load failed: %v", err)
-	}
-	for l.next() {
-		tokens = append(tokens, l.token)
-	}
-	return
-}
-
 func lexerCompare(t *testing.T, n int, expected, actual []Token) {
 	if len(expected) != len(actual) {
 		t.Errorf("Test case %d: expected %d token(s) but got %d", n, len(expected), len(actual))

caddyconfig/caddyfile/parse.go

@@ -16,11 +16,15 @@ package caddyfile
 
 import (
 	"bytes"
+	"fmt"
 	"io/ioutil"
 	"log"
 	"os"
 	"path/filepath"
+	"strconv"
 	"strings"
+
+	"github.com/caddyserver/caddy/v2"
 )
 
 // Parse parses the input just enough to group tokens, in
@@ -37,7 +41,13 @@ func Parse(filename string, input []byte) ([]ServerBlock, error) {
 	if err != nil {
 		return nil, err
 	}
-	p := parser{Dispenser: NewDispenser(tokens)}
+	p := parser{
+		Dispenser: NewDispenser(tokens),
+		importGraph: importGraph{
+			nodes: make(map[string]bool),
+			edges: make(adjacency),
+		},
+	}
 	return p.parseAll()
 }
 
@@ -57,21 +67,31 @@ func replaceEnvVars(input []byte) ([]byte, error) {
 		end += begin + len(spanOpen) // make end relative to input, not begin
 
 		// get the name; if there is no name, skip it
-		envVarName := input[begin+len(spanOpen) : end]
-		if len(envVarName) == 0 {
+		envString := input[begin+len(spanOpen) : end]
+		if len(envString) == 0 {
 			offset = end + len(spanClose)
 			continue
 		}
 
+		// split the string into a key and an optional default
+		envParts := strings.SplitN(string(envString), envVarDefaultDelimiter, 2)
+
+		// do a lookup for the env var, replace with the default if not found
+		envVarValue, found := os.LookupEnv(envParts[0])
+		if !found && len(envParts) == 2 {
+			envVarValue = envParts[1]
+		}
+
 		// get the value of the environment variable
-		envVarValue := []byte(os.ExpandEnv(os.Getenv(string(envVarName))))
+		// note that this causes one-level deep chaining
+		envVarBytes := []byte(envVarValue)
 
 		// splice in the value
 		input = append(input[:begin],
-			append(envVarValue, input[end+len(spanClose):]...)...)
+			append(envVarBytes, input[end+len(spanClose):]...)...)
 
 		// continue at the end of the replacement
-		offset = begin + len(envVarValue)
+		offset = begin + len(envVarBytes)
 	}
 	return input, nil
 }
@@ -84,16 +104,10 @@ func allTokens(filename string, input []byte) ([]Token, error) {
 	if err != nil {
 		return nil, err
 	}
-	l := new(lexer)
-	err = l.load(bytes.NewReader(input))
+	tokens, err := Tokenize(input, filename)
 	if err != nil {
 		return nil, err
 	}
-	var tokens []Token
-	for l.next() {
-		l.token.File = filename
-		tokens = append(tokens, l.token)
-	}
 	return tokens, nil
 }
 
@@ -103,6 +117,7 @@ type parser struct {
 	eof             bool        // if we encounter a valid EOF in a hard place
 	definedSnippets map[string][]Token
 	nesting         int
+	importGraph     importGraph
 }
 
 func (p *parser) parseAll() ([]ServerBlock, error) {
@@ -158,6 +173,15 @@ func (p *parser) begin() error {
 		if err != nil {
 			return err
 		}
+		// Just as we need to track which file the token comes from, we need to
+		// keep track of which snippets do the tokens come from. This is helpful
+		// in tracking import cycles across files/snippets by namespacing them. Without
+		// this we end up with false-positives in cycle-detection.
+		for k, v := range tokens {
+			v.inSnippet = true
+			v.snippetName = name
+			tokens[k] = v
+		}
 		p.definedSnippets[name] = tokens
 		// empty block keys so we don't save this block as a real server.
 		p.block.Keys = nil
@@ -292,17 +316,30 @@ func (p *parser) doImport() error {
 	if importPattern == "" {
 		return p.Err("Import requires a non-empty filepath")
 	}
-	if p.NextArg() {
-		return p.Err("Import takes only one argument (glob pattern or file)")
+
+	// grab remaining args as placeholder replacements
+	args := p.RemainingArgs()
+
+	// add args to the replacer
+	repl := caddy.NewEmptyReplacer()
+	for index, arg := range args {
+		repl.Set("args."+strconv.Itoa(index), arg)
 	}
-	// splice out the import directive and its argument (2 tokens total)
-	tokensBefore := p.tokens[:p.cursor-1]
+
+	// splice out the import directive and its arguments
+	// (2 tokens, plus the length of args)
+	tokensBefore := p.tokens[:p.cursor-1-len(args)]
 	tokensAfter := p.tokens[p.cursor+1:]
 	var importedTokens []Token
+	var nodes []string
 
 	// first check snippets. That is a simple, non-recursive replacement
 	if p.definedSnippets != nil && p.definedSnippets[importPattern] != nil {
 		importedTokens = p.definedSnippets[importPattern]
+		if len(importedTokens) > 0 {
+			// just grab the first one
+			nodes = append(nodes, fmt.Sprintf("%s:%s", importedTokens[0].File, importedTokens[0].snippetName))
+		}
 	} else {
 		// make path relative to the file of the _token_ being processed rather
 		// than current working directory (issue #867) and then use glob to get
@@ -338,7 +375,6 @@ func (p *parser) doImport() error {
 		}
 
 		// collect all the imported tokens
-
 		for _, importFile := range matches {
 			newTokens, err := p.doSingleImport(importFile)
 			if err != nil {
@@ -346,12 +382,34 @@ func (p *parser) doImport() error {
 			}
 			importedTokens = append(importedTokens, newTokens...)
 		}
+		nodes = matches
+	}
+
+	nodeName := p.File()
+	if p.Token().inSnippet {
+		nodeName += fmt.Sprintf(":%s", p.Token().snippetName)
+	}
+	p.importGraph.addNode(nodeName)
+	p.importGraph.addNodes(nodes)
+	if err := p.importGraph.addEdges(nodeName, nodes); err != nil {
+		p.importGraph.removeNodes(nodes)
+		return err
+	}
+
+	// copy the tokens so we don't overwrite p.definedSnippets
+	tokensCopy := make([]Token, len(importedTokens))
+	copy(tokensCopy, importedTokens)
+
+	// run the argument replacer on the tokens
+	for index, token := range tokensCopy {
+		token.Text = repl.ReplaceKnown(token.Text, "")
+		tokensCopy[index] = token
 	}
 
 	// splice the imported tokens in the place of the import statement
 	// and rewind cursor so Next() will land on first imported token
-	p.tokens = append(tokensBefore, append(importedTokens, tokensAfter...)...)
-	p.cursor--
+	p.tokens = append(tokensBefore, append(tokensCopy, tokensAfter...)...)
+	p.cursor -= len(args) + 1
 
 	return nil
 }
@@ -533,4 +591,7 @@ func (s Segment) Directive() string {
 
 // spanOpen and spanClose are used to bound spans that
 // contain the name of an environment variable.
-var spanOpen, spanClose = []byte{'{', '$'}, []byte{'}'}
+var (
+	spanOpen, spanClose    = []byte{'{', '$'}, []byte{'}'}
+	envVarDefaultDelimiter = ":"
+)

caddyconfig/caddyfile/parse_test.go

@@ -182,14 +182,17 @@ func TestParseOneAndImport(t *testing.T) {
 			"host1",
 		}, []int{1, 2}},
 
-		{`import testdata/import_test1.txt testdata/import_test2.txt`, true, []string{}, []int{}},
-
 		{`import testdata/not_found.txt`, true, []string{}, []int{}},
 
 		{`""`, false, []string{}, []int{}},
 
 		{``, false, []string{}, []int{}},
 
+		// import with args
+		{`import testdata/import_args0.txt a`, false, []string{"a"}, []int{}},
+		{`import testdata/import_args1.txt a b`, false, []string{"a", "b"}, []int{}},
+		{`import testdata/import_args*.txt a b`, false, []string{"a"}, []int{2}},
+
 		// test cases found by fuzzing!
 		{`import }{$"`, true, []string{}, []int{}},
 		{`import /*/*.txt`, true, []string{}, []int{}},
@@ -210,6 +213,7 @@ func TestParseOneAndImport(t *testing.T) {
 			t.Errorf("Test %d: Expected no error, but got: %v", i, err)
 		}
 
+		// t.Logf("%+v\n", result)
 		if len(result.Keys) != len(test.keys) {
 			t.Errorf("Test %d: Expected %d keys, got %d",
 				i, len(test.keys), len(result.Keys))
@@ -440,6 +444,28 @@ func TestParseAll(t *testing.T) {
 
 		{`import notfound/*`, false, [][]string{}},        // glob needn't error with no matches
 		{`import notfound/file.conf`, true, [][]string{}}, // but a specific file should
+
+		// recursive self-import
+		{`import testdata/import_recursive0.txt`, true, [][]string{}},
+		{`import testdata/import_recursive3.txt
+		import testdata/import_recursive1.txt`, true, [][]string{}},
+
+		// cyclic imports
+		{`(A) {
+			import A
+		}
+		:80
+		import A
+		`, true, [][]string{}},
+		{`(A) {
+			import B
+		}
+		(B) {
+			import A
+		}
+		:80
+		import A
+		`, true, [][]string{}},
 	} {
 		p := testParser(test.input)
 		blocks, err := p.parseAll()
@@ -474,6 +500,7 @@ func TestParseAll(t *testing.T) {
 
 func TestEnvironmentReplacement(t *testing.T) {
 	os.Setenv("FOOBAR", "foobar")
+	os.Setenv("CHAINED", "$FOOBAR")
 
 	for i, test := range []struct {
 		input  string
@@ -519,6 +546,22 @@ func TestEnvironmentReplacement(t *testing.T) {
 			input:  "{$FOOBAR}{$FOOBAR}",
 			expect: "foobarfoobar",
 		},
+		{
+			input:  "{$CHAINED}",
+			expect: "$FOOBAR", // should not chain env expands
+		},
+		{
+			input:  "{$FOO:default}",
+			expect: "default",
+		},
+		{
+			input:  "foo{$BAR:bar}baz",
+			expect: "foobarbaz",
+		},
+		{
+			input:  "foo{$BAR:$FOOBAR}baz",
+			expect: "foo$FOOBARbaz", // should not chain env expands
+		},
 		{
 			input:  "{$FOOBAR",
 			expect: "{$FOOBAR",
@@ -568,10 +611,6 @@ func TestSnippets(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	if len(blocks) != 1 {
 		t.Fatalf("Expect exactly one server block. Got %d.", len(blocks))
 	}
@@ -616,10 +655,6 @@ func TestImportedFilesIgnoreNonDirectiveImportTokens(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	auth := blocks[0].Segments[0]
 	line := auth[0].Text + " " + auth[1].Text + " " + auth[2].Text + " " + auth[3].Text
 	if line != "basicauth / import password" {
@@ -651,10 +686,6 @@ func TestSnippetAcrossMultipleFiles(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	if len(blocks) != 1 {
 		t.Fatalf("Expect exactly one server block. Got %d.", len(blocks))
 	}

caddyconfig/caddyfile/testdata/import_args0.txt

@@ -0,0 +1 @@
+{args.0}

caddyconfig/caddyfile/testdata/import_args1.txt

@@ -0,0 +1 @@
+{args.0} {args.1}

caddyconfig/caddyfile/testdata/import_recursive0.txt

@@ -0,0 +1 @@
+import import_recursive0.txt

caddyconfig/caddyfile/testdata/import_recursive1.txt

@@ -0,0 +1 @@
+import import_recursive2.txt

caddyconfig/caddyfile/testdata/import_recursive2.txt

@@ -0,0 +1 @@
+import import_recursive3.txt

caddyconfig/caddyfile/testdata/import_recursive3.txt

@@ -0,0 +1 @@
+import import_recursive1.txt

caddyconfig/configadapters.go

@@ -35,6 +35,14 @@ type Warning struct {
 	Message   string `json:"message,omitempty"`
 }
 
+func (w Warning) String() string {
+	var directive string
+	if w.Directive != "" {
+		directive = fmt.Sprintf(" (%s)", w.Directive)
+	}
+	return fmt.Sprintf("%s:%d%s: %s", w.File, w.Line, directive, w.Message)
+}
+
 // JSON encodes val as JSON, returning it as a json.RawMessage. Any
 // marshaling errors (which are highly unlikely with correct code)
 // are converted to warnings. This is convenient when filling config
@@ -51,14 +59,13 @@ func JSON(val interface{}, warnings *[]Warning) json.RawMessage {
 	return b
 }
 
-// JSONModuleObject is like JSON, except it marshals val into a JSON object
-// and then adds a key to that object named fieldName with the value fieldVal.
-// This is useful for JSON-encoding module values where the module name has to
-// be described within the object by a certain key; for example,
-// "responder": "file_server" for a file server HTTP responder. The val must
-// encode into a map[string]interface{} (i.e. it must be a struct or map),
-// and any errors are converted into warnings, so this can be conveniently
-// used when filling a struct. For correct code, there should be no errors.
+// JSONModuleObject is like JSON(), except it marshals val into a JSON object
+// with an added key named fieldName with the value fieldVal. This is useful
+// for encoding module values where the module name has to be described within
+// the object by a certain key; for example, `"handler": "file_server"` for a
+// file server HTTP handler (fieldName="handler" and fieldVal="file_server").
+// The val parameter must encode into a map[string]interface{} (i.e. it must be
+// a struct or map). Any errors are converted into warnings.
 func JSONModuleObject(val interface{}, fieldName, fieldVal string, warnings *[]Warning) json.RawMessage {
 	// encode to a JSON object first
 	enc, err := json.Marshal(val)
@@ -94,20 +101,15 @@ func JSONModuleObject(val interface{}, fieldName, fieldVal string, warnings *[]W
 	return result
 }
 
-// JSONIndent is used to JSON-marshal the final resulting Caddy
-// configuration in a consistent, human-readable way.
-func JSONIndent(val interface{}) ([]byte, error) {
-	return json.MarshalIndent(val, "", "\t")
-}
-
 // RegisterAdapter registers a config adapter with the given name.
-// This should usually be done at init-time.
-func RegisterAdapter(name string, adapter Adapter) error {
+// This should usually be done at init-time. It panics if the
+// adapter cannot be registered successfully.
+func RegisterAdapter(name string, adapter Adapter) {
 	if _, ok := configAdapters[name]; ok {
-		return fmt.Errorf("%s: already registered", name)
+		panic(fmt.Errorf("%s: already registered", name))
 	}
 	configAdapters[name] = adapter
-	return caddy.RegisterModule(adapterModule{name, adapter})
+	caddy.RegisterModule(adapterModule{name, adapter})
 }
 
 // GetAdapter returns the adapter with the given name,

caddyconfig/httpcaddyfile/addresses.go

@@ -18,8 +18,10 @@ import (
 	"fmt"
 	"net"
 	"reflect"
+	"sort"
 	"strconv"
 	"strings"
+	"unicode"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
@@ -138,7 +140,7 @@ func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBloc
 // association from multiple addresses to multiple server blocks; i.e. each element of
 // the returned slice) becomes a server definition in the output JSON.
 func (st *ServerType) consolidateAddrMappings(addrToServerBlocks map[string][]serverBlock) []sbAddrAssociation {
-	var sbaddrs []sbAddrAssociation
+	sbaddrs := make([]sbAddrAssociation, 0, len(addrToServerBlocks))
 	for addr, sblocks := range addrToServerBlocks {
 		// we start with knowing that at least this address
 		// maps to these server blocks
@@ -162,6 +164,13 @@ func (st *ServerType) consolidateAddrMappings(addrToServerBlocks map[string][]se
 
 		sbaddrs = append(sbaddrs, a)
 	}
+
+	// sort them by their first address (we know there will always be at least one)
+	// to avoid problems with non-deterministic ordering (makes tests flaky)
+	sort.Slice(sbaddrs, func(i, j int) bool {
+		return sbaddrs[i].addresses[0] < sbaddrs[j].addresses[0]
+	})
+
 	return sbaddrs
 }
 
@@ -199,7 +208,7 @@ func (st *ServerType) listenerAddrsForServerBlockKey(sblock serverBlock, key str
 	}
 
 	// the bind directive specifies hosts, but is optional
-	var lnHosts []string
+	lnHosts := make([]string, 0, len(sblock.pile))
 	for _, cfgVal := range sblock.pile["bind"] {
 		lnHosts = append(lnHosts, cfgVal.Value.([]string)...)
 	}
@@ -219,7 +228,7 @@ func (st *ServerType) listenerAddrsForServerBlockKey(sblock serverBlock, key str
 	}
 
 	// now turn map into list
-	var listenersList []string
+	listenersList := make([]string, 0, len(listeners))
 	for lnStr := range listeners {
 		listenersList = append(listenersList, lnStr)
 	}
@@ -333,8 +342,8 @@ func (a Address) Normalize() Address {
 
 	return Address{
 		Original: a.Original,
-		Scheme:   strings.ToLower(a.Scheme),
-		Host:     strings.ToLower(host),
+		Scheme:   lowerExceptPlaceholders(a.Scheme),
+		Host:     lowerExceptPlaceholders(host),
 		Port:     a.Port,
 		Path:     path,
 	}
@@ -361,3 +370,31 @@ func (a Address) Key() string {
 	}
 	return res
 }
+
+// lowerExceptPlaceholders lowercases s except within
+// placeholders (substrings in non-escaped '{ }' spans).
+// See https://github.com/caddyserver/caddy/issues/3264
+func lowerExceptPlaceholders(s string) string {
+	var sb strings.Builder
+	var escaped, inPlaceholder bool
+	for _, ch := range s {
+		if ch == '\\' && !escaped {
+			escaped = true
+			sb.WriteRune(ch)
+			continue
+		}
+		if ch == '{' && !escaped {
+			inPlaceholder = true
+		}
+		if ch == '}' && inPlaceholder && !escaped {
+			inPlaceholder = false
+		}
+		if inPlaceholder {
+			sb.WriteRune(ch)
+		} else {
+			sb.WriteRune(unicode.ToLower(ch))
+		}
+		escaped = false
+	}
+	return sb.String()
+}

caddyconfig/httpcaddyfile/addresses_test.go

@@ -108,6 +108,10 @@ func TestKeyNormalization(t *testing.T) {
 		input  string
 		expect string
 	}{
+		{
+			input:  "example.com",
+			expect: "example.com",
+		},
 		{
 			input:  "http://host:1234/path",
 			expect: "http://host:1234/path",
@@ -124,6 +128,22 @@ func TestKeyNormalization(t *testing.T) {
 			input:  "A:2015/Path",
 			expect: "a:2015/Path",
 		},
+		{
+			input:  "sub.{env.MY_DOMAIN}",
+			expect: "sub.{env.MY_DOMAIN}",
+		},
+		{
+			input:  "sub.ExAmPle",
+			expect: "sub.example",
+		},
+		{
+			input:  "sub.\\{env.MY_DOMAIN\\}",
+			expect: "sub.\\{env.my_domain\\}",
+		},
+		{
+			input:  "sub.{env.MY_DOMAIN}.com",
+			expect: "sub.{env.MY_DOMAIN}.com",
+		},
 		{
 			input:  ":80",
 			expect: ":80",
@@ -156,7 +176,7 @@ func TestKeyNormalization(t *testing.T) {
 			continue
 		}
 		if actual := addr.Normalize().Key(); actual != tc.expect {
-			t.Errorf("Test %d: Normalized key for address '%s' was '%s' but expected '%s'", i, tc.input, actual, tc.expect)
+			t.Errorf("Test %d: Input '%s': Expected '%s' but got '%s'", i, tc.input, tc.expect, actual)
 		}
 
 	}

caddyconfig/httpcaddyfile/builtins.go

@@ -15,10 +15,14 @@
 package httpcaddyfile
 
 import (
+	"encoding/base64"
+	"encoding/pem"
 	"fmt"
 	"html"
+	"io/ioutil"
 	"net/http"
 	"reflect"
+	"strconv"
 	"strings"
 
 	"github.com/caddyserver/caddy/v2"
@@ -26,6 +30,8 @@ import (
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
 	"go.uber.org/zap/zapcore"
 )
 
@@ -35,6 +41,8 @@ func init() {
 	RegisterHandlerDirective("root", parseRoot)
 	RegisterHandlerDirective("redir", parseRedir)
 	RegisterHandlerDirective("respond", parseRespond)
+	RegisterHandlerDirective("abort", parseAbort)
+	RegisterHandlerDirective("error", parseError)
 	RegisterHandlerDirective("route", parseRoute)
 	RegisterHandlerDirective("handle", parseHandle)
 	RegisterDirective("handle_errors", parseHandleErrors)
@@ -59,12 +67,21 @@ func parseBind(h Helper) ([]ConfigValue, error) {
 //         protocols <min> [<max>]
 //         ciphers   <cipher_suites...>
 //         curves    <curves...>
+//         client_auth {
+//             mode                   [request|require|verify_if_given|require_and_verify]
+//             trusted_ca_cert        <base64_der>
+//             trusted_ca_cert_file   <filename>
+//             trusted_leaf_cert      <base64_der>
+//             trusted_leaf_cert_file <filename>
+//         }
 //         alpn      <values...>
 //         load      <paths...>
 //         ca        <acme_ca_endpoint>
 //         ca_root   <pem_file>
-//         dns       <provider_name>
+//         dns       <provider_name> [...]
 //         on_demand
+//         eab    <key_id> <mac_key>
+//         issuer <module_name> [...]
 //     }
 //
 func parseTLS(h Helper) ([]ConfigValue, error) {
@@ -73,7 +90,9 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 	var folderLoader caddytls.FolderLoader
 	var certSelector caddytls.CustomCertSelectionPolicy
 	var acmeIssuer *caddytls.ACMEIssuer
+	var keyType string
 	var internalIssuer *caddytls.InternalIssuer
+	var issuers []certmagic.Issuer
 	var onDemand bool
 
 	for h.Next() {
@@ -107,10 +126,10 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 			// must load each cert only once; otherwise, they each get a
 			// different tag... since a cert loaded twice has the same
 			// bytes, it will overwrite the first one in the cache, and
-			// only the last cert (and its tag) will survive, so a any conn
-			// policy that is looking for any tag but the last one to be
-			// loaded won't find it, and TLS handshakes will fail (see end)
-			// of issue #3004)
+			// only the last cert (and its tag) will survive, so any conn
+			// policy that is looking for any tag other than the last one
+			// to be loaded won't find it, and TLS handshakes will fail
+			// (see end of issue #3004)
 			//
 			// tlsCertTags maps certificate filenames to their tag.
 			// This is used to remember which tag is used for each
@@ -143,7 +162,7 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 		}
 
 		var hasBlock bool
-		for h.NextBlock(0) {
+		for nesting := h.Nesting(); h.NextBlock(nesting); {
 			hasBlock = true
 
 			switch h.Val() {
@@ -181,6 +200,57 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 					cp.Curves = append(cp.Curves, h.Val())
 				}
 
+			case "client_auth":
+				cp.ClientAuthentication = &caddytls.ClientAuthentication{}
+				for nesting := h.Nesting(); h.NextBlock(nesting); {
+					subdir := h.Val()
+					switch subdir {
+					case "mode":
+						if !h.Args(&cp.ClientAuthentication.Mode) {
+							return nil, h.ArgErr()
+						}
+						if h.NextArg() {
+							return nil, h.ArgErr()
+						}
+
+					case "trusted_ca_cert",
+						"trusted_leaf_cert":
+						if !h.NextArg() {
+							return nil, h.ArgErr()
+						}
+						if subdir == "trusted_ca_cert" {
+							cp.ClientAuthentication.TrustedCACerts = append(cp.ClientAuthentication.TrustedCACerts, h.Val())
+						} else {
+							cp.ClientAuthentication.TrustedLeafCerts = append(cp.ClientAuthentication.TrustedLeafCerts, h.Val())
+						}
+
+					case "trusted_ca_cert_file",
+						"trusted_leaf_cert_file":
+						if !h.NextArg() {
+							return nil, h.ArgErr()
+						}
+						filename := h.Val()
+						certDataPEM, err := ioutil.ReadFile(filename)
+						if err != nil {
+							return nil, err
+						}
+						block, _ := pem.Decode(certDataPEM)
+						if block == nil || block.Type != "CERTIFICATE" {
+							return nil, h.Errf("no CERTIFICATE pem block found in %s", h.Val())
+						}
+						if subdir == "trusted_ca_cert_file" {
+							cp.ClientAuthentication.TrustedCACerts = append(cp.ClientAuthentication.TrustedCACerts,
+								base64.StdEncoding.EncodeToString(block.Bytes))
+						} else {
+							cp.ClientAuthentication.TrustedLeafCerts = append(cp.ClientAuthentication.TrustedLeafCerts,
+								base64.StdEncoding.EncodeToString(block.Bytes))
+						}
+
+					default:
+						return nil, h.Errf("unknown subdirective for client_auth: %s", subdir)
+					}
+				}
+
 			case "alpn":
 				args := h.RemainingArgs()
 				if len(args) == 0 {
@@ -201,22 +271,78 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 				}
 				acmeIssuer.CA = arg[0]
 
-			case "dns":
-				if !h.Next() {
+			case "key_type":
+				arg := h.RemainingArgs()
+				if len(arg) != 1 {
+					return nil, h.ArgErr()
+				}
+				keyType = arg[0]
+
+			case "eab":
+				arg := h.RemainingArgs()
+				if len(arg) != 2 {
 					return nil, h.ArgErr()
 				}
 				if acmeIssuer == nil {
 					acmeIssuer = new(caddytls.ACMEIssuer)
 				}
+				acmeIssuer.ExternalAccount = &acme.EAB{
+					KeyID:  arg[0],
+					MACKey: arg[1],
+				}
+
+			case "issuer":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				modName := h.Val()
+				modID := "tls.issuance." + modName
+				unm, err := caddyfile.UnmarshalModule(h.Dispenser, modID)
+				if err != nil {
+					return nil, err
+				}
+				issuer, ok := unm.(certmagic.Issuer)
+				if !ok {
+					return nil, h.Errf("module %s (%T) is not a certmagic.Issuer", modID, unm)
+				}
+				issuers = append(issuers, issuer)
+
+			case "dns":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
 				provName := h.Val()
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
 				if acmeIssuer.Challenges == nil {
 					acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
 				}
-				dnsProvModule, err := caddy.GetModule("tls.dns." + provName)
-				if err != nil {
-					return nil, h.Errf("getting DNS provider module named '%s': %v", provName, err)
+				if acmeIssuer.Challenges.DNS == nil {
+					acmeIssuer.Challenges.DNS = new(caddytls.DNSChallengeConfig)
 				}
-				acmeIssuer.Challenges.DNSRaw = caddyconfig.JSONModuleObject(dnsProvModule.New(), "provider", provName, h.warnings)
+				modID := "dns.providers." + provName
+				unm, err := caddyfile.UnmarshalModule(h.Dispenser, modID)
+				if err != nil {
+					return nil, err
+				}
+				acmeIssuer.Challenges.DNS.ProviderRaw = caddyconfig.JSONModuleObject(unm, "name", provName, h.warnings)
+
+			case "resolvers":
+				args := h.RemainingArgs()
+				if len(args) == 0 {
+					return nil, h.ArgErr()
+				}
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
+				if acmeIssuer.Challenges == nil {
+					acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
+				}
+				if acmeIssuer.Challenges.DNS == nil {
+					acmeIssuer.Challenges.DNS = new(caddytls.DNSChallengeConfig)
+				}
+				acmeIssuer.Challenges.DNS.Resolvers = args
 
 			case "ca_root":
 				arg := h.RemainingArgs()
@@ -246,50 +372,80 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 	}
 
 	// begin building the final config values
-	var configVals []ConfigValue
+	configVals := []ConfigValue{}
 
 	// certificate loaders
 	if len(fileLoader) > 0 {
 		configVals = append(configVals, ConfigValue{
-			Class: "tls.certificate_loader",
+			Class: "tls.cert_loader",
 			Value: fileLoader,
 		})
 	}
 	if len(folderLoader) > 0 {
 		configVals = append(configVals, ConfigValue{
-			Class: "tls.certificate_loader",
+			Class: "tls.cert_loader",
 			Value: folderLoader,
 		})
 	}
 
-	// issuer
-	if acmeIssuer != nil && internalIssuer != nil {
-		// the logic to support this would be complex
-		return nil, h.Err("cannot use both ACME and internal issuers in same server block")
+	// some tls subdirectives are shortcuts that implicitly configure issuers, and the
+	// user can also configure issuers explicitly using the issuer subdirective; the
+	// logic to support both would likely be complex, or at least unintuitive
+	if len(issuers) > 0 && (acmeIssuer != nil || internalIssuer != nil) {
+		return nil, h.Err("cannot mix issuer subdirective (explicit issuers) with other issuer-specific subdirectives (implicit issuers)")
 	}
-	if acmeIssuer != nil {
-		// fill in global defaults, if configured
-		if email := h.Option("email"); email != nil && acmeIssuer.Email == "" {
-			acmeIssuer.Email = email.(string)
-		}
-		if acmeCA := h.Option("acme_ca"); acmeCA != nil && acmeIssuer.CA == "" {
-			acmeIssuer.CA = acmeCA.(string)
-		}
-		if caPemFile := h.Option("acme_ca_root"); caPemFile != nil {
-			acmeIssuer.TrustedRootsPEMFiles = append(acmeIssuer.TrustedRootsPEMFiles, caPemFile.(string))
+	if acmeIssuer != nil && internalIssuer != nil {
+		return nil, h.Err("cannot create both ACME and internal certificate issuers")
+	}
+
+	// now we should either have: explicitly-created issuers, or an implicitly-created
+	// ACME or internal issuer, or no issuers at all
+	switch {
+	case len(issuers) > 0:
+		for _, issuer := range issuers {
+			configVals = append(configVals, ConfigValue{
+				Class: "tls.cert_issuer",
+				Value: issuer,
+			})
 		}
 
-		configVals = append(configVals, ConfigValue{
-			Class: "tls.cert_issuer",
-			Value: acmeIssuer,
-		})
-	} else if internalIssuer != nil {
+	case acmeIssuer != nil:
+		// implicit ACME issuers (from various subdirectives) - use defaults; there might be more than one
+		defaultIssuers := caddytls.DefaultIssuers()
+
+		// if a CA endpoint was set, override multiple implicit issuers since it's a specific one
+		if acmeIssuer.CA != "" {
+			defaultIssuers = []certmagic.Issuer{acmeIssuer}
+		}
+
+		for _, issuer := range defaultIssuers {
+			switch iss := issuer.(type) {
+			case *caddytls.ACMEIssuer:
+				issuer = acmeIssuer
+			case *caddytls.ZeroSSLIssuer:
+				iss.ACMEIssuer = acmeIssuer
+			}
+			configVals = append(configVals, ConfigValue{
+				Class: "tls.cert_issuer",
+				Value: issuer,
+			})
+		}
+
+	case internalIssuer != nil:
 		configVals = append(configVals, ConfigValue{
 			Class: "tls.cert_issuer",
 			Value: internalIssuer,
 		})
 	}
 
+	// certificate key type
+	if keyType != "" {
+		configVals = append(configVals, ConfigValue{
+			Class: "tls.key_type",
+			Value: keyType,
+		})
+	}
+
 	// on-demand TLS
 	if onDemand {
 		configVals = append(configVals, ConfigValue{
@@ -352,14 +508,14 @@ func parseRedir(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	if h.NextArg() {
 		code = h.Val()
 	}
-	if code == "permanent" {
-		code = "301"
-	}
-	if code == "temporary" || code == "" {
-		code = "302"
-	}
+
 	var body string
-	if code == "html" {
+	switch code {
+	case "permanent":
+		code = "301"
+	case "temporary", "":
+		code = "302"
+	case "html":
 		// Script tag comes first since that will better imitate a redirect in the browser's
 		// history, but the meta tag is a fallback for most non-JS clients.
 		const metaRedir = `<!DOCTYPE html>
@@ -374,6 +530,15 @@ func parseRedir(h Helper) (caddyhttp.MiddlewareHandler, error) {
 `
 		safeTo := html.EscapeString(to)
 		body = fmt.Sprintf(metaRedir, safeTo, safeTo, safeTo, safeTo)
+		code = "302"
+	default:
+		codeInt, err := strconv.Atoi(code)
+		if err != nil {
+			return nil, h.Errf("Not a supported redir code type or not valid integer: '%s'", code)
+		}
+		if codeInt < 300 || codeInt > 399 {
+			return nil, h.Errf("Redir code not in the 3xx range: '%v'", codeInt)
+		}
 	}
 
 	return caddyhttp.StaticResponse{
@@ -393,40 +558,46 @@ func parseRespond(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	return sr, nil
 }
 
+// parseAbort parses the abort directive.
+func parseAbort(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	h.Next() // consume directive
+	for h.Next() || h.NextBlock(0) {
+		return nil, h.ArgErr()
+	}
+	return &caddyhttp.StaticResponse{Abort: true}, nil
+}
+
+// parseError parses the error directive.
+func parseError(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	se := new(caddyhttp.StaticError)
+	err := se.UnmarshalCaddyfile(h.Dispenser)
+	if err != nil {
+		return nil, err
+	}
+	return se, nil
+}
+
 // parseRoute parses the route directive.
 func parseRoute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	sr := new(caddyhttp.Subroute)
 
-	for h.Next() {
-		for nesting := h.Nesting(); h.NextBlock(nesting); {
-			dir := h.Val()
+	allResults, err := parseSegmentAsConfig(h)
+	if err != nil {
+		return nil, err
+	}
 
-			dirFunc, ok := registeredDirectives[dir]
-			if !ok {
-				return nil, h.Errf("unrecognized directive: %s", dir)
-			}
-
-			subHelper := h
-			subHelper.Dispenser = h.NewFromNextSegment()
-
-			results, err := dirFunc(subHelper)
-			if err != nil {
-				return nil, h.Errf("parsing caddyfile tokens for '%s': %v", dir, err)
-			}
-			for _, result := range results {
-				switch handler := result.Value.(type) {
-				case caddyhttp.Route:
-					sr.Routes = append(sr.Routes, handler)
-				case caddyhttp.Subroute:
-					// directives which return a literal subroute instead of a route
-					// means they intend to keep those handlers together without
-					// them being reordered; we're doing that anyway since we're in
-					// the route directive, so just append its handlers
-					sr.Routes = append(sr.Routes, handler.Routes...)
-				default:
-					return nil, h.Errf("%s directive returned something other than an HTTP route or subroute: %#v (only handler directives can be used in routes)", dir, result.Value)
-				}
-			}
+	for _, result := range allResults {
+		switch handler := result.Value.(type) {
+		case caddyhttp.Route:
+			sr.Routes = append(sr.Routes, handler)
+		case caddyhttp.Subroute:
+			// directives which return a literal subroute instead of a route
+			// means they intend to keep those handlers together without
+			// them being reordered; we're doing that anyway since we're in
+			// the route directive, so just append its handlers
+			sr.Routes = append(sr.Routes, handler.Routes...)
+		default:
+			return nil, h.Errf("%s directive returned something other than an HTTP route or subroute: %#v (only handler directives can be used in routes)", result.directive, result.Value)
 		}
 	}
 
@@ -434,11 +605,11 @@ func parseRoute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 }
 
 func parseHandle(h Helper) (caddyhttp.MiddlewareHandler, error) {
-	return parseSegmentAsSubroute(h)
+	return ParseSegmentAsSubroute(h)
 }
 
 func parseHandleErrors(h Helper) ([]ConfigValue, error) {
-	subroute, err := parseSegmentAsSubroute(h)
+	subroute, err := ParseSegmentAsSubroute(h)
 	if err != nil {
 		return nil, err
 	}
@@ -459,8 +630,52 @@ func parseHandleErrors(h Helper) ([]ConfigValue, error) {
 //     }
 //
 func parseLog(h Helper) ([]ConfigValue, error) {
+	return parseLogHelper(h, nil)
+}
+
+// parseLogHelper is used both for the parseLog directive within Server Blocks,
+// as well as the global "log" option for configuring loggers at the global
+// level. The parseAsGlobalOption parameter is used to distinguish any differing logic
+// between the two.
+func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue, error) {
+	// When the globalLogNames parameter is passed in, we make
+	// modifications to the parsing behavior.
+	parseAsGlobalOption := globalLogNames != nil
+
 	var configValues []ConfigValue
 	for h.Next() {
+		// Logic below expects that a name is always present when a
+		// global option is being parsed.
+		var globalLogName string
+		if parseAsGlobalOption {
+			if h.NextArg() {
+				globalLogName = h.Val()
+
+				// Only a single argument is supported.
+				if h.NextArg() {
+					return nil, h.ArgErr()
+				}
+			} else {
+				// If there is no log name specified, we
+				// reference the default logger. See the
+				// setupNewDefault function in the logging
+				// package for where this is configured.
+				globalLogName = "default"
+			}
+
+			// Verify this name is unused.
+			_, used := globalLogNames[globalLogName]
+			if used {
+				return nil, h.Err("duplicate global log option for: " + globalLogName)
+			}
+			globalLogNames[globalLogName] = struct{}{}
+		} else {
+			// No arguments are supported for the server block log directive
+			if h.NextArg() {
+				return nil, h.ArgErr()
+			}
+		}
+
 		cl := new(caddy.CustomLog)
 
 		for h.NextBlock(0) {
@@ -484,21 +699,15 @@ func parseLog(h Helper) ([]ConfigValue, error) {
 				case "discard":
 					wo = caddy.DiscardWriter{}
 				default:
-					mod, err := caddy.GetModule("caddy.logging.writers." + moduleName)
-					if err != nil {
-						return nil, h.Errf("getting log writer module named '%s': %v", moduleName, err)
-					}
-					unm, ok := mod.New().(caddyfile.Unmarshaler)
-					if !ok {
-						return nil, h.Errf("log writer module '%s' is not a Caddyfile unmarshaler", mod)
-					}
-					err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+					modID := "caddy.logging.writers." + moduleName
+					unm, err := caddyfile.UnmarshalModule(h.Dispenser, modID)
 					if err != nil {
 						return nil, err
 					}
+					var ok bool
 					wo, ok = unm.(caddy.WriterOpener)
 					if !ok {
-						return nil, h.Errf("module %s is not a WriterOpener", mod)
+						return nil, h.Errf("module %s (%T) is not a WriterOpener", modID, unm)
 					}
 				}
 				cl.WriterRaw = caddyconfig.JSONModuleObject(wo, "output", moduleName, h.warnings)
@@ -508,21 +717,14 @@ func parseLog(h Helper) ([]ConfigValue, error) {
 					return nil, h.ArgErr()
 				}
 				moduleName := h.Val()
-				mod, err := caddy.GetModule("caddy.logging.encoders." + moduleName)
-				if err != nil {
-					return nil, h.Errf("getting log encoder module named '%s': %v", moduleName, err)
-				}
-				unm, ok := mod.New().(caddyfile.Unmarshaler)
-				if !ok {
-					return nil, h.Errf("log encoder module '%s' is not a Caddyfile unmarshaler", mod)
-				}
-				err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+				moduleID := "caddy.logging.encoders." + moduleName
+				unm, err := caddyfile.UnmarshalModule(h.Dispenser, moduleID)
 				if err != nil {
 					return nil, err
 				}
 				enc, ok := unm.(zapcore.Encoder)
 				if !ok {
-					return nil, h.Errf("module %s is not a zapcore.Encoder", mod)
+					return nil, h.Errf("module %s (%T) is not a zapcore.Encoder", moduleID, unm)
 				}
 				cl.EncoderRaw = caddyconfig.JSONModuleObject(enc, "format", moduleName, h.warnings)
 
@@ -535,22 +737,48 @@ func parseLog(h Helper) ([]ConfigValue, error) {
 					return nil, h.ArgErr()
 				}
 
+			case "include":
+				// This configuration is only allowed in the global options
+				if !parseAsGlobalOption {
+					return nil, h.ArgErr()
+				}
+				for h.NextArg() {
+					cl.Include = append(cl.Include, h.Val())
+				}
+
+			case "exclude":
+				// This configuration is only allowed in the global options
+				if !parseAsGlobalOption {
+					return nil, h.ArgErr()
+				}
+				for h.NextArg() {
+					cl.Exclude = append(cl.Exclude, h.Val())
+				}
+
 			default:
 				return nil, h.Errf("unrecognized subdirective: %s", h.Val())
 			}
 		}
 
 		var val namedCustomLog
+		// Skip handling of empty logging configs
 		if !reflect.DeepEqual(cl, new(caddy.CustomLog)) {
-			logCounter, ok := h.State["logCounter"].(int)
-			if !ok {
-				logCounter = 0
+			if parseAsGlobalOption {
+				// Use indicated name for global log options
+				val.name = globalLogName
+				val.log = cl
+			} else {
+				// Construct a log name for server log streams
+				logCounter, ok := h.State["logCounter"].(int)
+				if !ok {
+					logCounter = 0
+				}
+				val.name = fmt.Sprintf("log%d", logCounter)
+				cl.Include = []string{"http.log.access." + val.name}
+				val.log = cl
+				logCounter++
+				h.State["logCounter"] = logCounter
 			}
-			val.name = fmt.Sprintf("log%d", logCounter)
-			cl.Include = []string{"http.log.access." + val.name}
-			val.log = cl
-			logCounter++
-			h.State["logCounter"] = logCounter
 		}
 		configValues = append(configValues, ConfigValue{
 			Class: "custom_log",

caddyconfig/httpcaddyfile/builtins_test.go

@@ -0,0 +1,201 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	_ "github.com/caddyserver/caddy/v2/modules/logging"
+)
+
+func TestLogDirectiveSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		output      string
+		expectError bool
+	}{
+		{
+			input: `:8080 {
+				log
+			}
+			`,
+			output:      `{"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{}}}}}}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log {
+					output file foo.log
+				}
+			}
+			`,
+			output:      `{"logging":{"logs":{"default":{"exclude":["http.log.access.log0"]},"log0":{"writer":{"filename":"foo.log","output":"file"},"include":["http.log.access.log0"]}}},"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{"default_logger_name":"log0"}}}}}}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log {
+					format filter {
+						wrap console
+						fields {
+							common_log delete
+							request>remote_addr ip_mask {
+								ipv4 24
+								ipv6 32
+							}
+						}
+					}
+				}
+			}
+			`,
+			output:      `{"logging":{"logs":{"default":{"exclude":["http.log.access.log0"]},"log0":{"encoder":{"fields":{"common_log":{"filter":"delete"},"request\u003eremote_addr":{"filter":"ip_mask","ipv4_cidr":24,"ipv6_cidr":32}},"format":"filter","wrap":{"format":"console"}},"include":["http.log.access.log0"]}}},"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{"default_logger_name":"log0"}}}}}}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log invalid {
+					output file foo.log
+				}
+			}
+			`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		out, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+
+		if string(out) != tc.output {
+			t.Errorf("Test %d error output mismatch Expected: %s, got %s", i, tc.output, out)
+		}
+	}
+}
+
+func TestRedirDirectiveSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectError bool
+	}{
+		{
+			input: `:8080 {
+				redir :8081
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir * :8081
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /api/* :8081 300
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir :8081 300
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /api/* :8081 399
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir :8081 399
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html temporary
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir https://example.com{uri} permanent
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html permanent
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html html
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html htlm
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 200
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 400
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 temp
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 perm
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 php
+			}`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+	}
+}

caddyconfig/httpcaddyfile/directives.go

@@ -37,9 +37,11 @@ import (
 // The header directive goes second so that headers
 // can be manipulated before doing redirects.
 var directiveOrder = []string{
+	"map",
 	"root",
 
 	"header",
+	"request_body",
 
 	"redir",
 	"rewrite",
@@ -54,15 +56,21 @@ var directiveOrder = []string{
 	"encode",
 	"templates",
 
-	// special routing directives
+	// special routing & dispatching directives
 	"handle",
+	"handle_path",
 	"route",
+	"push",
 
 	// handlers that typically respond to requests
 	"respond",
+	"metrics",
 	"reverse_proxy",
 	"php_fastcgi",
 	"file_server",
+	"acme_server",
+	"abort",
+	"error",
 }
 
 // directiveIsOrdered returns true if dir is
@@ -97,20 +105,11 @@ func RegisterHandlerDirective(dir string, setupFunc UnmarshalHandlerFunc) {
 			return nil, h.ArgErr()
 		}
 
-		matcherSet, ok, err := h.MatcherToken()
+		matcherSet, err := h.ExtractMatcherSet()
 		if err != nil {
 			return nil, err
 		}
-		if ok {
-			// strip matcher token; we don't need to
-			// use the return value here because a
-			// new dispenser should have been made
-			// solely for this directive's tokens,
-			// with no other uses of same slice
-			h.Dispenser.Delete()
-		}
 
-		h.Dispenser.Reset() // pretend this lookahead never happened
 		val, err := setupFunc(h)
 		if err != nil {
 			return nil, err
@@ -120,6 +119,17 @@ func RegisterHandlerDirective(dir string, setupFunc UnmarshalHandlerFunc) {
 	})
 }
 
+// RegisterGlobalOption registers a unique global option opt with
+// an associated unmarshaling (setup) function. When the global
+// option opt is encountered in a Caddyfile, setupFunc will be
+// called to unmarshal its tokens.
+func RegisterGlobalOption(opt string, setupFunc UnmarshalGlobalFunc) {
+	if _, ok := registeredGlobalOptions[opt]; ok {
+		panic("global option " + opt + " already registered")
+	}
+	registeredGlobalOptions[opt] = setupFunc
+}
+
 // Helper is a type which helps setup a value from
 // Caddyfile tokens.
 type Helper struct {
@@ -184,7 +194,12 @@ func (h Helper) ExtractMatcherSet() (caddy.ModuleMap, error) {
 		return nil, err
 	}
 	if hasMatcher {
-		h.Dispenser.Delete() // strip matcher token
+		// strip matcher token; we don't need to
+		// use the return value here because a
+		// new dispenser should have been made
+		// solely for this directive's tokens,
+		// with no other uses of same slice
+		h.Dispenser.Delete()
 	}
 	h.Dispenser.Reset() // pretend this lookahead never happened
 	return matcherSet, nil
@@ -250,89 +265,37 @@ func (h Helper) NewBindAddresses(addrs []string) []ConfigValue {
 	return []ConfigValue{{Class: "bind", Value: addrs}}
 }
 
-// ConfigValue represents a value to be added to the final
-// configuration, or a value to be consulted when building
-// the final configuration.
-type ConfigValue struct {
-	// The kind of value this is. As the config is
-	// being built, the adapter will look in the
-	// "pile" for values belonging to a certain
-	// class when it is setting up a certain part
-	// of the config. The associated value will be
-	// type-asserted and placed accordingly.
-	Class string
-
-	// The value to be used when building the config.
-	// Generally its type is associated with the
-	// name of the Class.
-	Value interface{}
-
-	directive string
+// WithDispenser returns a new instance based on d. All others Helper
+// fields are copied, so typically maps are shared with this new instance.
+func (h Helper) WithDispenser(d *caddyfile.Dispenser) Helper {
+	h.Dispenser = d
+	return h
 }
 
-func sortRoutes(routes []ConfigValue) {
-	dirPositions := make(map[string]int)
-	for i, dir := range directiveOrder {
-		dirPositions[dir] = i
-	}
-
-	// while we are sorting, we will need to decode a route's path matcher
-	// in order to sub-sort by path length; we can amortize this operation
-	// for efficiency by storing the decoded matchers in a slice
-	decodedMatchers := make([]caddyhttp.MatchPath, len(routes))
-
-	sort.SliceStable(routes, func(i, j int) bool {
-		iDir, jDir := routes[i].directive, routes[j].directive
-		if iDir == jDir {
-			// directives are the same; sub-sort by path matcher length
-			// if there's only one matcher set and one path (common case)
-			iRoute, ok := routes[i].Value.(caddyhttp.Route)
-			if !ok {
-				return false
-			}
-			jRoute, ok := routes[j].Value.(caddyhttp.Route)
-			if !ok {
-				return false
-			}
-
-			// use already-decoded matcher, or decode if it's the first time seeing it
-			iPM, jPM := decodedMatchers[i], decodedMatchers[j]
-			if iPM == nil && len(iRoute.MatcherSetsRaw) == 1 {
-				var pathMatcher caddyhttp.MatchPath
-				_ = json.Unmarshal(iRoute.MatcherSetsRaw[0]["path"], &pathMatcher)
-				decodedMatchers[i] = pathMatcher
-				iPM = pathMatcher
-			}
-			if jPM == nil && len(jRoute.MatcherSetsRaw) == 1 {
-				var pathMatcher caddyhttp.MatchPath
-				_ = json.Unmarshal(jRoute.MatcherSetsRaw[0]["path"], &pathMatcher)
-				decodedMatchers[j] = pathMatcher
-				jPM = pathMatcher
-			}
-
-			// sort by longer path (more specific) first; missing
-			// path matchers are treated as zero-length paths
-			var iPathLen, jPathLen int
-			if iPM != nil {
-				iPathLen = len(iPM[0])
-			}
-			if jPM != nil {
-				jPathLen = len(jPM[0])
-			}
-			return iPathLen > jPathLen
-		}
-
-		return dirPositions[iDir] < dirPositions[jDir]
-	})
-}
-
-// parseSegmentAsSubroute parses the segment such that its subdirectives
+// ParseSegmentAsSubroute parses the segment such that its subdirectives
 // are themselves treated as directives, from which a subroute is built
 // and returned.
-func parseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
+func ParseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	allResults, err := parseSegmentAsConfig(h)
+	if err != nil {
+		return nil, err
+	}
+
+	return buildSubroute(allResults, h.groupCounter)
+}
+
+// parseSegmentAsConfig parses the segment such that its subdirectives
+// are themselves treated as directives, including named matcher definitions,
+// and the raw Config structs are returned.
+func parseSegmentAsConfig(h Helper) ([]ConfigValue, error) {
 	var allResults []ConfigValue
 
 	for h.Next() {
+		// don't allow non-matcher args on the first line
+		if h.NextArg() {
+			return nil, h.ArgErr()
+		}
+
 		// slice the linear list of tokens into top-level segments
 		var segments []caddyfile.Segment
 		for nesting := h.Nesting(); h.NextBlock(nesting); {
@@ -347,13 +310,17 @@ func parseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 		}
 
 		// find and extract any embedded matcher definitions in this scope
-		for i, seg := range segments {
+		for i := 0; i < len(segments); i++ {
+			seg := segments[i]
 			if strings.HasPrefix(seg.Directive(), matcherPrefix) {
+				// parse, then add the matcher to matcherDefs
 				err := parseMatcherDefinitions(caddyfile.NewDispenser(seg), matcherDefs)
 				if err != nil {
 					return nil, err
 				}
+				// remove the matcher segment (consumed), then step back the loop
 				segments = append(segments[:i], segments[i+1:]...)
+				i--
 			}
 		}
 
@@ -380,7 +347,83 @@ func parseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 		}
 	}
 
-	return buildSubroute(allResults, h.groupCounter)
+	return allResults, nil
+}
+
+// ConfigValue represents a value to be added to the final
+// configuration, or a value to be consulted when building
+// the final configuration.
+type ConfigValue struct {
+	// The kind of value this is. As the config is
+	// being built, the adapter will look in the
+	// "pile" for values belonging to a certain
+	// class when it is setting up a certain part
+	// of the config. The associated value will be
+	// type-asserted and placed accordingly.
+	Class string
+
+	// The value to be used when building the config.
+	// Generally its type is associated with the
+	// name of the Class.
+	Value interface{}
+
+	directive string
+}
+
+func sortRoutes(routes []ConfigValue) {
+	dirPositions := make(map[string]int)
+	for i, dir := range directiveOrder {
+		dirPositions[dir] = i
+	}
+
+	sort.SliceStable(routes, func(i, j int) bool {
+		// if the directives are different, just use the established directive order
+		iDir, jDir := routes[i].directive, routes[j].directive
+		if iDir != jDir {
+			return dirPositions[iDir] < dirPositions[jDir]
+		}
+
+		// directives are the same; sub-sort by path matcher length if there's
+		// only one matcher set and one path (this is a very common case and
+		// usually -- but not always -- helpful/expected, oh well; user can
+		// always take manual control of order using handler or route blocks)
+		iRoute, ok := routes[i].Value.(caddyhttp.Route)
+		if !ok {
+			return false
+		}
+		jRoute, ok := routes[j].Value.(caddyhttp.Route)
+		if !ok {
+			return false
+		}
+
+		// decode the path matchers, if there is just one of them
+		var iPM, jPM caddyhttp.MatchPath
+		if len(iRoute.MatcherSetsRaw) == 1 {
+			_ = json.Unmarshal(iRoute.MatcherSetsRaw[0]["path"], &iPM)
+		}
+		if len(jRoute.MatcherSetsRaw) == 1 {
+			_ = json.Unmarshal(jRoute.MatcherSetsRaw[0]["path"], &jPM)
+		}
+
+		// sort by longer path (more specific) first; missing path
+		// matchers or multi-matchers are treated as zero-length paths
+		var iPathLen, jPathLen int
+		if len(iPM) > 0 {
+			iPathLen = len(iPM[0])
+		}
+		if len(jPM) > 0 {
+			jPathLen = len(jPM[0])
+		}
+
+		// if both directives have no path matcher, use whichever one
+		// has any kind of matcher defined first.
+		if iPathLen == 0 && jPathLen == 0 {
+			return len(iRoute.MatcherSetsRaw) > 0 && len(jRoute.MatcherSetsRaw) == 0
+		}
+
+		// sort with the most-specific (longest) path first
+		return iPathLen > jPathLen
+	})
 }
 
 // serverBlock pairs a Caddyfile server block with
@@ -393,23 +436,30 @@ type serverBlock struct {
 }
 
 // hostsFromKeys returns a list of all the non-empty hostnames found in
-// the keys of the server block sb, unless allowEmpty is true, in which
-// case a key with no host (e.g. ":443") will be added to the list as an
-// empty string. Otherwise, if allowEmpty is false, and if sb has a key
-// that omits the hostname (i.e. is a catch-all/empty host), then the returned
-// list is empty, because the server block effectively matches ALL hosts.
-// The list may not be in a consistent order. If includePorts is true, then
-// any non-empty, non-standard ports will be included.
-func (sb serverBlock) hostsFromKeys(allowEmpty, includePorts bool) []string {
-	// first get each unique hostname
+// the keys of the server block sb. If logger mode is false, a key with
+// an empty hostname portion will return an empty slice, since that
+// server block is interpreted to effectively match all hosts. An empty
+// string is never added to the slice.
+//
+// If loggerMode is true, then the non-standard ports of keys will be
+// joined to the hostnames. This is to effectively match the Host
+// header of requests that come in for that key.
+//
+// The resulting slice is not sorted but will never have duplicates.
+func (sb serverBlock) hostsFromKeys(loggerMode bool) []string {
+	// ensure each entry in our list is unique
 	hostMap := make(map[string]struct{})
 	for _, addr := range sb.keys {
-		if addr.Host == "" && !allowEmpty {
-			// server block contains a key like ":443", i.e. the host portion
-			// is empty / catch-all, which means to match all hosts
-			return []string{}
+		if addr.Host == "" {
+			if !loggerMode {
+				// server block contains a key like ":443", i.e. the host portion
+				// is empty / catch-all, which means to match all hosts
+				return []string{}
+			}
+			// never append an empty string
+			continue
 		}
-		if includePorts &&
+		if loggerMode &&
 			addr.Port != "" &&
 			addr.Port != strconv.Itoa(caddyhttp.DefaultHTTPPort) &&
 			addr.Port != strconv.Itoa(caddyhttp.DefaultHTTPSPort) {
@@ -428,6 +478,17 @@ func (sb serverBlock) hostsFromKeys(allowEmpty, includePorts bool) []string {
 	return sblockHosts
 }
 
+// hasHostCatchAllKey returns true if sb has a key that
+// omits a host portion, i.e. it "catches all" hosts.
+func (sb serverBlock) hasHostCatchAllKey() bool {
+	for _, addr := range sb.keys {
+		if addr.Host == "" {
+			return true
+		}
+	}
+	return false
+}
+
 type (
 	// UnmarshalFunc is a function which can unmarshal Caddyfile
 	// tokens into zero or more config values using a Helper type.
@@ -444,6 +505,14 @@ type (
 	// for you. These are passed to a call to
 	// RegisterHandlerDirective.
 	UnmarshalHandlerFunc func(h Helper) (caddyhttp.MiddlewareHandler, error)
+
+	// UnmarshalGlobalFunc is a function which can unmarshal Caddyfile
+	// tokens from a global option. It is passed the tokens to parse and
+	// existing value from the previous instance of this global option
+	// (if any). It returns the value to associate with this global option.
+	UnmarshalGlobalFunc func(d *caddyfile.Dispenser, existingVal interface{}) (interface{}, error)
 )
 
 var registeredDirectives = make(map[string]UnmarshalFunc)
+
+var registeredGlobalOptions = make(map[string]UnmarshalGlobalFunc)

caddyconfig/httpcaddyfile/directives_test.go

@@ -0,0 +1,94 @@
+package httpcaddyfile
+
+import (
+	"reflect"
+	"sort"
+	"testing"
+)
+
+func TestHostsFromKeys(t *testing.T) {
+	for i, tc := range []struct {
+		keys             []Address
+		expectNormalMode []string
+		expectLoggerMode []string
+	}{
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+			},
+			[]string{"foo"},
+			[]string{"foo"},
+		},
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+				{Original: "bar", Host: "bar"},
+			},
+			[]string{"bar", "foo"},
+			[]string{"bar", "foo"},
+		},
+		{
+			[]Address{
+				{Original: ":2015", Port: "2015"},
+			},
+			[]string{}, []string{},
+		},
+		{
+			[]Address{
+				{Original: ":443", Port: "443"},
+			},
+			[]string{}, []string{},
+		},
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+				{Original: ":2015", Port: "2015"},
+			},
+			[]string{}, []string{"foo"},
+		},
+		{
+			[]Address{
+				{Original: "example.com:2015", Host: "example.com", Port: "2015"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com:2015"},
+		},
+		{
+			[]Address{
+				{Original: "example.com:80", Host: "example.com", Port: "80"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com"},
+		},
+		{
+			[]Address{
+				{Original: "https://:2015/foo", Scheme: "https", Port: "2015", Path: "/foo"},
+			},
+			[]string{},
+			[]string{},
+		},
+		{
+			[]Address{
+				{Original: "https://example.com:2015/foo", Scheme: "https", Host: "example.com", Port: "2015", Path: "/foo"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com:2015"},
+		},
+	} {
+		sb := serverBlock{keys: tc.keys}
+
+		// test in normal mode
+		actual := sb.hostsFromKeys(false)
+		sort.Strings(actual)
+		if !reflect.DeepEqual(tc.expectNormalMode, actual) {
+			t.Errorf("Test %d (loggerMode=false): Expected: %v Actual: %v", i, tc.expectNormalMode, actual)
+		}
+
+		// test in logger mode
+		actual = sb.hostsFromKeys(true)
+		sort.Strings(actual)
+		if !reflect.DeepEqual(tc.expectLoggerMode, actual) {
+			t.Errorf("Test %d (loggerMode=true): Expected: %v Actual: %v", i, tc.expectLoggerMode, actual)
+		}
+	}
+}

caddyconfig/httpcaddyfile/httptype.go

@@ -17,7 +17,9 @@ package httpcaddyfile
 import (
 	"encoding/json"
 	"fmt"
+	"log"
 	"reflect"
+	"regexp"
 	"sort"
 	"strconv"
 	"strings"
@@ -26,6 +28,7 @@ import (
 	"github.com/caddyserver/caddy/v2/caddyconfig"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
+	"github.com/caddyserver/caddy/v2/modules/caddypki"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
 )
 
@@ -33,12 +36,23 @@ func init() {
 	caddyconfig.RegisterAdapter("caddyfile", caddyfile.Adapter{ServerType: ServerType{}})
 }
 
+// App represents the configuration for a non-standard
+// Caddy app module (e.g. third-party plugin) which was
+// parsed from a global options block.
+type App struct {
+	// The JSON key for the app being configured
+	Name string
+
+	// The raw app config as JSON
+	Value json.RawMessage
+}
+
 // ServerType can set up a config from an HTTP Caddyfile.
 type ServerType struct {
 }
 
 // Setup makes a config from the tokens.
-func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
+func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 	options map[string]interface{}) (*caddy.Config, []caddyconfig.Warning, error) {
 	var warnings []caddyconfig.Warning
 	gc := counter{new(int)}
@@ -50,15 +64,18 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 	// chosen to handle a request - we actually will make each
 	// server block's route terminal so that only one will run
 	sbKeys := make(map[string]struct{})
-	var serverBlocks []serverBlock
-	for i, sblock := range originalServerBlocks {
+	originalServerBlocks := make([]serverBlock, 0, len(inputServerBlocks))
+	for i, sblock := range inputServerBlocks {
 		for j, k := range sblock.Keys {
+			if j == 0 && strings.HasPrefix(k, "@") {
+				return nil, warnings, fmt.Errorf("cannot define a matcher outside of a site block: '%s'", k)
+			}
 			if _, ok := sbKeys[k]; ok {
 				return nil, warnings, fmt.Errorf("duplicate site address not allowed: '%s' in %v (site block %d, key %d)", k, sblock.Keys, i, j)
 			}
 			sbKeys[k] = struct{}{}
 		}
-		serverBlocks = append(serverBlocks, serverBlock{
+		originalServerBlocks = append(originalServerBlocks, serverBlock{
 			block: sblock,
 			pile:  make(map[string][]ConfigValue),
 		})
@@ -66,39 +83,61 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 
 	// apply any global options
 	var err error
-	serverBlocks, err = st.evaluateGlobalOptionsBlock(serverBlocks, options)
+	originalServerBlocks, err = st.evaluateGlobalOptionsBlock(originalServerBlocks, options)
 	if err != nil {
 		return nil, warnings, err
 	}
 
-	for _, sb := range serverBlocks {
-		// replace shorthand placeholders (which are
-		// convenient when writing a Caddyfile) with
-		// their actual placeholder identifiers or
-		// variable names
-		replacer := strings.NewReplacer(
-			"{dir}", "{http.request.uri.path.dir}",
-			"{file}", "{http.request.uri.path.file}",
-			"{host}", "{http.request.host}",
-			"{hostport}", "{http.request.hostport}",
-			"{method}", "{http.request.method}",
-			"{path}", "{http.request.uri.path}",
-			"{query}", "{http.request.uri.query}",
-			"{remote}", "{http.request.remote}",
-			"{remote_host}", "{http.request.remote.host}",
-			"{remote_port}", "{http.request.remote.port}",
-			"{scheme}", "{http.request.scheme}",
-			"{uri}", "{http.request.uri}",
-			"{tls_cipher}", "{http.request.tls.cipher_suite}",
-			"{tls_version}", "{http.request.tls.version}",
-			"{tls_client_fingerprint}", "{http.request.tls.client.fingerprint}",
-			"{tls_client_issuer}", "{http.request.tls.client.issuer}",
-			"{tls_client_serial}", "{http.request.tls.client.serial}",
-			"{tls_client_subject}", "{http.request.tls.client.subject}",
-		)
+	// replace shorthand placeholders (which are
+	// convenient when writing a Caddyfile) with
+	// their actual placeholder identifiers or
+	// variable names
+	replacer := strings.NewReplacer(
+		"{dir}", "{http.request.uri.path.dir}",
+		"{file}", "{http.request.uri.path.file}",
+		"{host}", "{http.request.host}",
+		"{hostport}", "{http.request.hostport}",
+		"{port}", "{http.request.port}",
+		"{method}", "{http.request.method}",
+		"{path}", "{http.request.uri.path}",
+		"{query}", "{http.request.uri.query}",
+		"{remote}", "{http.request.remote}",
+		"{remote_host}", "{http.request.remote.host}",
+		"{remote_port}", "{http.request.remote.port}",
+		"{scheme}", "{http.request.scheme}",
+		"{uri}", "{http.request.uri}",
+		"{tls_cipher}", "{http.request.tls.cipher_suite}",
+		"{tls_version}", "{http.request.tls.version}",
+		"{tls_client_fingerprint}", "{http.request.tls.client.fingerprint}",
+		"{tls_client_issuer}", "{http.request.tls.client.issuer}",
+		"{tls_client_serial}", "{http.request.tls.client.serial}",
+		"{tls_client_subject}", "{http.request.tls.client.subject}",
+		"{tls_client_certificate_pem}", "{http.request.tls.client.certificate_pem}",
+	)
+
+	// these are placeholders that allow a user-defined final
+	// parameters, but we still want to provide a shorthand
+	// for those, so we use a regexp to replace
+	regexpReplacements := []struct {
+		search  *regexp.Regexp
+		replace string
+	}{
+		{regexp.MustCompile(`{query\.([\w-]*)}`), "{http.request.uri.query.$1}"},
+		{regexp.MustCompile(`{labels\.([\w-]*)}`), "{http.request.host.labels.$1}"},
+		{regexp.MustCompile(`{header\.([\w-]*)}`), "{http.request.header.$1}"},
+		{regexp.MustCompile(`{path\.([\w-]*)}`), "{http.request.uri.path.$1}"},
+		{regexp.MustCompile(`{re\.([\w-]*)\.([\w-]*)}`), "{http.regexp.$1.$2}"},
+	}
+
+	for _, sb := range originalServerBlocks {
 		for _, segment := range sb.block.Segments {
 			for i := 0; i < len(segment); i++ {
+				// simple string replacements
 				segment[i].Text = replacer.Replace(segment[i].Text)
+				// complex regexp replacements
+				for _, r := range regexpReplacements {
+					segment[i].Text = r.search.ReplaceAllString(segment[i].Text, r.replace)
+				}
 			}
 		}
 
@@ -147,6 +186,15 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 			if err != nil {
 				return nil, warnings, fmt.Errorf("parsing caddyfile tokens for '%s': %v", dir, err)
 			}
+
+			// As a special case, we want "handle_path" to be sorted
+			// at the same level as "handle", so we force them to use
+			// the same directive name after their parsing is complete.
+			// See https://github.com/caddyserver/caddy/issues/3675#issuecomment-678042377
+			if dir == "handle_path" {
+				dir = "handle"
+			}
+
 			for _, result := range results {
 				result.directive = dir
 				sb.pile[result.Class] = append(sb.pile[result.Class], result)
@@ -155,7 +203,7 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 	}
 
 	// map
-	sbmap, err := st.mapAddressToServerBlocks(serverBlocks, options)
+	sbmap, err := st.mapAddressToServerBlocks(originalServerBlocks, options)
 	if err != nil {
 		return nil, warnings, err
 	}
@@ -172,9 +220,10 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 
 	// now that each server is configured, make the HTTP app
 	httpApp := caddyhttp.App{
-		HTTPPort:  tryInt(options["http_port"], &warnings),
-		HTTPSPort: tryInt(options["https_port"], &warnings),
-		Servers:   servers,
+		HTTPPort:    tryInt(options["http_port"], &warnings),
+		HTTPSPort:   tryInt(options["https_port"], &warnings),
+		GracePeriod: tryDuration(options["grace_period"], &warnings),
+		Servers:     servers,
 	}
 
 	// then make the TLS app
@@ -183,31 +232,42 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 		return nil, warnings, err
 	}
 
-	// if experimental HTTP/3 is enabled, enable it on each server
-	if enableH3, ok := options["experimental_http3"].(bool); ok && enableH3 {
-		for _, srv := range httpApp.Servers {
-			srv.ExperimentalHTTP3 = true
-		}
+	// then make the PKI app
+	pkiApp, warnings, err := st.buildPKIApp(pairings, options, warnings)
+	if err != nil {
+		return nil, warnings, err
 	}
 
 	// extract any custom logs, and enforce configured levels
 	var customLogs []namedCustomLog
 	var hasDefaultLog bool
-	for _, sb := range serverBlocks {
-		for _, clVal := range sb.pile["custom_log"] {
-			ncl := clVal.Value.(namedCustomLog)
-			if ncl.name == "" {
-				continue
-			}
-			if ncl.name == "default" {
-				hasDefaultLog = true
-			}
-			if _, ok := options["debug"]; ok && ncl.log.Level == "" {
-				ncl.log.Level = "DEBUG"
-			}
-			customLogs = append(customLogs, ncl)
+	addCustomLog := func(ncl namedCustomLog) {
+		if ncl.name == "" {
+			return
+		}
+		if ncl.name == "default" {
+			hasDefaultLog = true
+		}
+		if _, ok := options["debug"]; ok && ncl.log.Level == "" {
+			ncl.log.Level = "DEBUG"
+		}
+		customLogs = append(customLogs, ncl)
+	}
+	// Apply global log options, when set
+	if options["log"] != nil {
+		for _, logValue := range options["log"].([]ConfigValue) {
+			addCustomLog(logValue.Value.(namedCustomLog))
 		}
 	}
+	// Apply server-specific log options
+	for _, p := range pairings {
+		for _, sb := range p.serverBlocks {
+			for _, clVal := range sb.pile["custom_log"] {
+				addCustomLog(clVal.Value.(namedCustomLog))
+			}
+		}
+	}
+
 	if !hasDefaultLog {
 		// if the default log was not customized, ensure we
 		// configure it with any applicable options
@@ -221,24 +281,34 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 
 	// annnd the top-level config, then we're done!
 	cfg := &caddy.Config{AppsRaw: make(caddy.ModuleMap)}
+
+	// loop through the configured options, and if any of
+	// them are an httpcaddyfile App, then we insert them
+	// into the config as raw Caddy apps
+	for _, opt := range options {
+		if app, ok := opt.(App); ok {
+			cfg.AppsRaw[app.Name] = app.Value
+		}
+	}
+
+	// insert the standard Caddy apps into the config
 	if len(httpApp.Servers) > 0 {
 		cfg.AppsRaw["http"] = caddyconfig.JSON(httpApp, &warnings)
 	}
 	if !reflect.DeepEqual(tlsApp, &caddytls.TLS{CertificatesRaw: make(caddy.ModuleMap)}) {
 		cfg.AppsRaw["tls"] = caddyconfig.JSON(tlsApp, &warnings)
 	}
+	if !reflect.DeepEqual(pkiApp, &caddypki.PKI{CAs: make(map[string]*caddypki.CA)}) {
+		cfg.AppsRaw["pki"] = caddyconfig.JSON(pkiApp, &warnings)
+	}
 	if storageCvtr, ok := options["storage"].(caddy.StorageConverter); ok {
 		cfg.StorageRaw = caddyconfig.JSONModuleObject(storageCvtr,
 			"module",
 			storageCvtr.(caddy.Module).CaddyModule().ID.Name(),
 			&warnings)
 	}
-	if adminConfig, ok := options["admin"].(string); ok && adminConfig != "" {
-		if adminConfig == "off" {
-			cfg.Admin = &caddy.AdminConfig{Disabled: true}
-		} else {
-			cfg.Admin = &caddy.AdminConfig{Listen: adminConfig}
-		}
+	if adminConfig, ok := options["admin"].(*caddy.AdminConfig); ok && adminConfig != nil {
+		cfg.Admin = adminConfig
 	}
 	if len(customLogs) > 0 {
 		if cfg.Logging == nil {
@@ -250,17 +320,16 @@ func (st ServerType) Setup(originalServerBlocks []caddyfile.ServerBlock,
 			if ncl.name != "" {
 				cfg.Logging.Logs[ncl.name] = ncl.log
 			}
-		}
-	}
-	if len(customLogs) > 0 {
-		if cfg.Logging == nil {
-			cfg.Logging = &caddy.Logging{
-				Logs: make(map[string]*caddy.CustomLog),
-			}
-		}
-		for _, ncl := range customLogs {
-			if ncl.name != "" {
-				cfg.Logging.Logs[ncl.name] = ncl.log
+			// most users seem to prefer not writing access logs
+			// to the default log when they are directed to a
+			// file or have any other special customization
+			if ncl.name != "default" && len(ncl.log.Include) > 0 {
+				defaultLog, ok := cfg.Logging.Logs["default"]
+				if !ok {
+					defaultLog = new(caddy.CustomLog)
+					cfg.Logging.Logs["default"] = defaultLog
+				}
+				defaultLog.Exclude = append(defaultLog.Exclude, ncl.log.Include...)
 			}
 		}
 	}
@@ -278,42 +347,68 @@ func (ServerType) evaluateGlobalOptionsBlock(serverBlocks []serverBlock, options
 	}
 
 	for _, segment := range serverBlocks[0].block.Segments {
-		dir := segment.Directive()
+		opt := segment.Directive()
 		var val interface{}
 		var err error
 		disp := caddyfile.NewDispenser(segment)
-		switch dir {
-		case "debug":
-			val = true
-		case "http_port":
-			val, err = parseOptHTTPPort(disp)
-		case "https_port":
-			val, err = parseOptHTTPSPort(disp)
-		case "default_sni":
-			val, err = parseOptSingleString(disp)
-		case "order":
-			val, err = parseOptOrder(disp)
-		case "experimental_http3":
-			val, err = parseOptExperimentalHTTP3(disp)
-		case "storage":
-			val, err = parseOptStorage(disp)
-		case "acme_ca", "acme_dns", "acme_ca_root":
-			val, err = parseOptSingleString(disp)
-		case "email":
-			val, err = parseOptSingleString(disp)
-		case "admin":
-			val, err = parseOptAdmin(disp)
-		case "on_demand_tls":
-			val, err = parseOptOnDemand(disp)
-		case "local_certs":
-			val = true
-		default:
-			return nil, fmt.Errorf("unrecognized parameter name: %s", dir)
+
+		optFunc, ok := registeredGlobalOptions[opt]
+		if !ok {
+			tkn := segment[0]
+			return nil, fmt.Errorf("%s:%d: unrecognized global option: %s", tkn.File, tkn.Line, opt)
 		}
+
+		val, err = optFunc(disp, options[opt])
 		if err != nil {
-			return nil, fmt.Errorf("%s: %v", dir, err)
+			return nil, fmt.Errorf("parsing caddyfile tokens for '%s': %v", opt, err)
+		}
+
+		// As a special case, fold multiple "servers" options together
+		// in an array instead of overwriting a possible existing value
+		if opt == "servers" {
+			existingOpts, ok := options[opt].([]serverOptions)
+			if !ok {
+				existingOpts = []serverOptions{}
+			}
+			serverOpts, ok := val.(serverOptions)
+			if !ok {
+				return nil, fmt.Errorf("unexpected type from 'servers' global options: %T", val)
+			}
+			options[opt] = append(existingOpts, serverOpts)
+			continue
+		}
+		// Additionally, fold multiple "log" options together into an
+		// array so that multiple loggers can be configured.
+		if opt == "log" {
+			existingOpts, ok := options[opt].([]ConfigValue)
+			if !ok {
+				existingOpts = []ConfigValue{}
+			}
+			logOpts, ok := val.([]ConfigValue)
+			if !ok {
+				return nil, fmt.Errorf("unexpected type from 'log' global options: %T", val)
+			}
+			options[opt] = append(existingOpts, logOpts...)
+			continue
+		}
+
+		options[opt] = val
+	}
+
+	// If we got "servers" options, we'll sort them by their listener address
+	if serverOpts, ok := options["servers"].([]serverOptions); ok {
+		sort.Slice(serverOpts, func(i, j int) bool {
+			return len(serverOpts[i].ListenerAddress) > len(serverOpts[j].ListenerAddress)
+		})
+
+		// Reject the config if there are duplicate listener address
+		seen := make(map[string]bool)
+		for _, entry := range serverOpts {
+			if _, alreadySeen := seen[entry.ListenerAddress]; alreadySeen {
+				return nil, fmt.Errorf("cannot have 'servers' global options with duplicate listener addresses: %s", entry.ListenerAddress)
+			}
+			seen[entry.ListenerAddress] = true
 		}
-		options[dir] = val
 	}
 
 	return serverBlocks[1:], nil
@@ -330,11 +425,38 @@ func (st *ServerType) serversFromPairings(
 	servers := make(map[string]*caddyhttp.Server)
 	defaultSNI := tryString(options["default_sni"], warnings)
 
+	httpPort := strconv.Itoa(caddyhttp.DefaultHTTPPort)
+	if hp, ok := options["http_port"].(int); ok {
+		httpPort = strconv.Itoa(hp)
+	}
+	httpsPort := strconv.Itoa(caddyhttp.DefaultHTTPSPort)
+	if hsp, ok := options["https_port"].(int); ok {
+		httpsPort = strconv.Itoa(hsp)
+	}
+	autoHTTPS := "on"
+	if ah, ok := options["auto_https"].(string); ok {
+		autoHTTPS = ah
+	}
+
 	for i, p := range pairings {
 		srv := &caddyhttp.Server{
 			Listen: p.addresses,
 		}
 
+		// handle the auto_https global option
+		if autoHTTPS != "on" {
+			srv.AutoHTTPS = new(caddyhttp.AutoHTTPSConfig)
+			if autoHTTPS == "off" {
+				srv.AutoHTTPS.Disabled = true
+			}
+			if autoHTTPS == "disable_redirects" {
+				srv.AutoHTTPS.DisableRedir = true
+			}
+			if autoHTTPS == "ignore_loaded_certs" {
+				srv.AutoHTTPS.IgnoreLoadedCerts = true
+			}
+		}
+
 		// sort server blocks by their keys; this is important because
 		// only the first matching site should be evaluated, and we should
 		// attempt to match most specific site first (host and path), in
@@ -345,7 +467,11 @@ func (st *ServerType) serversFromPairings(
 			// but I don't expect many blocks will have THAT many keys...
 			var iLongestPath, jLongestPath string
 			var iLongestHost, jLongestHost string
+			var iWildcardHost, jWildcardHost bool
 			for _, addr := range p.serverBlocks[i].keys {
+				if strings.Contains(addr.Host, "*") || addr.Host == "" {
+					iWildcardHost = true
+				}
 				if specificity(addr.Host) > specificity(iLongestHost) {
 					iLongestHost = addr.Host
 				}
@@ -354,6 +480,9 @@ func (st *ServerType) serversFromPairings(
 				}
 			}
 			for _, addr := range p.serverBlocks[j].keys {
+				if strings.Contains(addr.Host, "*") || addr.Host == "" {
+					jWildcardHost = true
+				}
 				if specificity(addr.Host) > specificity(jLongestHost) {
 					jLongestHost = addr.Host
 				}
@@ -361,13 +490,37 @@ func (st *ServerType) serversFromPairings(
 					jLongestPath = addr.Path
 				}
 			}
+			// catch-all blocks (blocks with no hostname) should always go
+			// last, even after blocks with wildcard hosts
+			if specificity(iLongestHost) == 0 {
+				return false
+			}
+			if specificity(jLongestHost) == 0 {
+				return true
+			}
+			if iWildcardHost != jWildcardHost {
+				// site blocks that have a key with a wildcard in the hostname
+				// must always be less specific than blocks without one; see
+				// https://github.com/caddyserver/caddy/issues/3410
+				return jWildcardHost && !iWildcardHost
+			}
 			if specificity(iLongestHost) == specificity(jLongestHost) {
 				return len(iLongestPath) > len(jLongestPath)
 			}
 			return specificity(iLongestHost) > specificity(jLongestHost)
 		})
 
-		var hasCatchAllTLSConnPolicy bool
+		var hasCatchAllTLSConnPolicy, addressQualifiesForTLS bool
+		autoHTTPSWillAddConnPolicy := autoHTTPS != "off"
+
+		// if a catch-all server block (one which accepts all hostnames) exists in this pairing,
+		// we need to know that so that we can configure logs properly (see #3878)
+		var catchAllSblockExists bool
+		for _, sblock := range p.serverBlocks {
+			if len(sblock.hostsFromKeys(false)) == 0 {
+				catchAllSblockExists = true
+			}
+		}
 
 		// create a subroute for each site in the server block
 		for _, sblock := range p.serverBlocks {
@@ -376,7 +529,14 @@ func (st *ServerType) serversFromPairings(
 				return nil, fmt.Errorf("server block %v: compiling matcher sets: %v", sblock.block.Keys, err)
 			}
 
-			hosts := sblock.hostsFromKeys(false, false)
+			hosts := sblock.hostsFromKeys(false)
+
+			// emit warnings if user put unspecified IP addresses; they probably want the bind directive
+			for _, h := range hosts {
+				if h == "0.0.0.0" || h == "::" {
+					log.Printf("[WARNING] Site block has unspecified IP address %s which only matches requests having that Host header; you probably want the 'bind' directive to configure the socket", h)
+				}
+			}
 
 			// tls: connection policies
 			if cpVals, ok := sblock.pile["tls.connection_policy"]; ok {
@@ -399,24 +559,55 @@ func (st *ServerType) serversFromPairings(
 						}
 					} else {
 						cp.DefaultSNI = defaultSNI
-						hasCatchAllTLSConnPolicy = true
 					}
 
-					srv.TLSConnPolicies = append(srv.TLSConnPolicies, cp)
+					// only append this policy if it actually changes something
+					if !cp.SettingsEmpty() {
+						srv.TLSConnPolicies = append(srv.TLSConnPolicies, cp)
+						hasCatchAllTLSConnPolicy = len(hosts) == 0
+					}
 				}
 			}
 
-			// exclude any hosts that were defined explicitly with
-			// "http://" in the key from automated cert management (issue #2998)
 			for _, addr := range sblock.keys {
-				if addr.Scheme == "http" && addr.Host != "" {
-					if srv.AutoHTTPS == nil {
-						srv.AutoHTTPS = new(caddyhttp.AutoHTTPSConfig)
-					}
-					if !sliceContains(srv.AutoHTTPS.Skip, addr.Host) {
-						srv.AutoHTTPS.Skip = append(srv.AutoHTTPS.Skip, addr.Host)
+				// if server only uses HTTPS port, auto-HTTPS will not apply
+				if listenersUseAnyPortOtherThan(srv.Listen, httpPort) {
+					// exclude any hosts that were defined explicitly with "http://"
+					// in the key from automated cert management (issue #2998)
+					if addr.Scheme == "http" && addr.Host != "" {
+						if srv.AutoHTTPS == nil {
+							srv.AutoHTTPS = new(caddyhttp.AutoHTTPSConfig)
+						}
+						if !sliceContains(srv.AutoHTTPS.Skip, addr.Host) {
+							srv.AutoHTTPS.Skip = append(srv.AutoHTTPS.Skip, addr.Host)
+						}
 					}
 				}
+
+				// we'll need to remember if the address qualifies for auto-HTTPS, so we
+				// can add a TLS conn policy if necessary
+				if addr.Scheme == "https" ||
+					(addr.Scheme != "http" && addr.Host != "" && addr.Port != httpPort) {
+					addressQualifiesForTLS = true
+				}
+				// predict whether auto-HTTPS will add the conn policy for us; if so, we
+				// may not need to add one for this server
+				autoHTTPSWillAddConnPolicy = autoHTTPSWillAddConnPolicy &&
+					(addr.Port == httpsPort || (addr.Port != httpPort && addr.Host != ""))
+			}
+
+			// Look for any config values that provide listener wrappers on the server block
+			for _, listenerConfig := range sblock.pile["listener_wrapper"] {
+				listenerWrapper, ok := listenerConfig.Value.(caddy.ListenerWrapper)
+				if !ok {
+					return nil, fmt.Errorf("config for a listener wrapper did not provide a value that implements caddy.ListenerWrapper")
+				}
+				jsonListenerWrapper := caddyconfig.JSONModuleObject(
+					listenerWrapper,
+					"wrapper",
+					listenerWrapper.(caddy.Module).CaddyModule().ID.Name(),
+					warnings)
+				srv.ListenerWrappersRaw = append(srv.ListenerWrappersRaw, jsonListenerWrapper)
 			}
 
 			// set up each handler directive, making sure to honor directive order
@@ -441,16 +632,45 @@ func (st *ServerType) serversFromPairings(
 			}
 
 			// add log associations
+			// see https://github.com/caddyserver/caddy/issues/3310
+			sblockLogHosts := sblock.hostsFromKeys(true)
 			for _, cval := range sblock.pile["custom_log"] {
 				ncl := cval.Value.(namedCustomLog)
 				if srv.Logs == nil {
-					srv.Logs = &caddyhttp.ServerLogConfig{
-						LoggerNames: make(map[string]string),
+					srv.Logs = new(caddyhttp.ServerLogConfig)
+				}
+				if sblock.hasHostCatchAllKey() {
+					// all requests for hosts not able to be listed should use
+					// this log because it's a catch-all-hosts server block
+					srv.Logs.DefaultLoggerName = ncl.name
+				} else {
+					// map each host to the user's desired logger name
+					for _, h := range sblockLogHosts {
+						// if the custom logger name is non-empty, add it to the map;
+						// otherwise, only map to an empty logger name if this or
+						// another site block on this server has a catch-all host (in
+						// which case only requests with mapped hostnames will be
+						// access-logged, so it'll be necessary to add them to the
+						// map even if they use default logger)
+						if ncl.name != "" || catchAllSblockExists {
+							if srv.Logs.LoggerNames == nil {
+								srv.Logs.LoggerNames = make(map[string]string)
+							}
+							srv.Logs.LoggerNames[h] = ncl.name
+						}
 					}
 				}
-				for _, h := range sblock.hostsFromKeys(true, true) {
-					srv.Logs.LoggerNames[h] = ncl.name
+			}
+			if srv.Logs != nil && len(sblock.pile["custom_log"]) == 0 {
+				// server has access logs enabled, but this server block does not
+				// enable access logs; therefore, all hosts of this server block
+				// should not be access-logged
+				if len(hosts) == 0 {
+					// if the server block has a catch-all-hosts key, then we should
+					// not log reqs to any host unless it appears in the map
+					srv.Logs.SkipUnmappedHosts = true
 				}
+				srv.Logs.SkipHosts = append(srv.Logs.SkipHosts, sblockLogHosts...)
 			}
 		}
 
@@ -473,7 +693,9 @@ func (st *ServerType) serversFromPairings(
 		// TODO: maybe a smarter way to handle this might be to just make the
 		// auto-HTTPS logic at provision-time detect if there is any connection
 		// policy missing for any HTTPS-enabled hosts, if so, add it... maybe?
-		if !hasCatchAllTLSConnPolicy && (len(srv.TLSConnPolicies) > 0 || defaultSNI != "") {
+		if addressQualifiesForTLS &&
+			!hasCatchAllTLSConnPolicy &&
+			(len(srv.TLSConnPolicies) > 0 || !autoHTTPSWillAddConnPolicy || defaultSNI != "") {
 			srv.TLSConnPolicies = append(srv.TLSConnPolicies, &caddytls.ConnectionPolicy{DefaultSNI: defaultSNI})
 		}
 
@@ -487,6 +709,11 @@ func (st *ServerType) serversFromPairings(
 		servers[fmt.Sprintf("srv%d", i)] = srv
 	}
 
+	err := applyServerOptions(servers, options, warnings)
+	if err != nil {
+		return nil, err
+	}
+
 	return servers, nil
 }
 
@@ -533,7 +760,7 @@ func detectConflictingSchemes(srv *caddyhttp.Server, serverBlocks []serverBlock,
 				if err := checkAndSetHTTP(addr); err != nil {
 					return err
 				}
-			} else if addr.Scheme == "https" || addr.Port == httpsPort {
+			} else if addr.Scheme == "https" || addr.Port == httpsPort || len(srv.TLSConnPolicies) > 0 {
 				if err := checkAndSetHTTPS(addr); err != nil {
 					return err
 				}
@@ -548,10 +775,17 @@ func detectConflictingSchemes(srv *caddyhttp.Server, serverBlocks []serverBlock,
 	return nil
 }
 
-// consolidateConnPolicies combines TLS connection policies that are the same,
-// for a cleaner overall output.
+// consolidateConnPolicies sorts any catch-all policy to the end, removes empty TLS connection
+// policies, and combines equivalent ones for a cleaner overall output.
 func consolidateConnPolicies(cps caddytls.ConnectionPolicies) (caddytls.ConnectionPolicies, error) {
+	// catch-all policies (those without any matcher) should be at the
+	// end, otherwise it nullifies any more specific policies
+	sort.SliceStable(cps, func(i, j int) bool {
+		return cps[j].MatchersRaw == nil && cps[i].MatchersRaw != nil
+	})
+
 	for i := 0; i < len(cps); i++ {
+		// compare it to the others
 		for j := 0; j < len(cps); j++ {
 			if j == i {
 				continue
@@ -742,7 +976,18 @@ func buildSubroute(routes []ConfigValue, groupCounter counter) (*caddyhttp.Subro
 		// root directives would overwrite previously-matched ones; they should not cascade
 		"root": {},
 	}
-	for meDir, info := range mutuallyExclusiveDirs {
+
+	// we need to deterministically loop over each of these directives
+	// in order to keep the group numbers consistent
+	keys := make([]string, 0, len(mutuallyExclusiveDirs))
+	for k := range mutuallyExclusiveDirs {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+
+	for _, meDir := range keys {
+		info := mutuallyExclusiveDirs[meDir]
+
 		// see how many instances of the directive there are
 		for _, r := range routes {
 			if r.directive == meDir {
@@ -919,7 +1164,7 @@ func (st *ServerType) compileEncodedMatcherSets(sblock serverBlock) ([]caddy.Mod
 	}
 
 	// finally, encode each of the matcher sets
-	var matcherSetsEnc []caddy.ModuleMap
+	matcherSetsEnc := make([]caddy.ModuleMap, 0, len(matcherSets))
 	for _, ms := range matcherSets {
 		msEncoded, err := encodeMatcherSet(ms)
 		if err != nil {
@@ -945,7 +1190,7 @@ func parseMatcherDefinitions(d *caddyfile.Dispenser, matchers map[string]caddy.M
 		// handle more than one segment); otherwise, we'd overwrite other
 		// instances of the matcher in this set
 		tokensByMatcherName := make(map[string][]caddyfile.Token)
-		for nesting := d.Nesting(); d.NextBlock(nesting); {
+		for nesting := d.Nesting(); d.NextArg() || d.NextBlock(nesting); {
 			matcherName := d.Val()
 			tokensByMatcherName[matcherName] = append(tokensByMatcherName[matcherName], d.NextSegment()...)
 		}
@@ -1002,6 +1247,14 @@ func tryString(val interface{}, warnings *[]caddyconfig.Warning) string {
 	return stringVal
 }
 
+func tryDuration(val interface{}, warnings *[]caddyconfig.Warning) caddy.Duration {
+	durationVal, ok := val.(caddy.Duration)
+	if val != nil && !ok && warnings != nil {
+		*warnings = append(*warnings, caddyconfig.Warning{Message: "not a duration type"})
+	}
+	return durationVal
+}
+
 // sliceContains returns true if needle is in haystack.
 func sliceContains(haystack []string, needle string) bool {
 	for _, s := range haystack {
@@ -1012,6 +1265,26 @@ func sliceContains(haystack []string, needle string) bool {
 	return false
 }
 
+// listenersUseAnyPortOtherThan returns true if there are any
+// listeners in addresses that use a port which is not otherPort.
+// Mostly borrowed from unexported method in caddyhttp package.
+func listenersUseAnyPortOtherThan(addresses []string, otherPort string) bool {
+	otherPortInt, err := strconv.Atoi(otherPort)
+	if err != nil {
+		return false
+	}
+	for _, lnAddr := range addresses {
+		laddrs, err := caddy.ParseNetworkAddress(lnAddr)
+		if err != nil {
+			continue
+		}
+		if uint(otherPortInt) > laddrs.EndPort || uint(otherPortInt) < laddrs.StartPort {
+			return true
+		}
+	}
+	return false
+}
+
 // specificity returns len(s) minus any wildcards (*) and
 // placeholders ({...}). Basically, it's a length count
 // that penalizes the use of wildcards and placeholders.

caddyconfig/httpcaddyfile/httptype_test.go

@@ -6,28 +6,57 @@ import (
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 )
 
-func TestServerType(t *testing.T) {
+func TestMatcherSyntax(t *testing.T) {
 	for i, tc := range []struct {
 		input       string
-		expectWarn  bool
 		expectError bool
 	}{
 		{
 			input: `http://localhost
 			@debug {
-			  query showdebug=1
+				query showdebug=1
 			}
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
 			input: `http://localhost
 			@debug {
-			  query bad format
+				query bad format
 			}
 			`,
-			expectWarn:  false,
+			expectError: true,
+		},
+		{
+			input: `http://localhost
+			@debug {
+				not {
+					path /somepath*
+				}
+			}
+			`,
+			expectError: false,
+		},
+		{
+			input: `http://localhost
+			@debug {
+				not path /somepath*
+			}
+			`,
+			expectError: false,
+		},
+		{
+			input: `http://localhost
+			@debug not path /somepath*
+			`,
+			expectError: false,
+		},
+		{
+			input: `@matcher {
+				path /matcher-not-allowed/outside-of-site-block/*
+			}
+			http://localhost
+			`,
 			expectError: true,
 		},
 	} {
@@ -36,12 +65,7 @@ func TestServerType(t *testing.T) {
 			ServerType: ServerType{},
 		}
 
-		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
-
-		if len(warnings) > 0 != tc.expectWarn {
-			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
-			continue
-		}
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
 
 		if err != nil != tc.expectError {
 			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
@@ -83,7 +107,6 @@ func TestSpecificity(t *testing.T) {
 func TestGlobalOptions(t *testing.T) {
 	for i, tc := range []struct {
 		input       string
-		expectWarn  bool
 		expectError bool
 	}{
 		{
@@ -93,7 +116,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -103,7 +125,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -113,7 +134,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -125,7 +145,54 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
+			expectError: true,
+		},
+		{
+			input: `
+				{
+					admin {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 127.0.0.1:2020 {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 192.168.1.1:2020 127.0.0.1:2020 {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectError: true,
+		},
+		{
+			input: `
+				{
+					admin off {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
 			expectError: true,
 		},
 	} {
@@ -134,12 +201,7 @@ func TestGlobalOptions(t *testing.T) {
 			ServerType: ServerType{},
 		}
 
-		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
-
-		if len(warnings) > 0 != tc.expectWarn {
-			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
-			continue
-		}
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
 
 		if err != nil != tc.expectError {
 			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)

caddyconfig/httpcaddyfile/options.go

@@ -16,14 +16,43 @@ package httpcaddyfile
 
 import (
 	"strconv"
-	"time"
 
 	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
 )
 
-func parseOptHTTPPort(d *caddyfile.Dispenser) (int, error) {
+func init() {
+	RegisterGlobalOption("debug", parseOptTrue)
+	RegisterGlobalOption("http_port", parseOptHTTPPort)
+	RegisterGlobalOption("https_port", parseOptHTTPSPort)
+	RegisterGlobalOption("grace_period", parseOptDuration)
+	RegisterGlobalOption("default_sni", parseOptSingleString)
+	RegisterGlobalOption("order", parseOptOrder)
+	RegisterGlobalOption("storage", parseOptStorage)
+	RegisterGlobalOption("storage_clean_interval", parseOptDuration)
+	RegisterGlobalOption("acme_ca", parseOptSingleString)
+	RegisterGlobalOption("acme_ca_root", parseOptSingleString)
+	RegisterGlobalOption("acme_dns", parseOptACMEDNS)
+	RegisterGlobalOption("acme_eab", parseOptACMEEAB)
+	RegisterGlobalOption("cert_issuer", parseOptCertIssuer)
+	RegisterGlobalOption("email", parseOptSingleString)
+	RegisterGlobalOption("admin", parseOptAdmin)
+	RegisterGlobalOption("on_demand_tls", parseOptOnDemand)
+	RegisterGlobalOption("local_certs", parseOptTrue)
+	RegisterGlobalOption("key_type", parseOptSingleString)
+	RegisterGlobalOption("auto_https", parseOptAutoHTTPS)
+	RegisterGlobalOption("servers", parseServerOptions)
+	RegisterGlobalOption("ocsp_stapling", parseOCSPStaplingOptions)
+	RegisterGlobalOption("log", parseLogOptions)
+}
+
+func parseOptTrue(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) { return true, nil }
+
+func parseOptHTTPPort(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	var httpPort int
 	for d.Next() {
 		var httpPortStr string
@@ -39,7 +68,7 @@ func parseOptHTTPPort(d *caddyfile.Dispenser) (int, error) {
 	return httpPort, nil
 }
 
-func parseOptHTTPSPort(d *caddyfile.Dispenser) (int, error) {
+func parseOptHTTPSPort(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	var httpsPort int
 	for d.Next() {
 		var httpsPortStr string
@@ -55,11 +84,7 @@ func parseOptHTTPSPort(d *caddyfile.Dispenser) (int, error) {
 	return httpsPort, nil
 }
 
-func parseOptExperimentalHTTP3(d *caddyfile.Dispenser) (bool, error) {
-	return true, nil
-}
-
-func parseOptOrder(d *caddyfile.Dispenser) ([]string, error) {
+func parseOptOrder(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	newOrder := directiveOrder
 
 	for d.Next() {
@@ -135,35 +160,110 @@ func parseOptOrder(d *caddyfile.Dispenser) ([]string, error) {
 	return newOrder, nil
 }
 
-func parseOptStorage(d *caddyfile.Dispenser) (caddy.StorageConverter, error) {
-	if !d.Next() {
+func parseOptStorage(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	if !d.Next() { // consume option name
 		return nil, d.ArgErr()
 	}
-	args := d.RemainingArgs()
-	if len(args) != 1 {
+	if !d.Next() { // get storage module name
 		return nil, d.ArgErr()
 	}
-	modName := args[0]
-	mod, err := caddy.GetModule("caddy.storage." + modName)
-	if err != nil {
-		return nil, d.Errf("getting storage module '%s': %v", modName, err)
-	}
-	unm, ok := mod.New().(caddyfile.Unmarshaler)
-	if !ok {
-		return nil, d.Errf("storage module '%s' is not a Caddyfile unmarshaler", mod.ID)
-	}
-	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
+	modID := "caddy.storage." + d.Val()
+	unm, err := caddyfile.UnmarshalModule(d, modID)
 	if err != nil {
 		return nil, err
 	}
 	storage, ok := unm.(caddy.StorageConverter)
 	if !ok {
-		return nil, d.Errf("module %s is not a StorageConverter", mod.ID)
+		return nil, d.Errf("module %s is not a caddy.StorageConverter", modID)
 	}
 	return storage, nil
 }
 
-func parseOptSingleString(d *caddyfile.Dispenser) (string, error) {
+func parseOptDuration(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	if !d.Next() { // consume option name
+		return nil, d.ArgErr()
+	}
+	if !d.Next() { // get duration value
+		return nil, d.ArgErr()
+	}
+	dur, err := caddy.ParseDuration(d.Val())
+	if err != nil {
+		return nil, err
+	}
+	return caddy.Duration(dur), nil
+}
+
+func parseOptACMEDNS(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	if !d.Next() { // consume option name
+		return nil, d.ArgErr()
+	}
+	if !d.Next() { // get DNS module name
+		return nil, d.ArgErr()
+	}
+	modID := "dns.providers." + d.Val()
+	unm, err := caddyfile.UnmarshalModule(d, modID)
+	if err != nil {
+		return nil, err
+	}
+	prov, ok := unm.(certmagic.ACMEDNSProvider)
+	if !ok {
+		return nil, d.Errf("module %s (%T) is not a certmagic.ACMEDNSProvider", modID, unm)
+	}
+	return prov, nil
+}
+
+func parseOptACMEEAB(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	eab := new(acme.EAB)
+	for d.Next() {
+		if d.NextArg() {
+			return nil, d.ArgErr()
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "key_id":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				eab.KeyID = d.Val()
+
+			case "mac_key":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				eab.MACKey = d.Val()
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
+		}
+	}
+	return eab, nil
+}
+
+func parseOptCertIssuer(d *caddyfile.Dispenser, existing interface{}) (interface{}, error) {
+	var issuers []certmagic.Issuer
+	if existing != nil {
+		issuers = existing.([]certmagic.Issuer)
+	}
+	for d.Next() { // consume option name
+		if !d.Next() { // get issuer module name
+			return nil, d.ArgErr()
+		}
+		modID := "tls.issuance." + d.Val()
+		unm, err := caddyfile.UnmarshalModule(d, modID)
+		if err != nil {
+			return nil, err
+		}
+		iss, ok := unm.(certmagic.Issuer)
+		if !ok {
+			return nil, d.Errf("module %s (%T) is not a certmagic.Issuer", modID, unm)
+		}
+		issuers = append(issuers, iss)
+	}
+	return issuers, nil
+}
+
+func parseOptSingleString(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	d.Next() // consume parameter name
 	if !d.Next() {
 		return "", d.ArgErr()
@@ -175,21 +275,43 @@ func parseOptSingleString(d *caddyfile.Dispenser) (string, error) {
 	return val, nil
 }
 
-func parseOptAdmin(d *caddyfile.Dispenser) (string, error) {
-	if d.Next() {
-		var listenAddress string
-		if !d.AllArgs(&listenAddress) {
-			return "", d.ArgErr()
+func parseOptAdmin(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	adminCfg := new(caddy.AdminConfig)
+	for d.Next() {
+		if d.NextArg() {
+			listenAddress := d.Val()
+			if listenAddress == "off" {
+				adminCfg.Disabled = true
+				if d.Next() { // Do not accept any remaining options including block
+					return nil, d.Err("No more option is allowed after turning off admin config")
+				}
+			} else {
+				adminCfg.Listen = listenAddress
+				if d.NextArg() { // At most 1 arg is allowed
+					return nil, d.ArgErr()
+				}
+			}
 		}
-		if listenAddress == "" {
-			listenAddress = caddy.DefaultAdminListen
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "enforce_origin":
+				adminCfg.EnforceOrigin = true
+
+			case "origins":
+				adminCfg.Origins = d.RemainingArgs()
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
 		}
-		return listenAddress, nil
 	}
-	return "", nil
+	if adminCfg.Listen == "" && !adminCfg.Disabled {
+		adminCfg.Listen = caddy.DefaultAdminListen
+	}
+	return adminCfg, nil
 }
 
-func parseOptOnDemand(d *caddyfile.Dispenser) (*caddytls.OnDemandConfig, error) {
+func parseOptOnDemand(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	var ond *caddytls.OnDemandConfig
 	for d.Next() {
 		if d.NextArg() {
@@ -210,7 +332,7 @@ func parseOptOnDemand(d *caddyfile.Dispenser) (*caddytls.OnDemandConfig, error)
 				if !d.NextArg() {
 					return nil, d.ArgErr()
 				}
-				dur, err := time.ParseDuration(d.Val())
+				dur, err := caddy.ParseDuration(d.Val())
 				if err != nil {
 					return nil, err
 				}
@@ -248,3 +370,84 @@ func parseOptOnDemand(d *caddyfile.Dispenser) (*caddytls.OnDemandConfig, error)
 	}
 	return ond, nil
 }
+
+func parseOptAutoHTTPS(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	d.Next() // consume parameter name
+	if !d.Next() {
+		return "", d.ArgErr()
+	}
+	val := d.Val()
+	if d.Next() {
+		return "", d.ArgErr()
+	}
+	if val != "off" && val != "disable_redirects" && val != "ignore_loaded_certs" {
+		return "", d.Errf("auto_https must be one of 'off', 'disable_redirects' or 'ignore_loaded_certs'")
+	}
+	return val, nil
+}
+
+func parseServerOptions(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	return unmarshalCaddyfileServerOptions(d)
+}
+
+func parseOCSPStaplingOptions(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	d.Next() // consume option name
+	var val string
+	if !d.AllArgs(&val) {
+		return nil, d.ArgErr()
+	}
+	if val != "off" {
+		return nil, d.Errf("invalid argument '%s'", val)
+	}
+	return certmagic.OCSPConfig{
+		DisableStapling: val == "off",
+	}, nil
+}
+
+// parseLogOptions parses the global log option. Syntax:
+//
+//     log [name] {
+//         output  <writer_module> ...
+//         format  <encoder_module> ...
+//         level   <level>
+//         include <namespaces...>
+//         exclude <namespaces...>
+//     }
+//
+// When the name argument is unspecified, this directive modifies the default
+// logger.
+//
+func parseLogOptions(d *caddyfile.Dispenser, existingVal interface{}) (interface{}, error) {
+	currentNames := make(map[string]struct{})
+	if existingVal != nil {
+		innerVals, ok := existingVal.([]ConfigValue)
+		if !ok {
+			return nil, d.Errf("existing log values of unexpected type: %T", existingVal)
+		}
+		for _, rawVal := range innerVals {
+			val, ok := rawVal.Value.(namedCustomLog)
+			if !ok {
+				return nil, d.Errf("existing log value of unexpected type: %T", existingVal)
+			}
+			currentNames[val.name] = struct{}{}
+		}
+	}
+
+	var warnings []caddyconfig.Warning
+	// Call out the same parser that handles server-specific log configuration.
+	configValues, err := parseLogHelper(
+		Helper{
+			Dispenser: d,
+			warnings:  &warnings,
+		},
+		currentNames,
+	)
+	if err != nil {
+		return nil, err
+	}
+	if len(warnings) > 0 {
+		return nil, d.Errf("warnings found in parsing global log options: %+v", warnings)
+	}
+
+	return configValues, nil
+}

caddyconfig/httpcaddyfile/options_test.go

@@ -0,0 +1,64 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	_ "github.com/caddyserver/caddy/v2/modules/logging"
+)
+
+func TestGlobalLogOptionSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		output      string
+		expectError bool
+	}{
+		// NOTE: Additional test cases of successful Caddyfile parsing
+		// are present in: caddytest/integration/caddyfile_adapt/
+		{
+			input: `{
+				log default
+			}
+			`,
+			output:      `{}`,
+			expectError: false,
+		},
+		{
+			input: `{
+				log example {
+					output file foo.log
+				}
+				log example {
+					format json
+				}
+			}
+			`,
+			expectError: true,
+		},
+		{
+			input: `{
+				log example /foo {
+					output file foo.log
+				}
+			}
+			`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		out, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %v", i, tc.expectError, err)
+			continue
+		}
+
+		if string(out) != tc.output {
+			t.Errorf("Test %d error output mismatch Expected: %s, got %s", i, tc.output, out)
+		}
+	}
+}

caddyconfig/httpcaddyfile/pkiapp.go

@@ -0,0 +1,41 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package httpcaddyfile
+
+import (
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/modules/caddypki"
+)
+
+func (st ServerType) buildPKIApp(
+	pairings []sbAddrAssociation,
+	options map[string]interface{},
+	warnings []caddyconfig.Warning,
+) (*caddypki.PKI, []caddyconfig.Warning, error) {
+
+	pkiApp := &caddypki.PKI{CAs: make(map[string]*caddypki.CA)}
+
+	for _, p := range pairings {
+		for _, sblock := range p.serverBlocks {
+			// find all the CAs that were defined and add them to the app config
+			for _, caCfgValue := range sblock.pile["pki.ca"] {
+				ca := caCfgValue.Value.(*caddypki.CA)
+				pkiApp.CAs[ca.ID] = ca
+			}
+		}
+	}
+
+	return pkiApp, warnings, nil
+}

caddyconfig/httpcaddyfile/serveroptions.go

@@ -0,0 +1,228 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package httpcaddyfile
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
+	"github.com/dustin/go-humanize"
+)
+
+// serverOptions collects server config overrides parsed from Caddyfile global options
+type serverOptions struct {
+	// If set, will only apply these options to servers that contain a
+	// listener address that matches exactly. If empty, will apply to all
+	// servers that were not already matched by another serverOptions.
+	ListenerAddress string
+
+	// These will all map 1:1 to the caddyhttp.Server struct
+	ListenerWrappersRaw []json.RawMessage
+	ReadTimeout         caddy.Duration
+	ReadHeaderTimeout   caddy.Duration
+	WriteTimeout        caddy.Duration
+	IdleTimeout         caddy.Duration
+	MaxHeaderBytes      int
+	AllowH2C            bool
+	ExperimentalHTTP3   bool
+	StrictSNIHost       *bool
+}
+
+func unmarshalCaddyfileServerOptions(d *caddyfile.Dispenser) (interface{}, error) {
+	serverOpts := serverOptions{}
+	for d.Next() {
+		if d.NextArg() {
+			serverOpts.ListenerAddress = d.Val()
+			if d.NextArg() {
+				return nil, d.ArgErr()
+			}
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "listener_wrappers":
+				for nesting := d.Nesting(); d.NextBlock(nesting); {
+					modID := "caddy.listeners." + d.Val()
+					unm, err := caddyfile.UnmarshalModule(d, modID)
+					if err != nil {
+						return nil, err
+					}
+					listenerWrapper, ok := unm.(caddy.ListenerWrapper)
+					if !ok {
+						return nil, fmt.Errorf("module %s (%T) is not a listener wrapper", modID, unm)
+					}
+					jsonListenerWrapper := caddyconfig.JSONModuleObject(
+						listenerWrapper,
+						"wrapper",
+						listenerWrapper.(caddy.Module).CaddyModule().ID.Name(),
+						nil,
+					)
+					serverOpts.ListenerWrappersRaw = append(serverOpts.ListenerWrappersRaw, jsonListenerWrapper)
+				}
+
+			case "timeouts":
+				for nesting := d.Nesting(); d.NextBlock(nesting); {
+					switch d.Val() {
+					case "read_body":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, d.Errf("parsing read_body timeout duration: %v", err)
+						}
+						serverOpts.ReadTimeout = caddy.Duration(dur)
+
+					case "read_header":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, d.Errf("parsing read_header timeout duration: %v", err)
+						}
+						serverOpts.ReadHeaderTimeout = caddy.Duration(dur)
+
+					case "write":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, d.Errf("parsing write timeout duration: %v", err)
+						}
+						serverOpts.WriteTimeout = caddy.Duration(dur)
+
+					case "idle":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, d.Errf("parsing idle timeout duration: %v", err)
+						}
+						serverOpts.IdleTimeout = caddy.Duration(dur)
+
+					default:
+						return nil, d.Errf("unrecognized timeouts option '%s'", d.Val())
+					}
+				}
+
+			case "max_header_size":
+				var sizeStr string
+				if !d.AllArgs(&sizeStr) {
+					return nil, d.ArgErr()
+				}
+				size, err := humanize.ParseBytes(sizeStr)
+				if err != nil {
+					return nil, d.Errf("parsing max_header_size: %v", err)
+				}
+				serverOpts.MaxHeaderBytes = int(size)
+
+			case "protocol":
+				for nesting := d.Nesting(); d.NextBlock(nesting); {
+					switch d.Val() {
+					case "allow_h2c":
+						if d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						serverOpts.AllowH2C = true
+
+					case "experimental_http3":
+						if d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						serverOpts.ExperimentalHTTP3 = true
+
+					case "strict_sni_host":
+						if d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						trueBool := true
+						serverOpts.StrictSNIHost = &trueBool
+
+					default:
+						return nil, d.Errf("unrecognized protocol option '%s'", d.Val())
+					}
+				}
+
+			default:
+				return nil, d.Errf("unrecognized servers option '%s'", d.Val())
+			}
+		}
+	}
+	return serverOpts, nil
+}
+
+// applyServerOptions sets the server options on the appropriate servers
+func applyServerOptions(
+	servers map[string]*caddyhttp.Server,
+	options map[string]interface{},
+	warnings *[]caddyconfig.Warning,
+) error {
+	// If experimental HTTP/3 is enabled, enable it on each server.
+	// We already know there won't be a conflict with serverOptions because
+	// we validated earlier that "experimental_http3" cannot be set at the same
+	// time as "servers"
+	if enableH3, ok := options["experimental_http3"].(bool); ok && enableH3 {
+		*warnings = append(*warnings, caddyconfig.Warning{Message: "the 'experimental_http3' global option is deprecated, please use the 'servers > protocol > experimental_http3' option instead"})
+		for _, srv := range servers {
+			srv.ExperimentalHTTP3 = true
+		}
+	}
+
+	serverOpts, ok := options["servers"].([]serverOptions)
+	if !ok {
+		return nil
+	}
+
+	for _, server := range servers {
+		// find the options that apply to this server
+		opts := func() *serverOptions {
+			for _, entry := range serverOpts {
+				if entry.ListenerAddress == "" {
+					return &entry
+				}
+				for _, listener := range server.Listen {
+					if entry.ListenerAddress == listener {
+						return &entry
+					}
+				}
+			}
+			return nil
+		}()
+
+		// if none apply, then move to the next server
+		if opts == nil {
+			continue
+		}
+
+		// set all the options
+		server.ListenerWrappersRaw = opts.ListenerWrappersRaw
+		server.ReadTimeout = opts.ReadTimeout
+		server.ReadHeaderTimeout = opts.ReadHeaderTimeout
+		server.WriteTimeout = opts.WriteTimeout
+		server.IdleTimeout = opts.IdleTimeout
+		server.MaxHeaderBytes = opts.MaxHeaderBytes
+		server.AllowH2C = opts.AllowH2C
+		server.ExperimentalHTTP3 = opts.ExperimentalHTTP3
+		server.StrictSNIHost = opts.StrictSNIHost
+	}
+
+	return nil
+}

caddyconfig/httpcaddyfile/tlsapp.go

@@ -16,14 +16,19 @@ package httpcaddyfile
 
 import (
 	"bytes"
+	"encoding/json"
 	"fmt"
 	"reflect"
 	"sort"
+	"strconv"
+	"strings"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
 	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
 )
 
 func (st ServerType) buildTLSApp(
@@ -35,25 +40,38 @@ func (st ServerType) buildTLSApp(
 	tlsApp := &caddytls.TLS{CertificatesRaw: make(caddy.ModuleMap)}
 	var certLoaders []caddytls.CertificateLoader
 
-	// count how many server blocks have a key with no host,
-	// and find all hosts that share a server block with a
-	// hostless key, so that they don't get forgotten/omitted
+	httpPort := strconv.Itoa(caddyhttp.DefaultHTTPPort)
+	if hp, ok := options["http_port"].(int); ok {
+		httpPort = strconv.Itoa(hp)
+	}
+	httpsPort := strconv.Itoa(caddyhttp.DefaultHTTPSPort)
+	if hsp, ok := options["https_port"].(int); ok {
+		httpsPort = strconv.Itoa(hsp)
+	}
+
+	// count how many server blocks have a TLS-enabled key with
+	// no host, and find all hosts that share a server block with
+	// a hostless key, so that they don't get forgotten/omitted
 	// by auto-HTTPS (since they won't appear in route matchers)
-	var serverBlocksWithHostlessKey int
-	hostsSharedWithHostlessKey := make(map[string]struct{})
+	var serverBlocksWithTLSHostlessKey int
+	httpsHostsSharedWithHostlessKey := make(map[string]struct{})
 	for _, pair := range pairings {
 		for _, sb := range pair.serverBlocks {
 			for _, addr := range sb.keys {
 				if addr.Host == "" {
-					serverBlocksWithHostlessKey++
+					// this address has no hostname, but if it's explicitly set
+					// to HTTPS, then we need to count it as being TLS-enabled
+					if addr.Scheme == "https" || addr.Port == httpsPort {
+						serverBlocksWithTLSHostlessKey++
+					}
 					// this server block has a hostless key, now
 					// go through and add all the hosts to the set
 					for _, otherAddr := range sb.keys {
 						if otherAddr.Original == addr.Original {
 							continue
 						}
-						if otherAddr.Host != "" {
-							hostsSharedWithHostlessKey[addr.Host] = struct{}{}
+						if otherAddr.Host != "" && otherAddr.Scheme != "http" && otherAddr.Port != httpPort {
+							httpsHostsSharedWithHostlessKey[otherAddr.Host] = struct{}{}
 						}
 					}
 					break
@@ -62,121 +80,161 @@ func (st ServerType) buildTLSApp(
 		}
 	}
 
+	// a catch-all automation policy is used as a "default" for all subjects that
+	// don't have custom configuration explicitly associated with them; this
+	// is only to add if the global settings or defaults are non-empty
 	catchAllAP, err := newBaseAutomationPolicy(options, warnings, false)
 	if err != nil {
 		return nil, warnings, err
 	}
+	if catchAllAP != nil {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, catchAllAP)
+	}
 
 	for _, p := range pairings {
+		// avoid setting up TLS automation policies for a server that is HTTP-only
+		if !listenersUseAnyPortOtherThan(p.addresses, httpPort) {
+			continue
+		}
+
 		for _, sblock := range p.serverBlocks {
 			// get values that populate an automation policy for this block
-			var ap *caddytls.AutomationPolicy
+			ap, err := newBaseAutomationPolicy(options, warnings, true)
+			if err != nil {
+				return nil, warnings, err
+			}
 
-			sblockHosts := sblock.hostsFromKeys(false, false)
-			if len(sblockHosts) == 0 {
+			sblockHosts := sblock.hostsFromKeys(false)
+			if len(sblockHosts) == 0 && catchAllAP != nil {
 				ap = catchAllAP
 			}
 
 			// on-demand tls
 			if _, ok := sblock.pile["tls.on_demand"]; ok {
-				if ap == nil {
-					var err error
-					ap, err = newBaseAutomationPolicy(options, warnings, true)
-					if err != nil {
-						return nil, warnings, err
-					}
-				}
 				ap.OnDemand = true
 			}
 
+			if keyTypeVals, ok := sblock.pile["tls.key_type"]; ok {
+				ap.KeyType = keyTypeVals[0].Value.(string)
+			}
+
 			// certificate issuers
 			if issuerVals, ok := sblock.pile["tls.cert_issuer"]; ok {
+				var issuers []certmagic.Issuer
 				for _, issuerVal := range issuerVals {
-					issuer := issuerVal.Value.(certmagic.Issuer)
-					if ap == nil {
-						var err error
-						ap, err = newBaseAutomationPolicy(options, warnings, true)
-						if err != nil {
-							return nil, warnings, err
+					issuers = append(issuers, issuerVal.Value.(certmagic.Issuer))
+				}
+				if ap == catchAllAP && !reflect.DeepEqual(ap.Issuers, issuers) {
+					return nil, warnings, fmt.Errorf("automation policy from site block is also default/catch-all policy because of key without hostname, and the two are in conflict: %#v != %#v", ap.Issuers, issuers)
+				}
+				ap.Issuers = issuers
+			}
+
+			// custom bind host
+			for _, cfgVal := range sblock.pile["bind"] {
+				for _, iss := range ap.Issuers {
+					// if an issuer was already configured and it is NOT an ACME issuer,
+					// skip, since we intend to adjust only ACME issuers; ensure we
+					// include any issuer that embeds/wraps an underlying ACME issuer
+					var acmeIssuer *caddytls.ACMEIssuer
+					if acmeWrapper, ok := iss.(acmeCapable); ok {
+						acmeIssuer = acmeWrapper.GetACMEIssuer()
+					}
+					if acmeIssuer == nil {
+						continue
+					}
+
+					// proceed to configure the ACME issuer's bind host, without
+					// overwriting any existing settings
+					if acmeIssuer.Challenges == nil {
+						acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
+					}
+					if acmeIssuer.Challenges.BindHost == "" {
+						// only binding to one host is supported
+						var bindHost string
+						if bindHosts, ok := cfgVal.Value.([]string); ok && len(bindHosts) > 0 {
+							bindHost = bindHosts[0]
 						}
+						acmeIssuer.Challenges.BindHost = bindHost
 					}
-					encoded := caddyconfig.JSONModuleObject(issuer, "module", issuer.(caddy.Module).CaddyModule().ID.Name(), &warnings)
-					if ap == catchAllAP && ap.IssuerRaw != nil && !bytes.Equal(ap.IssuerRaw, encoded) {
-						return nil, warnings, fmt.Errorf("conflicting issuer configuration: %s != %s", ap.IssuerRaw, encoded)
-					}
-					ap.IssuerRaw = encoded
 				}
 			}
 
-			if ap != nil {
-				// first make sure this block is allowed to create an automation policy;
-				// doing so is forbidden if it has a key with no host (i.e. ":443")
-				// and if there is a different server block that also has a key with no
-				// host -- since a key with no host matches any host, we need its
-				// associated automation policy to have an empty Subjects list, i.e. no
-				// host filter, which is indistinguishable between the two server blocks
-				// because automation is not done in the context of a particular server...
-				// this is an example of a poor mapping from Caddyfile to JSON but that's
-				// the least-leaky abstraction I could figure out
-				if len(sblockHosts) == 0 {
-					if serverBlocksWithHostlessKey > 1 {
-						// this server block and at least one other has a key with no host,
-						// making the two indistinguishable; it is misleading to define such
-						// a policy within one server block since it actually will apply to
-						// others as well
-						return nil, warnings, fmt.Errorf("cannot make a TLS automation policy from a server block that has a host-less address when there are other server block addresses lacking a host")
+			// first make sure this block is allowed to create an automation policy;
+			// doing so is forbidden if it has a key with no host (i.e. ":443")
+			// and if there is a different server block that also has a key with no
+			// host -- since a key with no host matches any host, we need its
+			// associated automation policy to have an empty Subjects list, i.e. no
+			// host filter, which is indistinguishable between the two server blocks
+			// because automation is not done in the context of a particular server...
+			// this is an example of a poor mapping from Caddyfile to JSON but that's
+			// the least-leaky abstraction I could figure out
+			if len(sblockHosts) == 0 {
+				if serverBlocksWithTLSHostlessKey > 1 {
+					// this server block and at least one other has a key with no host,
+					// making the two indistinguishable; it is misleading to define such
+					// a policy within one server block since it actually will apply to
+					// others as well
+					return nil, warnings, fmt.Errorf("cannot make a TLS automation policy from a server block that has a host-less address when there are other TLS-enabled server block addresses lacking a host")
+				}
+				if catchAllAP == nil {
+					// this server block has a key with no hosts, but there is not yet
+					// a catch-all automation policy (probably because no global options
+					// were set), so this one becomes it
+					catchAllAP = ap
+				}
+			}
+
+			// associate our new automation policy with this server block's hosts
+			ap.Subjects = sblockHosts
+			sort.Strings(ap.Subjects) // solely for deterministic test results
+
+			// if a combination of public and internal names were given
+			// for this same server block and no issuer was specified, we
+			// need to separate them out in the automation policies so
+			// that the internal names can use the internal issuer and
+			// the other names can use the default/public/ACME issuer
+			var ap2 *caddytls.AutomationPolicy
+			if len(ap.Issuers) == 0 {
+				var internal, external []string
+				for _, s := range ap.Subjects {
+					if !certmagic.SubjectQualifiesForCert(s) {
+						return nil, warnings, fmt.Errorf("subject does not qualify for certificate: '%s'", s)
 					}
-					if catchAllAP == nil {
-						// this server block has a key with no hosts, but there is not yet
-						// a catch-all automation policy (probably because no global options
-						// were set), so this one becomes it
-						catchAllAP = ap
+					// we don't use certmagic.SubjectQualifiesForPublicCert() because of one nuance:
+					// names like *.*.tld that may not qualify for a public certificate are actually
+					// fine when used with OnDemand, since OnDemand (currently) does not obtain
+					// wildcards (if it ever does, there will be a separate config option to enable
+					// it that we would need to check here) since the hostname is known at handshake;
+					// and it is unexpected to switch to internal issuer when the user wants to get
+					// regular certificates on-demand for a class of certs like *.*.tld.
+					if !certmagic.SubjectIsIP(s) && !certmagic.SubjectIsInternal(s) && (strings.Count(s, "*.") < 2 || ap.OnDemand) {
+						external = append(external, s)
+					} else {
+						internal = append(internal, s)
 					}
 				}
-
-				// associate our new automation policy with this server block's hosts,
-				// unless, of course, the server block has a key with no hosts, in which
-				// case its automation policy becomes or blends with the default/global
-				// automation policy because, of necessity, it applies to all hostnames
-				// (i.e. it has no Subjects filter) -- in that case, we'll append it last
-				if ap != catchAllAP {
-					ap.Subjects = sblockHosts
-
-					// if a combination of public and internal names were given
-					// for this same server block and no issuer was specified, we
-					// need to separate them out in the automation policies so
-					// that the internal names can use the internal issuer and
-					// the other names can use the default/public/ACME issuer
-					var ap2 *caddytls.AutomationPolicy
-					if ap.Issuer == nil {
-						var internal, external []string
-						for _, s := range ap.Subjects {
-							if certmagic.SubjectQualifiesForPublicCert(s) {
-								external = append(external, s)
-							} else {
-								internal = append(internal, s)
-							}
-						}
-						if len(external) > 0 && len(internal) > 0 {
-							ap.Subjects = external
-							apCopy := *ap
-							ap2 = &apCopy
-							ap2.Subjects = internal
-							ap2.IssuerRaw = caddyconfig.JSONModuleObject(caddytls.InternalIssuer{}, "module", "internal", &warnings)
-						}
-					}
-					if tlsApp.Automation == nil {
-						tlsApp.Automation = new(caddytls.AutomationConfig)
-					}
-					tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap)
-					if ap2 != nil {
-						tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap2)
-					}
+				if len(external) > 0 && len(internal) > 0 {
+					ap.Subjects = external
+					apCopy := *ap
+					ap2 = &apCopy
+					ap2.Subjects = internal
+					ap2.IssuersRaw = []json.RawMessage{caddyconfig.JSONModuleObject(caddytls.InternalIssuer{}, "module", "internal", &warnings)}
 				}
 			}
+			if tlsApp.Automation == nil {
+				tlsApp.Automation = new(caddytls.AutomationConfig)
+			}
+			tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap)
+			if ap2 != nil {
+				tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap2)
+			}
 
 			// certificate loaders
-			if clVals, ok := sblock.pile["tls.certificate_loader"]; ok {
+			if clVals, ok := sblock.pile["tls.cert_loader"]; ok {
 				for _, clVal := range clVals {
 					certLoaders = append(certLoaders, clVal.Value.(caddytls.CertificateLoader))
 				}
@@ -202,7 +260,7 @@ func (st ServerType) buildTLSApp(
 				}
 				clVal := reflect.ValueOf(cl)
 				for i := 0; i < clVal.Len(); i++ {
-					combined = reflect.Append(reflect.Value(combined), clVal.Index(i))
+					combined = reflect.Append(combined, clVal.Index(i))
 				}
 				loadersByName[name] = combined.Interface().(caddytls.CertificateLoader)
 			}
@@ -220,29 +278,91 @@ func (st ServerType) buildTLSApp(
 		tlsApp.Automation.OnDemand = onDemand
 	}
 
-	// if there is a global/catch-all automation policy, ensure it goes last
-	if catchAllAP != nil {
+	// set the storage clean interval if configured
+	if storageCleanInterval, ok := options["storage_clean_interval"].(caddy.Duration); ok {
 		if tlsApp.Automation == nil {
 			tlsApp.Automation = new(caddytls.AutomationConfig)
 		}
-		tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, catchAllAP)
+		tlsApp.Automation.StorageCleanInterval = storageCleanInterval
 	}
 
 	// if any hostnames appear on the same server block as a key with
 	// no host, they will not be used with route matchers because the
 	// hostless key matches all hosts, therefore, it wouldn't be
 	// considered for auto-HTTPS, so we need to make sure those hosts
-	// are manually considered for managed certificates
+	// are manually considered for managed certificates; we also need
+	// to make sure that any of these names which are internal-only
+	// get internal certificates by default rather than ACME
 	var al caddytls.AutomateLoader
-	for h := range hostsSharedWithHostlessKey {
+	internalAP := &caddytls.AutomationPolicy{
+		IssuersRaw: []json.RawMessage{json.RawMessage(`{"module":"internal"}`)},
+	}
+	for h := range httpsHostsSharedWithHostlessKey {
 		al = append(al, h)
+		if !certmagic.SubjectQualifiesForPublicCert(h) {
+			internalAP.Subjects = append(internalAP.Subjects, h)
+		}
 	}
 	if len(al) > 0 {
 		tlsApp.CertificatesRaw["automate"] = caddyconfig.JSON(al, &warnings)
 	}
+	if len(internalAP.Subjects) > 0 {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, internalAP)
+	}
 
-	// do a little verification & cleanup
+	// if there are any global options set for issuers (ACME ones in particular), make sure they
+	// take effect in every automation policy that does not have any issuers
 	if tlsApp.Automation != nil {
+		globalEmail := options["email"]
+		globalACMECA := options["acme_ca"]
+		globalACMECARoot := options["acme_ca_root"]
+		globalACMEDNS := options["acme_dns"]
+		globalACMEEAB := options["acme_eab"]
+		hasGlobalACMEDefaults := globalEmail != nil || globalACMECA != nil || globalACMECARoot != nil || globalACMEDNS != nil || globalACMEEAB != nil
+		if hasGlobalACMEDefaults {
+			for _, ap := range tlsApp.Automation.Policies {
+				if len(ap.Issuers) == 0 {
+					ap.Issuers = caddytls.DefaultIssuers()
+
+					// if a specific endpoint is configured, can't use multiple default issuers
+					if globalACMECA != nil {
+						if strings.Contains(globalACMECA.(string), "zerossl") {
+							ap.Issuers = []certmagic.Issuer{&caddytls.ZeroSSLIssuer{ACMEIssuer: new(caddytls.ACMEIssuer)}}
+						} else {
+							ap.Issuers = []certmagic.Issuer{new(caddytls.ACMEIssuer)}
+						}
+					}
+				}
+			}
+		}
+	}
+
+	// finalize and verify policies; do cleanup
+	if tlsApp.Automation != nil {
+		for i, ap := range tlsApp.Automation.Policies {
+			// ensure all issuers have global defaults filled in
+			for j, issuer := range ap.Issuers {
+				err := fillInGlobalACMEDefaults(issuer, options)
+				if err != nil {
+					return nil, warnings, fmt.Errorf("filling in global issuer defaults for AP %d, issuer %d: %v", i, j, err)
+				}
+			}
+
+			// encode all issuer values we created, so they will be rendered in the output
+			if len(ap.Issuers) > 0 && ap.IssuersRaw == nil {
+				for _, iss := range ap.Issuers {
+					issuerName := iss.(caddy.Module).CaddyModule().ID.Name()
+					ap.IssuersRaw = append(ap.IssuersRaw, caddyconfig.JSONModuleObject(iss, "module", issuerName, &warnings))
+				}
+			}
+		}
+
+		// consolidate automation policies that are the exact same
+		tlsApp.Automation.Policies = consolidateAutomationPolicies(tlsApp.Automation.Policies)
+
 		// ensure automation policies don't overlap subjects (this should be
 		// an error at provision-time as well, but catch it in the adapt phase
 		// for convenience)
@@ -256,26 +376,70 @@ func (st ServerType) buildTLSApp(
 			}
 		}
 
-		// consolidate automation policies that are the exact same
-		tlsApp.Automation.Policies = consolidateAutomationPolicies(tlsApp.Automation.Policies)
+		// if nothing remains, remove any excess values to clean up the resulting config
+		if len(tlsApp.Automation.Policies) == 0 {
+			tlsApp.Automation.Policies = nil
+		}
+		if reflect.DeepEqual(tlsApp.Automation, new(caddytls.AutomationConfig)) {
+			tlsApp.Automation = nil
+		}
 	}
 
 	return tlsApp, warnings, nil
 }
 
+type acmeCapable interface{ GetACMEIssuer() *caddytls.ACMEIssuer }
+
+func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]interface{}) error {
+	acmeWrapper, ok := issuer.(acmeCapable)
+	if !ok {
+		return nil
+	}
+	acmeIssuer := acmeWrapper.GetACMEIssuer()
+	if acmeIssuer == nil {
+		return nil
+	}
+
+	globalEmail := options["email"]
+	globalACMECA := options["acme_ca"]
+	globalACMECARoot := options["acme_ca_root"]
+	globalACMEDNS := options["acme_dns"]
+	globalACMEEAB := options["acme_eab"]
+
+	if globalEmail != nil && acmeIssuer.Email == "" {
+		acmeIssuer.Email = globalEmail.(string)
+	}
+	if globalACMECA != nil && acmeIssuer.CA == "" {
+		acmeIssuer.CA = globalACMECA.(string)
+	}
+	if globalACMECARoot != nil && !sliceContains(acmeIssuer.TrustedRootsPEMFiles, globalACMECARoot.(string)) {
+		acmeIssuer.TrustedRootsPEMFiles = append(acmeIssuer.TrustedRootsPEMFiles, globalACMECARoot.(string))
+	}
+	if globalACMEDNS != nil && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.DNS == nil) {
+		acmeIssuer.Challenges = &caddytls.ChallengesConfig{
+			DNS: &caddytls.DNSChallengeConfig{
+				ProviderRaw: caddyconfig.JSONModuleObject(globalACMEDNS, "name", globalACMEDNS.(caddy.Module).CaddyModule().ID.Name(), nil),
+			},
+		}
+	}
+	if globalACMEEAB != nil && acmeIssuer.ExternalAccount == nil {
+		acmeIssuer.ExternalAccount = globalACMEEAB.(*acme.EAB)
+	}
+	return nil
+}
+
 // newBaseAutomationPolicy returns a new TLS automation policy that gets
 // its values from the global options map. It should be used as the base
 // for any other automation policies. A nil policy (and no error) will be
 // returned if there are no default/global options. However, if always is
 // true, a non-nil value will always be returned (unless there is an error).
 func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddyconfig.Warning, always bool) (*caddytls.AutomationPolicy, error) {
-	acmeCA, hasACMECA := options["acme_ca"]
-	acmeDNS, hasACMEDNS := options["acme_dns"]
-	acmeCARoot, hasACMECARoot := options["acme_ca_root"]
-	email, hasEmail := options["email"]
-	localCerts, hasLocalCerts := options["local_certs"]
+	issuers, hasIssuers := options["cert_issuer"]
+	_, hasLocalCerts := options["local_certs"]
+	keyType, hasKeyType := options["key_type"]
+	ocspStapling, hasOCSPStapling := options["ocsp_stapling"]
 
-	hasGlobalAutomationOpts := hasACMECA || hasACMEDNS || hasACMECARoot || hasEmail || hasLocalCerts
+	hasGlobalAutomationOpts := hasIssuers || hasLocalCerts || hasKeyType || hasOCSPStapling
 
 	// if there are no global options related to automation policies
 	// set, then we can just return right away
@@ -287,35 +451,24 @@ func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddycon
 	}
 
 	ap := new(caddytls.AutomationPolicy)
+	if hasKeyType {
+		ap.KeyType = keyType.(string)
+	}
 
-	if localCerts != nil {
-		// internal issuer enabled trumps any ACME configurations; useful in testing
-		ap.IssuerRaw = caddyconfig.JSONModuleObject(caddytls.InternalIssuer{}, "module", "internal", &warnings)
-	} else {
-		if acmeCA == nil {
-			acmeCA = ""
-		}
-		if email == nil {
-			email = ""
-		}
-		mgr := caddytls.ACMEIssuer{
-			CA:    acmeCA.(string),
-			Email: email.(string),
-		}
-		if acmeDNS != nil {
-			provName := acmeDNS.(string)
-			dnsProvModule, err := caddy.GetModule("tls.dns." + provName)
-			if err != nil {
-				return nil, fmt.Errorf("getting DNS provider module named '%s': %v", provName, err)
-			}
-			mgr.Challenges = &caddytls.ChallengesConfig{
-				DNSRaw: caddyconfig.JSONModuleObject(dnsProvModule.New(), "provider", provName, &warnings),
-			}
-		}
-		if acmeCARoot != nil {
-			mgr.TrustedRootsPEMFiles = []string{acmeCARoot.(string)}
-		}
-		ap.IssuerRaw = caddyconfig.JSONModuleObject(mgr, "module", "acme", &warnings)
+	if hasIssuers && hasLocalCerts {
+		return nil, fmt.Errorf("global options are ambiguous: local_certs is confusing when combined with cert_issuer, because local_certs is also a specific kind of issuer")
+	}
+
+	if hasIssuers {
+		ap.Issuers = issuers.([]certmagic.Issuer)
+	} else if hasLocalCerts {
+		ap.Issuers = []certmagic.Issuer{new(caddytls.InternalIssuer)}
+	}
+
+	if hasOCSPStapling {
+		ocspConfig := ocspStapling.(certmagic.OCSPConfig)
+		ap.DisableOCSPStapling = ocspConfig.DisableStapling
+		ap.OCSPOverrides = ocspConfig.ResponderOverrides
 	}
 
 	return ap, nil
@@ -324,16 +477,39 @@ func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddycon
 // consolidateAutomationPolicies combines automation policies that are the same,
 // for a cleaner overall output.
 func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls.AutomationPolicy {
-	for i := 0; i < len(aps); i++ {
-		for j := 0; j < len(aps); j++ {
-			if j == i {
-				continue
-			}
+	// sort from most specific to least specific; we depend on this ordering
+	sort.SliceStable(aps, func(i, j int) bool {
+		if automationPolicyIsSubset(aps[i], aps[j]) {
+			return true
+		}
+		if automationPolicyIsSubset(aps[j], aps[i]) {
+			return false
+		}
+		return len(aps[i].Subjects) > len(aps[j].Subjects)
+	})
 
+	emptyAPCount := 0
+	// compute the number of empty policies (disregarding subjects) - see #4128
+	emptyAP := new(caddytls.AutomationPolicy)
+	for i := 0; i < len(aps); i++ {
+		emptyAP.Subjects = aps[i].Subjects
+		if reflect.DeepEqual(aps[i], emptyAP) {
+			emptyAPCount++
+		}
+	}
+	// If all policies are empty, we can return nil, as there is no need to set any policy
+	if emptyAPCount == len(aps) {
+		return nil
+	}
+
+	// remove or combine duplicate policies
+outer:
+	for i := 0; i < len(aps); i++ {
+		// compare only with next policies; we sorted by specificity so we must not delete earlier policies
+		for j := i + 1; j < len(aps); j++ {
 			// if they're exactly equal in every way, just keep one of them
 			if reflect.DeepEqual(aps[i], aps[j]) {
 				aps = append(aps[:j], aps[j+1:]...)
-				i--
 				break
 			}
 
@@ -343,30 +519,77 @@ func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls
 			// otherwise the one without any subjects (a catch-all) would be
 			// eaten up by the one with subjects; and if both have subjects, we
 			// need to combine their lists
-			if bytes.Equal(aps[i].IssuerRaw, aps[j].IssuerRaw) &&
+			if reflect.DeepEqual(aps[i].IssuersRaw, aps[j].IssuersRaw) &&
 				bytes.Equal(aps[i].StorageRaw, aps[j].StorageRaw) &&
 				aps[i].MustStaple == aps[j].MustStaple &&
 				aps[i].KeyType == aps[j].KeyType &&
 				aps[i].OnDemand == aps[j].OnDemand &&
 				aps[i].RenewalWindowRatio == aps[j].RenewalWindowRatio {
-				if len(aps[i].Subjects) == 0 && len(aps[j].Subjects) > 0 {
-					aps = append(aps[:j], aps[j+1:]...)
-				} else if len(aps[i].Subjects) > 0 && len(aps[j].Subjects) == 0 {
-					aps = append(aps[:i], aps[i+1:]...)
+				if len(aps[i].Subjects) > 0 && len(aps[j].Subjects) == 0 {
+					// later policy (at j) has no subjects ("catch-all"), so we can
+					// remove the identical-but-more-specific policy that comes first
+					// AS LONG AS it is not shadowed by another policy before it; e.g.
+					// if policy i is for example.com, policy i+1 is '*.com', and policy
+					// j is catch-all, we cannot remove policy i because that would
+					// cause example.com to be served by the less specific policy for
+					// '*.com', which might be different (yes we've seen this happen)
+					if automationPolicyShadows(i, aps) >= j {
+						aps = append(aps[:i], aps[i+1:]...)
+						i--
+						continue outer
+					}
 				} else {
-					aps[i].Subjects = append(aps[i].Subjects, aps[j].Subjects...)
+					// avoid repeated subjects
+					for _, subj := range aps[j].Subjects {
+						if !sliceContains(aps[i].Subjects, subj) {
+							aps[i].Subjects = append(aps[i].Subjects, subj)
+						}
+					}
 					aps = append(aps[:j], aps[j+1:]...)
+					j--
 				}
-				i--
-				break
 			}
 		}
 	}
 
-	// ensure any catch-all policies go last
-	sort.SliceStable(aps, func(i, j int) bool {
-		return len(aps[i].Subjects) > len(aps[j].Subjects)
-	})
-
 	return aps
 }
+
+// automationPolicyIsSubset returns true if a's subjects are a subset
+// of b's subjects.
+func automationPolicyIsSubset(a, b *caddytls.AutomationPolicy) bool {
+	if len(b.Subjects) == 0 {
+		return true
+	}
+	if len(a.Subjects) == 0 {
+		return false
+	}
+	for _, aSubj := range a.Subjects {
+		var inSuperset bool
+		for _, bSubj := range b.Subjects {
+			if certmagic.MatchWildcard(aSubj, bSubj) {
+				inSuperset = true
+				break
+			}
+		}
+		if !inSuperset {
+			return false
+		}
+	}
+	return true
+}
+
+// automationPolicyShadows returns the index of a policy that aps[i] shadows;
+// in other words, for all policies after position i, if that policy covers
+// the same subjects but is less specific, that policy's position is returned,
+// or -1 if no shadowing is found. For example, if policy i is for
+// "foo.example.com" and policy i+2 is for "*.example.com", then i+2 will be
+// returned, since that policy is shadowed by i, which is in front.
+func automationPolicyShadows(i int, aps []*caddytls.AutomationPolicy) int {
+	for j := i + 1; j < len(aps); j++ {
+		if automationPolicyIsSubset(aps[i], aps[j]) {
+			return j
+		}
+	}
+	return -1
+}

caddyconfig/httpcaddyfile/tlsapp_test.go

@@ -0,0 +1,56 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/modules/caddytls"
+)
+
+func TestAutomationPolicyIsSubset(t *testing.T) {
+	for i, test := range []struct {
+		a, b   []string
+		expect bool
+	}{
+		{
+			a:      []string{"example.com"},
+			b:      []string{},
+			expect: true,
+		},
+		{
+			a:      []string{},
+			b:      []string{"example.com"},
+			expect: false,
+		},
+		{
+			a:      []string{"foo.example.com"},
+			b:      []string{"*.example.com"},
+			expect: true,
+		},
+		{
+			a:      []string{"foo.example.com"},
+			b:      []string{"foo.example.com"},
+			expect: true,
+		},
+		{
+			a:      []string{"foo.example.com"},
+			b:      []string{"example.com"},
+			expect: false,
+		},
+		{
+			a:      []string{"example.com", "foo.example.com"},
+			b:      []string{"*.com", "*.*.com"},
+			expect: true,
+		},
+		{
+			a:      []string{"example.com", "foo.example.com"},
+			b:      []string{"*.com"},
+			expect: false,
+		},
+	} {
+		apA := &caddytls.AutomationPolicy{Subjects: test.a}
+		apB := &caddytls.AutomationPolicy{Subjects: test.b}
+		if actual := automationPolicyIsSubset(apA, apB); actual != test.expect {
+			t.Errorf("Test %d: Expected %t but got %t (A: %v  B: %v)", i, test.expect, actual, test.a, test.b)
+		}
+	}
+}

caddyconfig/httploader.go

@@ -0,0 +1,151 @@
+package caddyconfig
+
+import (
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"time"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func init() {
+	caddy.RegisterModule(HTTPLoader{})
+}
+
+// HTTPLoader can load Caddy configs over HTTP(S). It can adapt the config
+// based on the Content-Type header of the HTTP response.
+type HTTPLoader struct {
+	// The method for the request. Default: GET
+	Method string `json:"method,omitempty"`
+
+	// The URL of the request.
+	URL string `json:"url,omitempty"`
+
+	// HTTP headers to add to the request.
+	Headers http.Header `json:"header,omitempty"`
+
+	// Maximum time allowed for a complete connection and request.
+	Timeout caddy.Duration `json:"timeout,omitempty"`
+
+	TLS *struct {
+		// Present this instance's managed remote identity credentials to the server.
+		UseServerIdentity bool `json:"use_server_identity,omitempty"`
+
+		// PEM-encoded client certificate filename to present to the server.
+		ClientCertificateFile string `json:"client_certificate_file,omitempty"`
+
+		// PEM-encoded key to use with the client certificate.
+		ClientCertificateKeyFile string `json:"client_certificate_key_file,omitempty"`
+
+		// List of PEM-encoded CA certificate files to add to the same trust
+		// store as RootCAPool (or root_ca_pool in the JSON).
+		RootCAPEMFiles []string `json:"root_ca_pem_files,omitempty"`
+	} `json:"tls,omitempty"`
+}
+
+// CaddyModule returns the Caddy module information.
+func (HTTPLoader) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "caddy.config_loaders.http",
+		New: func() caddy.Module { return new(HTTPLoader) },
+	}
+}
+
+// LoadConfig loads a Caddy config.
+func (hl HTTPLoader) LoadConfig(ctx caddy.Context) ([]byte, error) {
+	client, err := hl.makeClient(ctx)
+	if err != nil {
+		return nil, err
+	}
+
+	method := hl.Method
+	if method == "" {
+		method = http.MethodGet
+	}
+
+	req, err := http.NewRequest(method, hl.URL, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header = hl.Headers
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode >= 400 {
+		return nil, fmt.Errorf("server responded with HTTP %d", resp.StatusCode)
+	}
+
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	result, warnings, err := adaptByContentType(resp.Header.Get("Content-Type"), body)
+	if err != nil {
+		return nil, err
+	}
+	for _, warn := range warnings {
+		ctx.Logger(hl).Warn(warn.String())
+	}
+
+	return result, nil
+}
+
+func (hl HTTPLoader) makeClient(ctx caddy.Context) (*http.Client, error) {
+	client := &http.Client{
+		Timeout: time.Duration(hl.Timeout),
+	}
+
+	if hl.TLS != nil {
+		var tlsConfig *tls.Config
+
+		// client authentication
+		if hl.TLS.UseServerIdentity {
+			certs, err := ctx.IdentityCredentials(ctx.Logger(hl))
+			if err != nil {
+				return nil, fmt.Errorf("getting server identity credentials: %v", err)
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.Certificates = certs
+		} else if hl.TLS.ClientCertificateFile != "" && hl.TLS.ClientCertificateKeyFile != "" {
+			cert, err := tls.LoadX509KeyPair(hl.TLS.ClientCertificateFile, hl.TLS.ClientCertificateKeyFile)
+			if err != nil {
+				return nil, err
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.Certificates = []tls.Certificate{cert}
+		}
+
+		// trusted server certs
+		if len(hl.TLS.RootCAPEMFiles) > 0 {
+			rootPool := x509.NewCertPool()
+			for _, pemFile := range hl.TLS.RootCAPEMFiles {
+				pemData, err := ioutil.ReadFile(pemFile)
+				if err != nil {
+					return nil, fmt.Errorf("failed reading ca cert: %v", err)
+				}
+				rootPool.AppendCertsFromPEM(pemData)
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.RootCAs = rootPool
+		}
+
+		client.Transport = &http.Transport{TLSClientConfig: tlsConfig}
+	}
+
+	return client, nil
+}
+
+var _ caddy.ConfigLoader = (*HTTPLoader)(nil)

caddyconfig/load.go

@@ -69,8 +69,8 @@ func (al adminLoad) Routes() []caddy.AdminRoute {
 func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	if r.Method != http.MethodPost {
 		return caddy.APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method not allowed"),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method not allowed"),
 		}
 	}
 
@@ -81,8 +81,8 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	_, err := io.Copy(buf, r.Body)
 	if err != nil {
 		return caddy.APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("reading request body: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("reading request body: %v", err),
 		}
 	}
 	body := buf.Bytes()
@@ -90,45 +90,21 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	// if the config is formatted other than Caddy's native
 	// JSON, we need to adapt it before loading it
 	if ctHeader := r.Header.Get("Content-Type"); ctHeader != "" {
-		ct, _, err := mime.ParseMediaType(ctHeader)
+		result, warnings, err := adaptByContentType(ctHeader, body)
 		if err != nil {
 			return caddy.APIError{
-				Code: http.StatusBadRequest,
-				Err:  fmt.Errorf("invalid Content-Type: %v", err),
+				HTTPStatus: http.StatusBadRequest,
+				Err:        err,
 			}
 		}
-		if !strings.HasSuffix(ct, "/json") {
-			slashIdx := strings.Index(ct, "/")
-			if slashIdx < 0 {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("malformed Content-Type"),
-				}
-			}
-			adapterName := ct[slashIdx+1:]
-			cfgAdapter := GetAdapter(adapterName)
-			if cfgAdapter == nil {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("unrecognized config adapter '%s'", adapterName),
-				}
-			}
-			result, warnings, err := cfgAdapter.Adapt(body, nil)
+		if len(warnings) > 0 {
+			respBody, err := json.Marshal(warnings)
 			if err != nil {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("adapting config using %s adapter: %v", adapterName, err),
-				}
+				caddy.Log().Named("admin.api.load").Error(err.Error())
 			}
-			if len(warnings) > 0 {
-				respBody, err := json.Marshal(warnings)
-				if err != nil {
-					caddy.Log().Named("admin.api.load").Error(err.Error())
-				}
-				_, _ = w.Write(respBody)
-			}
-			body = result
+			_, _ = w.Write(respBody)
 		}
+		body = result
 	}
 
 	forceReload := r.Header.Get("Cache-Control") == "must-revalidate"
@@ -136,8 +112,8 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	err = caddy.Load(body, forceReload)
 	if err != nil {
 		return caddy.APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("loading config: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("loading config: %v", err),
 		}
 	}
 
@@ -146,6 +122,47 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	return nil
 }
 
+// adaptByContentType adapts body to Caddy JSON using the adapter specified by contenType.
+// If contentType is empty or ends with "/json", the input will be returned, as a no-op.
+func adaptByContentType(contentType string, body []byte) ([]byte, []Warning, error) {
+	// assume JSON as the default
+	if contentType == "" {
+		return body, nil, nil
+	}
+
+	ct, _, err := mime.ParseMediaType(contentType)
+	if err != nil {
+		return nil, nil, caddy.APIError{
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("invalid Content-Type: %v", err),
+		}
+	}
+
+	// if already JSON, no need to adapt
+	if strings.HasSuffix(ct, "/json") {
+		return body, nil, nil
+	}
+
+	// adapter name should be suffix of MIME type
+	slashIdx := strings.Index(ct, "/")
+	if slashIdx < 0 {
+		return nil, nil, fmt.Errorf("malformed Content-Type")
+	}
+
+	adapterName := ct[slashIdx+1:]
+	cfgAdapter := GetAdapter(adapterName)
+	if cfgAdapter == nil {
+		return nil, nil, fmt.Errorf("unrecognized config adapter '%s'", adapterName)
+	}
+
+	result, warnings, err := cfgAdapter.Adapt(body, nil)
+	if err != nil {
+		return nil, nil, fmt.Errorf("adapting config using %s adapter: %v", adapterName, err)
+	}
+
+	return result, warnings, nil
+}
+
 var bufPool = sync.Pool{
 	New: func() interface{} {
 		return new(bytes.Buffer)

caddytest/caddy.ca.cer

@@ -0,0 +1,20 @@
+-----BEGIN CERTIFICATE-----
+MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQEL
+BQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkw
+ODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcN
+AQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU
+7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl0
+3WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45t
+wOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNx
+tdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTU
+ApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAd
+BgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS
+2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5u
+NY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkq
+hkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfK
+D66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEO
+fG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnk
+oNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZ
+ks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdle
+Ih6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ==
+-----END CERTIFICATE-----

caddytest/caddytest.go

@@ -11,14 +11,18 @@ import (
 	"log"
 	"net"
 	"net/http"
+	"net/http/cookiejar"
 	"os"
 	"path"
+	"reflect"
 	"regexp"
 	"runtime"
 	"strings"
 	"testing"
 	"time"
 
+	"github.com/aryann/difflib"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
 	caddycmd "github.com/caddyserver/caddy/v2/cmd"
 
 	// plug in Caddy modules here
@@ -31,12 +35,18 @@ type Defaults struct {
 	AdminPort int
 	// Certificates we expect to be loaded before attempting to run the tests
 	Certifcates []string
+	// TestRequestTimeout is the time to wait for a http request to
+	TestRequestTimeout time.Duration
+	// LoadRequestTimeout is the time to wait for the config to be loaded against the caddy server
+	LoadRequestTimeout time.Duration
 }
 
 // Default testing values
 var Default = Defaults{
-	AdminPort:   2019,
-	Certifcates: []string{"/caddy.localhost.crt", "/caddy.localhost.key"},
+	AdminPort:          2019,
+	Certifcates:        []string{"/caddy.localhost.crt", "/caddy.localhost.key"},
+	TestRequestTimeout: 5 * time.Second,
+	LoadRequestTimeout: 5 * time.Second,
 }
 
 var (
@@ -44,6 +54,32 @@ var (
 	matchCert = regexp.MustCompile(`(/[\w\d\.]+\.crt)`)
 )
 
+// Tester represents an instance of a test client.
+type Tester struct {
+	Client       *http.Client
+	configLoaded bool
+	t            *testing.T
+}
+
+// NewTester will create a new testing client with an attached cookie jar
+func NewTester(t *testing.T) *Tester {
+
+	jar, err := cookiejar.New(nil)
+	if err != nil {
+		t.Fatalf("failed to create cookiejar: %s", err)
+	}
+
+	return &Tester{
+		Client: &http.Client{
+			Transport: CreateTestingTransport(),
+			Jar:       jar,
+			Timeout:   Default.TestRequestTimeout,
+		},
+		configLoaded: false,
+		t:            t,
+	}
+}
+
 type configLoadError struct {
 	Response string
 }
@@ -57,52 +93,58 @@ func timeElapsed(start time.Time, name string) {
 
 // InitServer this will configure the server with a configurion of a specific
 // type. The configType must be either "json" or the adapter type.
-func InitServer(t *testing.T, rawConfig string, configType string) {
-	if err := initServer(t, rawConfig, configType); errors.Is(err, &configLoadError{}) {
-		t.Logf("failed to load config: %s", err)
-		t.Fail()
+func (tc *Tester) InitServer(rawConfig string, configType string) {
+
+	if err := tc.initServer(rawConfig, configType); err != nil {
+		tc.t.Logf("failed to load config: %s", err)
+		tc.t.Fail()
+	}
+	if err := tc.ensureConfigRunning(rawConfig, configType); err != nil {
+		tc.t.Logf("failed ensurng config is running: %s", err)
+		tc.t.Fail()
 	}
 }
 
 // InitServer this will configure the server with a configurion of a specific
 // type. The configType must be either "json" or the adapter type.
-func initServer(t *testing.T, rawConfig string, configType string) error {
+func (tc *Tester) initServer(rawConfig string, configType string) error {
 
 	if testing.Short() {
-		t.SkipNow()
+		tc.t.SkipNow()
 		return nil
 	}
 
 	err := validateTestPrerequisites()
 	if err != nil {
-		t.Skipf("skipping tests as failed integration prerequisites. %s", err)
+		tc.t.Skipf("skipping tests as failed integration prerequisites. %s", err)
 		return nil
 	}
 
-	t.Cleanup(func() {
-		if t.Failed() {
+	tc.t.Cleanup(func() {
+		if tc.t.Failed() && tc.configLoaded {
+
 			res, err := http.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
 			if err != nil {
-				t.Log("unable to read the current config")
+				tc.t.Log("unable to read the current config")
+				return
 			}
 			defer res.Body.Close()
-			body, err := ioutil.ReadAll(res.Body)
+			body, _ := ioutil.ReadAll(res.Body)
 
 			var out bytes.Buffer
-			json.Indent(&out, body, "", "  ")
-			t.Logf("----------- failed with config -----------\n%s", out.String())
+			_ = json.Indent(&out, body, "", "  ")
+			tc.t.Logf("----------- failed with config -----------\n%s", out.String())
 		}
 	})
 
 	rawConfig = prependCaddyFilePath(rawConfig)
 	client := &http.Client{
-		Timeout: time.Second * 2,
+		Timeout: Default.LoadRequestTimeout,
 	}
-
 	start := time.Now()
 	req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/load", Default.AdminPort), strings.NewReader(rawConfig))
 	if err != nil {
-		t.Errorf("failed to create request. %s", err)
+		tc.t.Errorf("failed to create request. %s", err)
 		return err
 	}
 
@@ -114,7 +156,7 @@ func initServer(t *testing.T, rawConfig string, configType string) error {
 
 	res, err := client.Do(req)
 	if err != nil {
-		t.Errorf("unable to contact caddy server. %s", err)
+		tc.t.Errorf("unable to contact caddy server. %s", err)
 		return err
 	}
 	timeElapsed(start, "caddytest: config load time")
@@ -122,7 +164,7 @@ func initServer(t *testing.T, rawConfig string, configType string) error {
 	defer res.Body.Close()
 	body, err := ioutil.ReadAll(res.Body)
 	if err != nil {
-		t.Errorf("unable to read response. %s", err)
+		tc.t.Errorf("unable to read response. %s", err)
 		return err
 	}
 
@@ -130,23 +172,61 @@ func initServer(t *testing.T, rawConfig string, configType string) error {
 		return configLoadError{Response: string(body)}
 	}
 
+	tc.configLoaded = true
 	return nil
 }
 
-var hasValidated bool
-var arePrerequisitesValid bool
-
-func validateTestPrerequisites() error {
-
-	if hasValidated {
-		if !arePrerequisitesValid {
-			return errors.New("caddy integration prerequisites failed. see first error")
+func (tc *Tester) ensureConfigRunning(rawConfig string, configType string) error {
+	expectedBytes := []byte(prependCaddyFilePath(rawConfig))
+	if configType != "json" {
+		adapter := caddyconfig.GetAdapter(configType)
+		if adapter == nil {
+			return fmt.Errorf("adapter of config type is missing: %s", configType)
 		}
-		return nil
+		expectedBytes, _, _ = adapter.Adapt([]byte(rawConfig), nil)
 	}
 
-	hasValidated = true
-	arePrerequisitesValid = false
+	var expected interface{}
+	err := json.Unmarshal(expectedBytes, &expected)
+	if err != nil {
+		return err
+	}
+
+	client := &http.Client{
+		Timeout: Default.LoadRequestTimeout,
+	}
+
+	fetchConfig := func(client *http.Client) interface{} {
+		resp, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
+		if err != nil {
+			return nil
+		}
+		defer resp.Body.Close()
+		actualBytes, err := ioutil.ReadAll(resp.Body)
+		if err != nil {
+			return nil
+		}
+		var actual interface{}
+		err = json.Unmarshal(actualBytes, &actual)
+		if err != nil {
+			return nil
+		}
+		return actual
+	}
+
+	for retries := 4; retries > 0; retries-- {
+		if reflect.DeepEqual(expected, fetchConfig(client)) {
+			return nil
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	tc.t.Errorf("POSTed configuration isn't active")
+	return errors.New("EnsureConfigRunning: POSTed configuration isn't active")
+}
+
+// validateTestPrerequisites ensures the certificates are available in the
+// designated path and Caddy sub-process is running.
+func validateTestPrerequisites() error {
 
 	// check certificates are found
 	for _, certName := range Default.Certifcates {
@@ -162,31 +242,26 @@ func validateTestPrerequisites() error {
 			caddycmd.Main()
 		}()
 
-		// wait for caddy to start
-		retries := 4
-		for ; retries > 0 && isCaddyAdminRunning() != nil; retries-- {
+		// wait for caddy to start serving the initial config
+		for retries := 4; retries > 0 && isCaddyAdminRunning() != nil; retries-- {
 			time.Sleep(10 * time.Millisecond)
 		}
 	}
 
-	// assert that caddy is running
-	if err := isCaddyAdminRunning(); err != nil {
-		return err
-	}
-
-	arePrerequisitesValid = true
-	return nil
+	// one more time to return the error
+	return isCaddyAdminRunning()
 }
 
 func isCaddyAdminRunning() error {
 	// assert that caddy is running
 	client := &http.Client{
-		Timeout: time.Second * 2,
+		Timeout: Default.LoadRequestTimeout,
 	}
-	_, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
+	resp, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
 	if err != nil {
-		return errors.New("caddy integration test caddy server not running. Expected to be listening on localhost:2019")
+		return fmt.Errorf("caddy integration test caddy server not running. Expected to be listening on localhost:%d", Default.AdminPort)
 	}
+	resp.Body.Close()
 
 	return nil
 }
@@ -210,8 +285,8 @@ func prependCaddyFilePath(rawConfig string) string {
 	return r
 }
 
-// creates a testing transport that forces call dialing connections to happen locally
-func createTestingTransport() *http.Transport {
+// CreateTestingTransport creates a testing transport that forces call dialing connections to happen locally
+func CreateTestingTransport() *http.Transport {
 
 	dialer := net.Dialer{
 		Timeout:   5 * time.Second,
@@ -234,85 +309,244 @@ func createTestingTransport() *http.Transport {
 		IdleConnTimeout:       90 * time.Second,
 		TLSHandshakeTimeout:   5 * time.Second,
 		ExpectContinueTimeout: 1 * time.Second,
-		TLSClientConfig:       &tls.Config{InsecureSkipVerify: true},
+		TLSClientConfig:       &tls.Config{InsecureSkipVerify: true}, //nolint:gosec
 	}
 }
 
 // AssertLoadError will load a config and expect an error
 func AssertLoadError(t *testing.T, rawConfig string, configType string, expectedError string) {
-	err := initServer(t, rawConfig, configType)
+
+	tc := NewTester(t)
+
+	err := tc.initServer(rawConfig, configType)
 	if !strings.Contains(err.Error(), expectedError) {
 		t.Errorf("expected error \"%s\" but got \"%s\"", expectedError, err.Error())
 	}
 }
 
-// AssertGetResponse request a URI and assert the status code and the body contains a string
-func AssertGetResponse(t *testing.T, requestURI string, statusCode int, expectedBody string) (*http.Response, string) {
-	resp, body := AssertGetResponseBody(t, requestURI, statusCode)
-	if !strings.Contains(body, expectedBody) {
-		t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", requestURI, expectedBody, body)
-	}
-	return resp, string(body)
-}
-
-// AssertGetResponseBody request a URI and assert the status code matches
-func AssertGetResponseBody(t *testing.T, requestURI string, expectedStatusCode int) (*http.Response, string) {
-
-	client := &http.Client{
-		Transport: createTestingTransport(),
-	}
-
-	resp, err := client.Get(requestURI)
-	if err != nil {
-		t.Errorf("failed to call server %s", err)
-		return nil, ""
-	}
-
-	defer resp.Body.Close()
-
-	if expectedStatusCode != resp.StatusCode {
-		t.Errorf("requesting \"%s\" expected status code: %d but got %d", requestURI, expectedStatusCode, resp.StatusCode)
-	}
-
-	body, err := ioutil.ReadAll(resp.Body)
-	if err != nil {
-		t.Errorf("unable to read the response body %s", err)
-		return nil, ""
-	}
-
-	return resp, string(body)
-}
-
 // AssertRedirect makes a request and asserts the redirection happens
-func AssertRedirect(t *testing.T, requestURI string, expectedToLocation string, expectedStatusCode int) *http.Response {
+func (tc *Tester) AssertRedirect(requestURI string, expectedToLocation string, expectedStatusCode int) *http.Response {
 
 	redirectPolicyFunc := func(req *http.Request, via []*http.Request) error {
 		return http.ErrUseLastResponse
 	}
 
-	client := &http.Client{
-		CheckRedirect: redirectPolicyFunc,
-		Transport:     createTestingTransport(),
-	}
+	// using the existing client, we override the check redirect policy for this test
+	old := tc.Client.CheckRedirect
+	tc.Client.CheckRedirect = redirectPolicyFunc
+	defer func() { tc.Client.CheckRedirect = old }()
 
-	resp, err := client.Get(requestURI)
+	resp, err := tc.Client.Get(requestURI)
 	if err != nil {
-		t.Errorf("failed to call server %s", err)
+		tc.t.Errorf("failed to call server %s", err)
 		return nil
 	}
 
 	if expectedStatusCode != resp.StatusCode {
-		t.Errorf("requesting \"%s\" expected status code: %d but got %d", requestURI, expectedStatusCode, resp.StatusCode)
+		tc.t.Errorf("requesting \"%s\" expected status code: %d but got %d", requestURI, expectedStatusCode, resp.StatusCode)
 	}
 
 	loc, err := resp.Location()
 	if err != nil {
-		t.Errorf("requesting \"%s\" expected location: \"%s\" but got error: %s", requestURI, expectedToLocation, err)
+		tc.t.Errorf("requesting \"%s\" expected location: \"%s\" but got error: %s", requestURI, expectedToLocation, err)
 	}
-
-	if expectedToLocation != loc.String() {
-		t.Errorf("requesting \"%s\" expected location: \"%s\" but got \"%s\"", requestURI, expectedToLocation, loc.String())
+	if loc == nil && expectedToLocation != "" {
+		tc.t.Errorf("requesting \"%s\" expected a Location header, but didn't get one", requestURI)
+	}
+	if loc != nil {
+		if expectedToLocation != loc.String() {
+			tc.t.Errorf("requesting \"%s\" expected location: \"%s\" but got \"%s\"", requestURI, expectedToLocation, loc.String())
+		}
 	}
 
 	return resp
 }
+
+// CompareAdapt adapts a config and then compares it against an expected result
+func CompareAdapt(t *testing.T, filename, rawConfig string, adapterName string, expectedResponse string) bool {
+
+	cfgAdapter := caddyconfig.GetAdapter(adapterName)
+	if cfgAdapter == nil {
+		t.Logf("unrecognized config adapter '%s'", adapterName)
+		return false
+	}
+
+	options := make(map[string]interface{})
+
+	result, warnings, err := cfgAdapter.Adapt([]byte(rawConfig), options)
+	if err != nil {
+		t.Logf("adapting config using %s adapter: %v", adapterName, err)
+		return false
+	}
+
+	// prettify results to keep tests human-manageable
+	var prettyBuf bytes.Buffer
+	err = json.Indent(&prettyBuf, result, "", "\t")
+	if err != nil {
+		return false
+	}
+	result = prettyBuf.Bytes()
+
+	if len(warnings) > 0 {
+		for _, w := range warnings {
+			t.Logf("warning: %s:%d: %s: %s", filename, w.Line, w.Directive, w.Message)
+		}
+	}
+
+	diff := difflib.Diff(
+		strings.Split(expectedResponse, "\n"),
+		strings.Split(string(result), "\n"))
+
+	// scan for failure
+	failed := false
+	for _, d := range diff {
+		if d.Delta != difflib.Common {
+			failed = true
+			break
+		}
+	}
+
+	if failed {
+		for _, d := range diff {
+			switch d.Delta {
+			case difflib.Common:
+				fmt.Printf("  %s\n", d.Payload)
+			case difflib.LeftOnly:
+				fmt.Printf(" - %s\n", d.Payload)
+			case difflib.RightOnly:
+				fmt.Printf(" + %s\n", d.Payload)
+			}
+		}
+		return false
+	}
+	return true
+}
+
+// AssertAdapt adapts a config and then tests it against an expected result
+func AssertAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) {
+	ok := CompareAdapt(t, "Caddyfile", rawConfig, adapterName, expectedResponse)
+	if !ok {
+		t.Fail()
+	}
+}
+
+// Generic request functions
+
+func applyHeaders(t *testing.T, req *http.Request, requestHeaders []string) {
+	requestContentType := ""
+	for _, requestHeader := range requestHeaders {
+		arr := strings.SplitAfterN(requestHeader, ":", 2)
+		k := strings.TrimRight(arr[0], ":")
+		v := strings.TrimSpace(arr[1])
+		if k == "Content-Type" {
+			requestContentType = v
+		}
+		t.Logf("Request header: %s => %s", k, v)
+		req.Header.Set(k, v)
+	}
+
+	if requestContentType == "" {
+		t.Logf("Content-Type header not provided")
+	}
+}
+
+// AssertResponseCode will execute the request and verify the status code, returns a response for additional assertions
+func (tc *Tester) AssertResponseCode(req *http.Request, expectedStatusCode int) *http.Response {
+
+	resp, err := tc.Client.Do(req)
+	if err != nil {
+		tc.t.Fatalf("failed to call server %s", err)
+	}
+
+	if expectedStatusCode != resp.StatusCode {
+		tc.t.Errorf("requesting \"%s\" expected status code: %d but got %d", req.RequestURI, expectedStatusCode, resp.StatusCode)
+	}
+
+	return resp
+}
+
+// AssertResponse request a URI and assert the status code and the body contains a string
+func (tc *Tester) AssertResponse(req *http.Request, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	resp := tc.AssertResponseCode(req, expectedStatusCode)
+
+	defer resp.Body.Close()
+	bytes, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		tc.t.Fatalf("unable to read the response body %s", err)
+	}
+
+	body := string(bytes)
+
+	if body != expectedBody {
+		tc.t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", req.RequestURI, expectedBody, body)
+	}
+
+	return resp, body
+}
+
+// Verb specific test functions
+
+// AssertGetResponse GET a URI and expect a statusCode and body text
+func (tc *Tester) AssertGetResponse(requestURI string, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("GET", requestURI, nil)
+	if err != nil {
+		tc.t.Fatalf("unable to create request %s", err)
+	}
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertDeleteResponse request a URI and expect a statusCode and body text
+func (tc *Tester) AssertDeleteResponse(requestURI string, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("DELETE", requestURI, nil)
+	if err != nil {
+		tc.t.Fatalf("unable to create request %s", err)
+	}
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPostResponseBody POST to a URI and assert the response code and body
+func (tc *Tester) AssertPostResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("POST", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPutResponseBody PUT to a URI and assert the response code and body
+func (tc *Tester) AssertPutResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("PUT", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPatchResponseBody PATCH to a URI and assert the response code and body
+func (tc *Tester) AssertPatchResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("PATCH", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}

caddytest/integration/autohttps_test.go

@@ -0,0 +1,105 @@
+package integration
+
+import (
+	"net/http"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestAutoHTTPtoHTTPSRedirectsImplicitPort(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	localhost
+	respond "Yahaha! You found me!"
+  `, "caddyfile")
+
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
+}
+
+func TestAutoHTTPtoHTTPSRedirectsExplicitPortSameAsHTTPSPort(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	localhost:9443
+	respond "Yahaha! You found me!"
+  `, "caddyfile")
+
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
+}
+
+func TestAutoHTTPtoHTTPSRedirectsExplicitPortDifferentFromHTTPSPort(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	localhost:1234
+	respond "Yahaha! You found me!"
+  `, "caddyfile")
+
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost:1234/", http.StatusPermanentRedirect)
+}
+
+func TestAutoHTTPRedirectsWithHTTPListenerFirstInAddresses(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+{
+  "apps": {
+    "http": {
+      "http_port": 9080,
+      "https_port": 9443,
+      "servers": {
+        "ingress_server": {
+          "listen": [
+            ":9080",
+            ":9443"
+          ],
+          "routes": [
+            {
+              "match": [
+                {
+				  "host": ["localhost"]
+                }
+              ]
+            }
+          ]
+        }
+      }
+    }
+  }
+}
+`, "json")
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
+}
+
+func TestAutoHTTPRedirectsInsertedBeforeUserDefinedCatchAll(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+		local_certs
+	}
+	http://:9080 {
+		respond "Foo"
+	}
+	http://baz.localhost:9080 {
+		respond "Baz"
+	}
+	bar.localhost {
+		respond "Bar"
+	}
+  `, "caddyfile")
+	tester.AssertRedirect("http://bar.localhost:9080/", "https://bar.localhost/", http.StatusPermanentRedirect)
+	tester.AssertGetResponse("http://foo.localhost:9080/", 200, "Foo")
+	tester.AssertGetResponse("http://baz.localhost:9080/", 200, "Baz")
+}

caddytest/integration/caddyfile_adapt/auto_https_disable_redirects.txt

@@ -0,0 +1,34 @@
+{
+	auto_https disable_redirects
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"disable_redirects": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/auto_https_ignore_loaded_certs.txt

@@ -0,0 +1,34 @@
+{
+	auto_https ignore_loaded_certs
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"ignore_loaded_certificates": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/auto_https_off.txt

@@ -0,0 +1,37 @@
+{
+	auto_https off
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"tls_connection_policies": [
+						{}
+					],
+					"automatic_https": {
+						"disable": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/encode_options.txt

@@ -0,0 +1,85 @@
+:80
+
+# All the options
+encode gzip zstd {
+	minimum_length 256
+	match {
+		status 2xx 4xx 500
+		header Content-Type text/*
+		header Content-Type application/json*
+		header Content-Type application/javascript*
+		header Content-Type application/xhtml+xml*
+		header Content-Type application/atom+xml*
+		header Content-Type application/rss+xml*
+		header Content-Type image/svg+xml*
+	}
+}
+
+# Long way with a block for each encoding
+encode {
+	zstd
+	gzip 5
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"encodings": {
+										"gzip": {},
+										"zstd": {}
+									},
+									"handler": "encode",
+									"match": {
+										"headers": {
+											"Content-Type": [
+												"text/*",
+												"application/json*",
+												"application/javascript*",
+												"application/xhtml+xml*",
+												"application/atom+xml*",
+												"application/rss+xml*",
+												"image/svg+xml*"
+											]
+										},
+										"status_code": [
+											2,
+											4,
+											500
+										]
+									},
+									"minimum_length": 256,
+									"prefer": [
+										"gzip",
+										"zstd"
+									]
+								},
+								{
+									"encodings": {
+										"gzip": {
+											"level": 5
+										},
+										"zstd": {}
+									},
+									"handler": "encode",
+									"prefer": [
+										"zstd",
+										"gzip"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_precompressed.txt

@@ -0,0 +1,41 @@
+:80
+
+file_server {
+	precompressed zstd br gzip
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "file_server",
+									"hide": [
+										"./Caddyfile"
+									],
+									"precompressed": {
+										"br": {},
+										"gzip": {},
+										"zstd": {}
+									},
+									"precompressed_order": [
+										"zstd",
+										"br",
+										"gzip"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_status.txt

@@ -0,0 +1,112 @@
+localhost
+
+root * /srv
+
+handle /nope* {
+	file_server {
+		status 403
+	}
+}
+
+handle /custom-status* {
+	file_server {
+		status {env.CUSTOM_STATUS}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "vars",
+													"root": "/srv"
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "file_server",
+																	"hide": [
+																		"./Caddyfile"
+																	],
+																	"status_code": "{env.CUSTOM_STATUS}"
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/custom-status*"
+													]
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "file_server",
+																	"hide": [
+																		"./Caddyfile"
+																	],
+																	"status_code": 403
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/nope*"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options.txt

@@ -0,0 +1,77 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	grace_period 5s
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_ca_root /path/to/ca.crt
+
+	email test@example.com
+	admin off
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	local_certs
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"disabled": true
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"grace_period": 5000000000,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						],
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_acme.txt

@@ -0,0 +1,89 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_eab {
+		key_id 4K2scIVbBpNd-78scadB2g
+		mac_key abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh
+	}
+	acme_ca_root /path/to/ca.crt
+	email test@example.com
+	admin off
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	storage_clean_interval 7d
+
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"disabled": true
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuers": [
+							{
+								"ca": "https://example.com",
+								"email": "test@example.com",
+								"external_account": {
+									"key_id": "4K2scIVbBpNd-78scadB2g",
+									"mac_key": "abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh"
+								},
+								"module": "acme",
+								"trusted_roots_pem_files": [
+									"/path/to/ca.crt"
+								]
+							}
+						],
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				},
+				"storage_clean_interval": 604800000000000
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_admin.txt

@@ -0,0 +1,83 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_ca_root /path/to/ca.crt
+
+	email test@example.com
+	admin {
+		origins localhost:2019 [::1]:2019 127.0.0.1:2019 192.168.10.128
+	}
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	local_certs
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"listen": "localhost:2019",
+		"origins": [
+			"localhost:2019",
+			"[::1]:2019",
+			"127.0.0.1:2019",
+			"192.168.10.128"
+		]
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						],
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_log_and_site.txt

@@ -0,0 +1,77 @@
+{
+	log {
+		output file caddy.log
+		include some-log-source
+		exclude admin.api admin2.api
+	}
+	log custom-logger {
+		output file caddy.log
+		level WARN
+		include custom-log-source
+	}
+}
+
+:8884 {
+	log {
+		format json
+		output file access.log
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"custom-logger": {
+				"writer": {
+					"filename": "caddy.log",
+					"output": "file"
+				},
+				"level": "WARN",
+				"include": [
+					"custom-log-source"
+				]
+			},
+			"default": {
+				"writer": {
+					"filename": "caddy.log",
+					"output": "file"
+				},
+				"include": [
+					"some-log-source"
+				],
+				"exclude": [
+					"admin.api",
+					"admin2.api",
+					"custom-log-source",
+					"http.log.access.log0"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "access.log",
+					"output": "file"
+				},
+				"encoder": {
+					"format": "json"
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"logs": {
+						"default_logger_name": "log0"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_log_basic.txt

@@ -0,0 +1,18 @@
+{
+	log {
+		output file foo.log
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"writer": {
+					"filename": "foo.log",
+					"output": "file"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_log_custom.txt

@@ -0,0 +1,39 @@
+{
+	log custom-logger {
+		format filter {
+			wrap console
+			fields {
+				common_log delete
+				request>remote_addr ip_mask {
+					ipv4 24
+					ipv6 32
+				}
+			}
+		}
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"custom-logger": {
+				"encoder": {
+					"fields": {
+						"common_log": {
+							"filter": "delete"
+						},
+						"request\u003eremote_addr": {
+							"filter": "ip_mask",
+							"ipv4_cidr": 24,
+							"ipv6_cidr": 32
+						}
+					},
+					"format": "filter",
+					"wrap": {
+						"format": "console"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_log_multi.txt

@@ -0,0 +1,26 @@
+{
+	log first {
+		output file foo.log
+	}
+	log second {
+		format json
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"first": {
+				"writer": {
+					"filename": "foo.log",
+					"output": "file"
+				}
+			},
+			"second": {
+				"encoder": {
+					"format": "json"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_server_options_multi.txt

@@ -0,0 +1,78 @@
+{
+	servers {
+		timeouts {
+			idle 90s
+		}
+	}
+	servers :80 {
+		timeouts {
+			idle 60s
+		}
+	}
+	servers :443 {
+		timeouts {
+			idle 30s
+		}
+	}
+}
+
+foo.com {
+}
+
+http://bar.com {
+}
+
+:8080 {
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"idle_timeout": 30000000000,
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"foo.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":80"
+					],
+					"idle_timeout": 60000000000,
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"bar.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv2": {
+					"listen": [
+						":8080"
+					],
+					"idle_timeout": 90000000000
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_server_options_single.txt

@@ -0,0 +1,62 @@
+{
+	servers {
+		listener_wrappers {
+			tls
+		}
+		timeouts {
+			read_body 30s
+			read_header 30s
+			write 30s
+			idle 30s
+		}
+		max_header_size 100MB
+		protocol {
+			allow_h2c
+			experimental_http3
+			strict_sni_host
+		}
+	}
+}
+
+foo.com {
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"listener_wrappers": [
+						{
+							"wrapper": "tls"
+						}
+					],
+					"read_timeout": 30000000000,
+					"read_header_timeout": 30000000000,
+					"write_timeout": 30000000000,
+					"idle_timeout": 30000000000,
+					"max_header_bytes": 100000000,
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"foo.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"strict_sni_host": true,
+					"experimental_http3": true,
+					"allow_h2c": true
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/handle_path.txt

@@ -0,0 +1,52 @@
+:80
+handle_path /api/v1/* {
+	respond "API v1"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/api/v1/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"strip_path_prefix": "/api/v1"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"body": "API v1",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/handle_path_sorting.txt

@@ -0,0 +1,105 @@
+:80 {
+	handle /api/* {
+		respond "api"
+	}
+
+	handle_path /static/* {
+		respond "static"
+	}
+
+	handle {
+		respond "handle"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"group": "group3",
+							"match": [
+								{
+									"path": [
+										"/static/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"strip_path_prefix": "/static"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"body": "static",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						},
+						{
+							"group": "group3",
+							"match": [
+								{
+									"path": [
+										"/api/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "api",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						},
+						{
+							"group": "group3",
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "handle",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/header.txt

@@ -0,0 +1,132 @@
+:80 {
+	header Denis "Ritchie"
+	header +Edsger "Dijkstra"
+	header ?John "von Neumann"
+	header -Wolfram
+	header {
+		Grace: "Hopper" # some users habitually suffix field names with a colon
+		+Ray "Solomonoff"
+		?Tim "Berners-Lee"
+		defer
+	}
+	@images path /images/*
+	header @images {
+		Cache-Control "public, max-age=3600, stale-while-revalidate=86400"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/images/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "headers",
+									"response": {
+										"set": {
+											"Cache-Control": [
+												"public, max-age=3600, stale-while-revalidate=86400"
+											]
+										}
+									}
+								}
+							]
+						},
+						{
+							"handle": [
+								{
+									"handler": "headers",
+									"response": {
+										"set": {
+											"Denis": [
+												"Ritchie"
+											]
+										}
+									}
+								},
+								{
+									"handler": "headers",
+									"response": {
+										"add": {
+											"Edsger": [
+												"Dijkstra"
+											]
+										}
+									}
+								},
+								{
+									"handler": "headers",
+									"response": {
+										"require": {
+											"headers": {
+												"John": null
+											}
+										},
+										"set": {
+											"John": [
+												"von Neumann"
+											]
+										}
+									}
+								},
+								{
+									"handler": "headers",
+									"response": {
+										"deferred": true,
+										"delete": [
+											"Wolfram"
+										]
+									}
+								},
+								{
+									"handler": "headers",
+									"response": {
+										"add": {
+											"Ray": [
+												"Solomonoff"
+											]
+										},
+										"deferred": true,
+										"set": {
+											"Grace": [
+												"Hopper"
+											]
+										}
+									}
+								},
+								{
+									"handler": "headers",
+									"response": {
+										"require": {
+											"headers": {
+												"Tim": null
+											}
+										},
+										"set": {
+											"Tim": [
+												"Berners-Lee"
+											]
+										}
+									}
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_hostnames.txt

@@ -0,0 +1,45 @@
+# https://github.com/caddyserver/caddy/issues/3977
+http://* {
+	respond "Hello, world!"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "Hello, world!",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_any_address.txt

@@ -0,0 +1,37 @@
+:80 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/version"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "hello from localhost",
+									"handler": "static_response",
+									"status_code": 200
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_domain.txt

@@ -0,0 +1,54 @@
+http://a.caddy.localhost {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_hostless_block.txt

@@ -0,0 +1,28 @@
+# Issue #4113
+:80, http://example.com {
+	respond "foo"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"body": "foo",
+									"handler": "static_response"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_localhost.txt

@@ -0,0 +1,54 @@
+localhost:80 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_non_standard_port.txt

@@ -0,0 +1,59 @@
+http://a.caddy.localhost:81 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":81"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"a.caddy.localhost"
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/https_on_domain.txt

@@ -0,0 +1,54 @@
+a.caddy.localhost {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_file.txt

@@ -0,0 +1,49 @@
+example.com
+
+import testdata/import_respond.txt Groot Rocket
+import testdata/import_respond.txt you "the confused man"
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "'I am Groot', hears Rocket",
+													"handler": "static_response"
+												},
+												{
+													"body": "'I am you', hears the confused man",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_snippet.txt

@@ -0,0 +1,83 @@
+(logging) {
+	log {
+		output file /var/log/caddy/{args.0}.access.log
+	}
+}
+
+a.example.com {
+	import logging a.example.com
+}
+
+b.example.com {
+	import logging b.example.com
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0",
+					"http.log.access.log1"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "/var/log/caddy/a.example.com.access.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			},
+			"log1": {
+				"writer": {
+					"filename": "/var/log/caddy/b.example.com.access.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log1"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"b.example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"logs": {
+						"logger_names": {
+							"a.example.com": "log0",
+							"b.example.com": "log1"
+						}
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_snippet_env_placeholder.txt

@@ -0,0 +1,31 @@
+(foo) {
+	respond {env.FOO}
+}
+
+:80 {
+	import foo
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"body": "{env.FOO}",
+									"handler": "static_response"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_except_catchall_blocks.txt

@@ -0,0 +1,78 @@
+http://localhost:2020 {
+	log
+	respond 200
+}
+
+:2020 {
+	respond 418
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":2020"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"status_code": 200
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"status_code": 418
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"localhost"
+						]
+					},
+					"logs": {
+						"logger_names": {
+							"localhost:2020": ""
+						},
+						"skip_unmapped_hosts": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_filters.txt

@@ -0,0 +1,70 @@
+:80
+
+log {
+	output stdout
+	format filter {
+		wrap console
+		fields {
+			request>headers>Authorization replace REDACTED
+			request>headers>Server delete
+			request>remote_addr ip_mask {
+				ipv4 24
+				ipv6 32
+			}
+		}
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0"
+				]
+			},
+			"log0": {
+				"writer": {
+					"output": "stdout"
+				},
+				"encoder": {
+					"fields": {
+						"request\u003eheaders\u003eAuthorization": {
+							"filter": "replace",
+							"value": "REDACTED"
+						},
+						"request\u003eheaders\u003eServer": {
+							"filter": "delete"
+						},
+						"request\u003eremote_addr": {
+							"filter": "ip_mask",
+							"ipv4_cidr": 24,
+							"ipv6_cidr": 32
+						}
+					},
+					"format": "filter",
+					"wrap": {
+						"format": "console"
+					}
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"logs": {
+						"default_logger_name": "log0"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_roll_days.txt

@@ -0,0 +1,47 @@
+:80
+
+log {
+	output file /var/log/access.log {
+		roll_size 1gb
+		roll_keep 5
+		roll_keep_for 90d
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "/var/log/access.log",
+					"output": "file",
+					"roll_keep": 5,
+					"roll_keep_days": 90,
+					"roll_size_mb": 954
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"logs": {
+						"default_logger_name": "log0"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/matcher_syntax.txt

@@ -0,0 +1,216 @@
+:80 {
+	@matcher {
+		method GET
+	}
+	respond @matcher "get"
+
+	@matcher2 method POST
+	respond @matcher2 "post"
+
+	@matcher3 not method PUT
+	respond @matcher3 "not put"
+
+	@matcher4 vars "{http.request.uri}" "/vars-matcher"
+	respond @matcher4 "from vars matcher"
+
+	@matcher5 vars_regexp static "{http.request.uri}" `\.([a-f0-9]{6})\.(css|js)$`
+	respond @matcher5 "from vars_regexp matcher with name"
+
+	@matcher6 vars_regexp "{http.request.uri}" `\.([a-f0-9]{6})\.(css|js)$`
+	respond @matcher6 "from vars_regexp matcher without name"
+
+	@matcher7 {
+		header Foo bar
+		header Foo foobar
+		header Bar foo
+	}
+	respond @matcher7 "header matcher merging values of the same field"
+
+	@matcher8 {
+		query foo=bar foo=baz bar=foo
+		query bar=baz
+	}
+	respond @matcher8 "query matcher merging pairs with the same keys"
+
+	@matcher9 {
+		header !Foo
+		header Bar foo
+	}
+	respond @matcher9 "header matcher with null field matcher"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"method": [
+										"GET"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "get",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"method": [
+										"POST"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "post",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"not": [
+										{
+											"method": [
+												"PUT"
+											]
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "not put",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"vars": {
+										"{http.request.uri}": "/vars-matcher"
+									}
+								}
+							],
+							"handle": [
+								{
+									"body": "from vars matcher",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"vars_regexp": {
+										"{http.request.uri}": {
+											"name": "static",
+											"pattern": "\\.([a-f0-9]{6})\\.(css|js)$"
+										}
+									}
+								}
+							],
+							"handle": [
+								{
+									"body": "from vars_regexp matcher with name",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"vars_regexp": {
+										"{http.request.uri}": {
+											"pattern": "\\.([a-f0-9]{6})\\.(css|js)$"
+										}
+									}
+								}
+							],
+							"handle": [
+								{
+									"body": "from vars_regexp matcher without name",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"header": {
+										"Bar": [
+											"foo"
+										],
+										"Foo": [
+											"bar",
+											"foobar"
+										]
+									}
+								}
+							],
+							"handle": [
+								{
+									"body": "header matcher merging values of the same field",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"query": {
+										"bar": [
+											"foo",
+											"baz"
+										],
+										"foo": [
+											"bar",
+											"baz"
+										]
+									}
+								}
+							],
+							"handle": [
+								{
+									"body": "query matcher merging pairs with the same keys",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"header": {
+										"Bar": [
+											"foo"
+										],
+										"Foo": null
+									}
+								}
+							],
+							"handle": [
+								{
+									"body": "header matcher with null field matcher",
+									"handler": "static_response"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/matchers_in_route.txt

@@ -0,0 +1,31 @@
+:80 {
+	route {
+		# unused matchers should not panic
+		# see https://github.com/caddyserver/caddy/issues/3745
+		@matcher1 path /path1
+		@matcher2 path /path2
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "subroute"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/metrics_disable_om.txt

@@ -0,0 +1,36 @@
+:80 {
+	metrics /metrics {
+		disable_openmetrics
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/metrics"
+									]
+								}
+							],
+							"handle": [
+								{
+									"disable_openmetrics": true,
+									"handler": "metrics"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/metrics_syntax.txt

@@ -0,0 +1,33 @@
+:80 {
+	metrics /metrics
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/metrics"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "metrics"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/not_block_merging.txt

@@ -0,0 +1,49 @@
+:80
+
+@test {
+	not {
+		header Abc "123"
+		header Bcd "123"
+	}
+}
+respond @test 403
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"not": [
+										{
+											"header": {
+												"Abc": [
+													"123"
+												],
+												"Bcd": [
+													"123"
+												]
+											}
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "static_response",
+									"status_code": 403
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_expanded_form.txt

@@ -0,0 +1,132 @@
+:8886
+
+route {
+	# Add trailing slash for directory requests
+	@canonicalPath {
+		file {
+			try_files {path}/index.php
+		}
+		not path */
+	}
+	redir @canonicalPath {path}/ 308
+
+	# If the requested file does not exist, try index files
+	@indexFiles {
+		file {
+			try_files {path} {path}/index.php index.php
+			split_path .php
+		}
+	}
+	rewrite @indexFiles {http.matchers.file.relative}
+
+	# Proxy PHP files to the FastCGI responder
+	@phpFiles {
+		path *.php
+	}
+	reverse_proxy @phpFiles 127.0.0.1:9000 {
+		transport fastcgi {
+			split .php
+		}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8886"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"headers": {
+														"Location": [
+															"{http.request.uri.path}/"
+														]
+													},
+													"status_code": 308
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"try_files": [
+															"{http.request.uri.path}/index.php"
+														]
+													},
+													"not": [
+														{
+															"path": [
+																"*/"
+															]
+														}
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"uri": "{http.matchers.file.relative}"
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"split_path": [
+															".php"
+														],
+														"try_files": [
+															"{http.request.uri.path}",
+															"{http.request.uri.path}/index.php",
+															"index.php"
+														]
+													}
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"transport": {
+														"protocol": "fastcgi",
+														"split_path": [
+															".php"
+														]
+													},
+													"upstreams": [
+														{
+															"dial": "127.0.0.1:9000"
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"*.php"
+													]
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_index_off.txt

@@ -0,0 +1,72 @@
+:8884
+
+php_fastcgi localhost:9000 {
+	# some php_fastcgi-specific subdirectives
+	split .php .php5
+	env VAR1 value1
+	env VAR2 value2
+	root /var/www
+	index off
+	dial_timeout 3s
+	read_timeout 10s
+	write_timeout 20s
+
+	# passed through to reverse_proxy (directive order doesn't matter!)
+	lb_policy random
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"*.php",
+										"*.php5"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"load_balancing": {
+										"selection_policy": {
+											"policy": "random"
+										}
+									},
+									"transport": {
+										"dial_timeout": 3000000000,
+										"env": {
+											"VAR1": "value1",
+											"VAR2": "value2"
+										},
+										"protocol": "fastcgi",
+										"read_timeout": 10000000000,
+										"root": "/var/www",
+										"split_path": [
+											".php",
+											".php5"
+										],
+										"write_timeout": 20000000000
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_matcher.txt

@@ -0,0 +1,112 @@
+:8884
+
+@api host example.com
+php_fastcgi @api localhost:9000
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"headers": {
+														"Location": [
+															"{http.request.uri.path}/"
+														]
+													},
+													"status_code": 308
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"try_files": [
+															"{http.request.uri.path}/index.php"
+														]
+													},
+													"not": [
+														{
+															"path": [
+																"*/"
+															]
+														}
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"uri": "{http.matchers.file.relative}"
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"split_path": [
+															".php"
+														],
+														"try_files": [
+															"{http.request.uri.path}",
+															"{http.request.uri.path}/index.php",
+															"index.php"
+														]
+													}
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"transport": {
+														"protocol": "fastcgi",
+														"split_path": [
+															".php"
+														]
+													},
+													"upstreams": [
+														{
+															"dial": "localhost:9000"
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"*.php"
+													]
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_subdirectives.txt

@@ -0,0 +1,118 @@
+:8884
+
+php_fastcgi localhost:9000 {
+	# some php_fastcgi-specific subdirectives
+	split .php .php5
+	env VAR1 value1
+	env VAR2 value2
+	root /var/www
+	index index.php5
+
+	# passed through to reverse_proxy (directive order doesn't matter!)
+	lb_policy random
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}/index.php5"
+										]
+									},
+									"not": [
+										{
+											"path": [
+												"*/"
+											]
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "static_response",
+									"headers": {
+										"Location": [
+											"{http.request.uri.path}/"
+										]
+									},
+									"status_code": 308
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}",
+											"{http.request.uri.path}/index.php5",
+											"index.php5"
+										],
+										"split_path": [
+											".php",
+											".php5"
+										]
+									}
+								}
+							],
+							"handle": [
+								{
+									"handler": "rewrite",
+									"uri": "{http.matchers.file.relative}"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"path": [
+										"*.php",
+										"*.php5"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"load_balancing": {
+										"selection_policy": {
+											"policy": "random"
+										}
+									},
+									"transport": {
+										"env": {
+											"VAR1": "value1",
+											"VAR2": "value2"
+										},
+										"protocol": "fastcgi",
+										"root": "/var/www",
+										"split_path": [
+											".php",
+											".php5"
+										]
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/portless_upstream.txt

@@ -0,0 +1,113 @@
+whoami.example.com {
+	reverse_proxy whoami
+}
+
+app.example.com {
+	reverse_proxy app:80
+}
+unix.example.com {
+	reverse_proxy unix//path/to/socket
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"whoami.example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"upstreams": [
+														{
+															"dial": "whoami:80"
+														}
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"unix.example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"upstreams": [
+														{
+															"dial": "unix//path/to/socket"
+														}
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"app.example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"upstreams": [
+														{
+															"dial": "app:80"
+														}
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/request_body.txt

@@ -0,0 +1,46 @@
+localhost
+
+request_body {
+	max_size 1MB
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "request_body",
+													"max_size": 1000000
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/request_header.txt

@@ -0,0 +1,90 @@
+:80
+
+@matcher path /something*
+request_header @matcher Denis "Ritchie"
+
+request_header +Edsger "Dijkstra"
+request_header -Wolfram
+
+@images path /images/*
+request_header @images Cache-Control "public, max-age=3600, stale-while-revalidate=86400"
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/something*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "headers",
+									"request": {
+										"set": {
+											"Denis": [
+												"Ritchie"
+											]
+										}
+									}
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"path": [
+										"/images/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "headers",
+									"request": {
+										"set": {
+											"Cache-Control": [
+												"public, max-age=3600, stale-while-revalidate=86400"
+											]
+										}
+									}
+								}
+							]
+						},
+						{
+							"handle": [
+								{
+									"handler": "headers",
+									"request": {
+										"add": {
+											"Edsger": [
+												"Dijkstra"
+											]
+										}
+									}
+								},
+								{
+									"handler": "headers",
+									"request": {
+										"delete": [
+											"Wolfram"
+										]
+									}
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_empty_non_http_transport.txt

@@ -0,0 +1,36 @@
+:8884
+
+reverse_proxy 127.0.0.1:65535 {
+	transport fastcgi
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"transport": {
+										"protocol": "fastcgi"
+									},
+									"upstreams": [
+										{
+											"dial": "127.0.0.1:65535"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_h2c_shorthand.txt

@@ -0,0 +1,38 @@
+:8884
+
+reverse_proxy h2c://localhost:8080
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"transport": {
+										"protocol": "http",
+										"versions": [
+											"h2c",
+											"2"
+										]
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:8080"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_handle_response.txt

@@ -0,0 +1,193 @@
+:8884
+
+reverse_proxy 127.0.0.1:65535 {
+	@accel header X-Accel-Redirect *
+	handle_response @accel {
+		respond "Header X-Accel-Redirect!"
+	}
+
+	@another {
+		header X-Another *
+	}
+	handle_response @another {
+		respond "Header X-Another!"
+	}
+
+	@401 status 401
+	handle_response @401 {
+		respond "Status 401!"
+	}
+
+	handle_response {
+		respond "Any! This should be last in the JSON!"
+	}
+
+	@403 {
+		status 403
+	}
+	handle_response @403 {
+		respond "Status 403!"
+	}
+
+	@multi {
+		status 401 403
+		status 404
+		header Foo *
+		header Bar *
+	}
+	handle_response @multi {
+		respond "Headers Foo, Bar AND statuses 401, 403 and 404!"
+	}
+
+	@changeStatus status 500
+	handle_response @changeStatus 400
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handle_response": [
+										{
+											"match": {
+												"headers": {
+													"X-Accel-Redirect": [
+														"*"
+													]
+												}
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Header X-Accel-Redirect!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"headers": {
+													"X-Another": [
+														"*"
+													]
+												}
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Header X-Another!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"status_code": [
+													401
+												]
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Status 401!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"status_code": [
+													403
+												]
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Status 403!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"headers": {
+													"Bar": [
+														"*"
+													],
+													"Foo": [
+														"*"
+													]
+												},
+												"status_code": [
+													401,
+													403,
+													404
+												]
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Headers Foo, Bar AND statuses 401, 403 and 404!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"status_code": [
+													500
+												]
+											},
+											"status_code": 400
+										},
+										{
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Any! This should be last in the JSON!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										}
+									],
+									"handler": "reverse_proxy",
+									"upstreams": [
+										{
+											"dial": "127.0.0.1:65535"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}