go.mod: Update CertMagic

Adds one more debug log

.github/CONTRIBUTING.md

@@ -23,13 +23,13 @@ Other menu items:
 
 ### Contributing code
 
-You can have a huge impact on the project by helping with its code. To contribute code to Caddy, open a [pull request](https://github.com/caddyserver/caddy/pulls) (PR). If you're new to our community, that's okay: **we gladly welcome pull requests from anyone, regardless of your native language or coding experience.** You can get familiar with Caddy's code base by using [code search at Sourcegraph](https://sourcegraph.com/github.com/caddyserver/caddy/-/search).
+You can have a huge impact on the project by helping with its code. To contribute code to Caddy, first submit or comment in an issue to discuss your contribution, then open a [pull request](https://github.com/caddyserver/caddy/pulls) (PR). If you're new to our community, that's okay: **we gladly welcome pull requests from anyone, regardless of your native language or coding experience.** You can get familiar with Caddy's code base by using [code search at Sourcegraph](https://sourcegraph.com/github.com/caddyserver/caddy).
 
 We hold contributions to a high standard for quality :bowtie:, so don't be surprised if we ask for revisions—even if it seems small or insignificant. Please don't take it personally. :blue_heart: If your change is on the right track, we can guide you to make it mergable.
 
 Here are some of the expectations we have of contributors:
 
-- **Open an issue to propose your change first.** This way we can avoid confusion, coordinate what everyone is working on, and ensure that any changes are in-line with the project's goals and the best interests of its users. We can also discuss the best possible implementation. If there's already an issue about it, comment on the existing issue to claim it.
+- **Open an issue to propose your change first.** This way we can avoid confusion, coordinate what everyone is working on, and ensure that any changes are in-line with the project's goals and the best interests of its users. We can also discuss the best possible implementation. If there's already an issue about it, comment on the existing issue to claim it. A lot of valuable time can be saved by discussing a proposal first.
 
 - **Keep pull requests small.** Smaller PRs are more likely to be merged because they are easier to review! We might ask you to break up large PRs into smaller ones. [An example of what we want to avoid.](https://twitter.com/iamdevloper/status/397664295875805184)
 
@@ -45,16 +45,18 @@ Here are some of the expectations we have of contributors:
 
 - **Use comments properly.** We expect good godoc comments for package-level functions, types, and values. Comments are also useful whenever the purpose for a line of code is not obvious.
 
+- **Pull requests may still get closed.** The longer a PR stays open and idle, the more likely it is to be closed. If we haven't reviewed it in a while, it probably means the change is not a priority. Please don't take this personally, we're trying to balance a lot of tasks! If nobody else has commented or reacted to the PR, it likely means your change is useful only to you. The reality is this happens quite a bit. We don't tend to accept PRs that aren't generally helpful. For these reasons or others, the PR may get closed even after a review. We are not obligated to accept all proposed changes, even if the best justification we can give is something vague like, "It doesn't sit right." Sometimes PRs are just the wrong thing or the wrong time. Because it is open source, you can always build your own modified version of Caddy with a change you need, even if we reject it in the official repo.
+
 We often grant [collaborator status](#collaborator-instructions) to contributors who author one or more significant, high-quality PRs that are merged into the code base!
 
 
 #### HOW TO MAKE A PULL REQUEST TO CADDY
 
-Contributing to Go projects on GitHub is fun and easy. We recommend the following workflow:
+Contributing to Go projects on GitHub is fun and easy. After you have proposed your change in an issue, we recommend the following workflow:
 
 1. [Fork this repo](https://github.com/caddyserver/caddy). This makes a copy of the code you can write to.
 
-2. If you don't already have this repo (caddyserver/caddy.git) repo on your computer, get it with `go get github.com/caddyserver/caddy/v2`.
+2. If you don't already have this repo (caddyserver/caddy.git) repo on your computer, clone it down: `git clone https://github.com/caddyserver/caddy.git`
 
 3. Tell git that it can push the caddyserver/caddy.git repo to your fork by adding a remote: `git remote add myfork https://github.com/<your-username>/caddy.git`
 
@@ -85,9 +87,9 @@ Many people on the forums could benefit from your experience and expertise, too.
 
 Like every software, Caddy has its flaws. If you find one, [search the issues](https://github.com/caddyserver/caddy/issues) to see if it has already been reported. If not, [open a new issue](https://github.com/caddyserver/caddy/issues/new) and describe the bug, and somebody will look into it! (This repository is only for Caddy and its standard modules.)
 
-**You can help stop bugs in their tracks!** Speed up the patch by identifying the bug in the code. This can sometimes be done by adding `fmt.Println()` statements (or similar) in relevant code paths to narrow down where the problem may be. It's a good way to [introduce yourself to the Go language](https://tour.golang.org), too.
+**You can help us fix bugs!** Speed up the patch by identifying the bug in the code. This can sometimes be done by adding `fmt.Println()` statements (or similar) in relevant code paths to narrow down where the problem may be. It's a good way to [introduce yourself to the Go language](https://tour.golang.org), too.
 
-Please follow the issue template so we have all the needed information. Unredacted&mdash;yes, actual values matter. We need to be able to repeat the bug using your instructions. Please simplify the issue as much as possible. The burden is on you to convince us that it is actually a bug in Caddy. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you!
+We may reply with an issue template. Please follow the template so we have all the needed information. Unredacted&mdash;yes, actual values matter. We need to be able to repeat the bug using your instructions. Please simplify the issue as much as possible. If you don't, we might close your report. The burden is on you to make it easily reproducible and to convince us that it is actually a bug in Caddy. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you!
 
 We suggest reading [How to Report Bugs Effectively](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html).
 
@@ -98,11 +100,12 @@ Please be kind. :smile: Remember that Caddy comes at no cost to you, and you're
 Maintainers---or more generally, developers---need three things to act on bugs:
 
 1. To agree or be convinced that it's a bug (reporter's responsibility).
-	- A bug is undesired or surprising behavior which violates documentation or the spec.
+	- A bug is unintentional, undesired, or surprising behavior which violates documentation or relevant spec. It might be either a mistake in the documentation or a bug in the code.
+	- This project usually does not work around bugs in other software, systems, and dependencies; instead, we recommend that those bugs are fixed at their source. This sometimes means we close issues or reject PRs that attempt to fix, workaround, or hide bugs in other projects.
 
 2. To be able to understand what is happening (mostly reporter's responsibility).
-	- If the reporter can provide satisfactory instructions such that a developer can reproduce the bug, the developer will likely be able to understand the bug, write a test case, and implement a fix.
-	- Otherwise, the burden is on the reporter to test possible solutions. This is discouraged because it loosens the feedback loop, slows down debugging efforts, obscures the true nature of the problem from the developers, and is unlikely to result in new test cases.
+	- If the reporter can provide satisfactory instructions such that a developer can reproduce the bug, the developer will likely be able to understand the bug, write a test case, and implement a fix. This is the least amount of work for everyone and path to the fastest resolution.
+	- Otherwise, the burden is on the reporter to test possible solutions. This is less preferable because it loosens the feedback loop, slows down debugging efforts, obscures the true nature of the problem from the developers, and is unlikely to result in new test cases.
 
 3. A solution, or ideas toward a solution (mostly maintainer's responsibility).
 	- Sometimes the best solution is a documentation change.
@@ -112,7 +115,7 @@ Maintainers---or more generally, developers---need three things to act on bugs:
 
 Thus, at the very least, the reporter is expected to:
 
-1. Convince the reader that it's a bug (if it's not obvious).
+1. Convince the reader that it's a bug in Caddy (if it's not obvious).
 2. Reduce the problem down to the minimum specific steps required to reproduce it.
 
 The maintainer is usually able to do the rest; but of course the reporter may invest additional effort to speed up the process.
@@ -123,7 +126,7 @@ The maintainer is usually able to do the rest; but of course the reporter may in
 
 First, [search to see if your feature has already been requested](https://github.com/caddyserver/caddy/issues). If it has, you can add a :+1: reaction to vote for it. If your feature idea is new, open an issue to request the feature. Please describe your idea thoroughly so that we know how to implement it! Really vague requests may not be helpful or actionable and, without clarification, will have to be closed.
 
-While we really do value your requests and implement many of them, not all features are a good fit for Caddy. Most of those [make good modules](#writing-a-caddy-module), which can be made by anyone! But if a feature is not in the best interest of the Caddy project or its users in general, we may politely decline to implement it into Caddy core.
+While we really do value your requests and implement many of them, not all features are a good fit for Caddy. Most of those [make good modules](#writing-a-caddy-module), which can be made by anyone! But if a feature is not in the best interest of the Caddy project or its users in general, we may politely decline to implement it into Caddy core. Additionally, some features are bad ideas altogether (for either obvious or non-obvious reasons) which may be rejected. We'll try to explain why we reject a feature, but sometimes the best we can do is, "It's not a good fit for the project."
 
 
 ### Improving documentation
@@ -132,11 +135,11 @@ Caddy's documentation is available at [https://caddyserver.com/docs](https://cad
 
 Note that third-party module documentation is not hosted by the Caddy website, other than basic usage examples. They are managed by the individual module authors, and you will have to contact them to change their documentation.
 
-
+Our documentation is scoped to the Caddy project only: it is not for describing how other software or systems work, even if they relate to Caddy or web servers. That kind of content [can be found in our community wiki](https://caddy.community/c/wiki/13), however.
 
 ## Collaborator Instructions
 
-Collaborators have push rights to the repository. We grant this permission after one or more successful, high-quality PRs are merged! We thank them for their help.The expectations we have of collaborators are:
+Collaborators have push rights to the repository. We grant this permission after one or more successful, high-quality PRs are merged! We thank them for their help. The expectations we have of collaborators are:
 
 - **Help review pull requests.** Be meticulous, but also kind. We love our contributors, but we critique the contribution to make it better. Multiple, thorough reviews make for the best contributions! Here are some questions to consider:
 	- Can the change be made more elegant?
@@ -167,7 +170,7 @@ Collaborators have push rights to the repository. We grant this permission after
 
 
 
-## Values
+## Values (WIP)
 
 - A person is always more important than code. People don't like being handled "efficiently". But we can still process issues and pull requests efficiently while being kind, patient, and considerate.
 

.github/SECURITY.md

@@ -2,9 +2,6 @@
 
 The Caddy project would like to make sure that it stays on top of all practically-exploitable vulnerabilities.
 
-Some security problems are more the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please report only vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing or BGP hijacks a vulnerability in the Caddy web server).
-
-Please note that we consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems.
 
 ## Supported Versions
 
@@ -14,11 +11,46 @@ Please note that we consider publicly-registered domain names to be public infor
 | 1.x     | :x:                |
 | < 1.x   | :x:                |
 
+
+## Acceptable Scope
+
+A security report must demonstrate a security bug in the source code from this repository.
+
+Some security problems are the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please only report vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing, BGP hijacks, or missing/misconfigured HTTP headers a vulnerability in the Caddy web server).
+
+Vulnerabilities caused by misconfigurations are out of scope. Yes, it is entirely possible to craft and use a configuration that is unsafe, just like with every other web server; we recommend against doing that.
+
+We do not accept reports if the steps imply or require a compromised system or third-party software, as we cannot control those. We expect that users secure their own systems and keep all their software patched. For example, if untrusted users are able to upload/write/host arbitrary files in the web root directory, it is NOT a security bug in Caddy if those files get served to clients; however, it _would_ be a valid report if a bug in Caddy's source code unintentionally gave unauthorized users the ability to upload unsafe files or delete files without relying on an unpatched system or piece of software.
+
+Client-side exploits are out of scope. In other words, it is not a bug in Caddy if the web browser does something unsafe, even if the downloaded content was served by Caddy. (Those kinds of exploits can generally be mitigated by proper configuration of HTTP headers.) As a general rule, the content served by Caddy is not considered in scope because content is configurable by the site owner or the associated web application.
+
+Security bugs in code dependencies are out of scope. Instead, if a dependency has patched a relevant security bug, please feel free to open a public issue or pull request to update that dependency in our code.
+
+
 ## Reporting a Vulnerability
 
-Please email Matt Holt (the author) directly: matt [at] lightcodelabs [dot com].
+We get a lot of difficult reports that turn out to be invalid. Clear, obvious reports tend to be the most credible (but are also rare).
 
-We'll need enough information to verify the bug and make a patch. It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, resources permitting. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. Thank you for understanding.
+First please ensure your report falls within the accepted scope of security bugs (above).
+
+We'll need enough information to verify the bug and make a patch. To speed things up, please include:
+
+- Most minimal possible config (without redactions!)
+- Command(s)
+- Precise HTTP requests (`curl -v` and its output please)
+- Full log output (please enable debug mode)
+- Specific minimal steps to reproduce the issue from scratch
+- A working patch
+
+Please DO NOT use containers, VMs, cloud instances or services, or any other complex infrastructure in your steps. Always prefer `curl` instead of web browsers.
+
+We consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems. Do not redact domain names from your reports. The actual content of your domain name affects Caddy's behavior, so we need the exact domain name(s) to reproduce with, or your report will be ignored.
+
+It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, depending on available resources. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. (We get a lot of invalid reports.) Thank you for understanding.
+
+When you are ready, please email Matt Holt (the author) directly: matt [at] lightcodelabs [dot com].
+
+Please don't encrypt the email body. It only makes the process more complicated.
 
 Please also understand that due to our nature as an open source project, we do not have a budget to award security bounties. We can only thank you.
 

.github/workflows/ci.yml

@@ -6,9 +6,11 @@ on:
   push:
     branches:
       - master
+      - 2.*
   pull_request:
     branches:
       - master
+      - 2.*
 
 jobs:
   test:
@@ -17,7 +19,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ ubuntu-latest, macos-latest, windows-latest ]
-        go: [ '1.14', '1.15' ]
+        go: [ '1.16', '1.17' ]
 
         # Set some variables per OS, usable via ${{ matrix.VAR }}
         # CADDY_BIN_PATH: the path to the compiled Caddy binary, for artifact publishing
@@ -64,6 +66,7 @@ jobs:
         go env
         printf "\n\nSystem environment:\n\n"
         env
+        printf "Git version: $(git version)\n\n"
         # Calculate the short SHA1 hash of the git commit
         echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
         echo "::set-output name=go_cache::$(go env GOCACHE)"
@@ -137,7 +140,7 @@ jobs:
 
           # The environment is fresh, so there's no point in keeping accepting and adding the key.
           rsync -arz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress --delete --exclude '.git' . caddy-ci@ci-s390x.caddyserver.com:/var/tmp/"$short_sha"
-          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -t caddy-ci@ci-s390x.caddyserver.com "cd /var/tmp/$short_sha; CGO_ENABLED=0 /usr/local/go/bin/go test -v ./..."
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -t caddy-ci@ci-s390x.caddyserver.com "cd /var/tmp/$short_sha; go version; go env; printf "\n\n";CGO_ENABLED=0 go test -v ./..."
           test_result=$?
 
           # There's no need leaving the files around
@@ -153,6 +156,7 @@ jobs:
     steps:
       - name: checkout
         uses: actions/checkout@v2
+      
       - uses: goreleaser/goreleaser-action@v2
         with:
           version: latest

.github/workflows/cross-build.yml

@@ -4,9 +4,11 @@ on:
   push:
     branches:
       - master
+      - 2.*
   pull_request:
     branches:
       - master
+      - 2.*
 
 jobs:
   cross-build-test:
@@ -14,7 +16,7 @@ jobs:
       fail-fast: false
       matrix:
         goos: ['android', 'linux', 'solaris', 'illumos', 'dragonfly', 'freebsd', 'openbsd', 'plan9', 'windows', 'darwin', 'netbsd']
-        go: [ '1.14', '1.15' ]
+        go: [ '1.17' ]
     runs-on: ubuntu-latest
     continue-on-error: true
     steps:

.github/workflows/lint.yml

@@ -4,9 +4,11 @@ on:
   push:
     branches:
       - master
+      - 2.*
   pull_request:
     branches:
       - master
+      - 2.*
 
 jobs:
   # From https://github.com/golangci/golangci-lint-action

.github/workflows/release.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       matrix:
         os: [ ubuntu-latest ]
-        go: [ '1.15' ]
+        go: [ '1.17' ]
     runs-on: ${{ matrix.os }}
 
     steps:

.gitignore

@@ -10,7 +10,6 @@ Caddyfile
 # build artifacts and helpers
 cmd/caddy/caddy
 cmd/caddy/caddy.exe
-cmd/caddy/setcap*
 
 # mac specific
 .DS_Store
@@ -22,3 +21,7 @@ vendor
 dist
 caddy-build
 caddy-dist
+
+# IDE files
+.idea/
+.vscode/

.goreleaser.yml

@@ -11,6 +11,9 @@ before:
     # GoReleaser doesn't seem to offer {{.Tag}} at this stage, so we have to embed it into the env
     # so we run: TAG=$(git describe --abbrev=0) goreleaser release --rm-dist --skip-publish --skip-validate
     - go mod edit -require=github.com/caddyserver/caddy/v2@{{.Env.TAG}} ./caddy-build/go.mod
+    # as of Go 1.16, `go` commands no longer automatically change go.{mod,sum}. We now have to explicitly
+    # run `go mod tidy`. The `/bin/sh -c '...'` is because goreleaser can't find cd in PATH without shell invocation.
+    - /bin/sh -c 'cd ./caddy-build && go mod tidy'
     - git clone --depth 1 https://github.com/caddyserver/dist caddy-dist
     - go mod download
 
@@ -76,7 +79,7 @@ nfpms:
     homepage: https://caddyserver.com
     maintainer: Matthew Holt <mholt@users.noreply.github.com>
     description: |
-      Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go
+      Caddy - Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go
     license: Apache 2.0
 
     formats:

README.md

@@ -69,13 +69,13 @@
 
 The simplest, cross-platform way is to download from [GitHub Releases](https://github.com/caddyserver/caddy/releases) and place the executable file in your PATH.
 
-For other install options, see https://caddyserver.com/docs/download.
+For other install options, see https://caddyserver.com/docs/install.
 
 ## Build from source
 
 Requirements:
 
-- [Go 1.14 or newer](https://golang.org/dl/)
+- [Go 1.16 or newer](https://golang.org/dl/)
 
 ### For development
  
@@ -87,17 +87,9 @@ $ cd caddy/cmd/caddy/
 $ go build
 ```
 
-When you run Caddy, it may try to bind to low ports unless otherwise specified in your config. If your OS requires elevated privileges, you will need to give your new binary permission to do so. On Linux, this can be done easily with: `sudo setcap cap_net_bind_service=+ep ./caddy`
+When you run Caddy, it may try to bind to low ports unless otherwise specified in your config. If your OS requires elevated privileges for this, you will need to give your new binary permission to do so. On Linux, this can be done easily with: `sudo setcap cap_net_bind_service=+ep ./caddy`
 
-If you prefer to use `go run` which creates temporary binaries, you can still do this. Make an executable file called `setcap.sh` (or whatever you want) with these contents:
-
-```bash
-#!/bin/sh
-sudo setcap cap_net_bind_service=+ep "$1"
-"$@"
-```
-
-then you can use `go run` like so:
+If you prefer to use `go run` which only creates temporary binaries, you can still do this with the included `setcap.sh` like so:
 
 ```bash
 $ go run -exec ./setcap.sh main.go
@@ -125,7 +117,7 @@ $ xcaddy build
 2. Change into it: `cd caddy`
 3. Copy [Caddy's main.go](https://github.com/caddyserver/caddy/blob/master/cmd/caddy/main.go) into the empty folder. Add imports for any custom plugins you want to add.
 4. Initialize a Go module: `go mod init caddy`
-5. (Optional) Pin Caddy version: `go get github.com/caddyserver/caddy/v2@version` replacing `version` with a git tag or commit.
+5. (Optional) Pin Caddy version: `go get github.com/caddyserver/caddy/v2@version` replacing `version` with a git tag, commit, or branch name.
 6. (Optional) Add plugins by adding their import: `_ "import/path/here"`
 7. Compile: `go build`
 
@@ -174,7 +166,7 @@ The docs are also open source. You can contribute to them here: https://github.c
 
 - We **strongly recommend** that all professionals or companies using Caddy get a support contract through [Ardan Labs](https://www.ardanlabs.com/my/contact-us?dd=caddy) before help is needed.
 
-- A [sponsorship](https://github.com/sponsors/mholt) goes a long way!
+- A [sponsorship](https://github.com/sponsors/mholt) goes a long way! If Caddy is benefitting your company, please consider a sponsorship! This not only helps fund full-time work to ensure the longevity of the project, it's also a great look for your company to your customers and potential customers!
 
 - Individuals can exchange help for free on our community forum at https://caddy.community. Remember that people give help out of their spare time and good will. The best way to get help is to give it first!
 
@@ -184,11 +176,11 @@ Please use our [issue tracker](https://github.com/caddyserver/caddy/issues) only
 
 ## About
 
-**The name "Caddy" is trademarked.** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". Caddy is a registered trademark of apilayer GmbH.
+**The name "Caddy" is trademarked.** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". Caddy is a registered trademark of Stack Holdings GmbH.
 
 - _Project on Twitter: [@caddyserver](https://twitter.com/caddyserver)_
 - _Author on Twitter: [@mholt6](https://twitter.com/mholt6)_
 
-Caddy is a project of [ZeroSSL](https://zerossl.com), an [apilayer](https://apilayer.com) company.
+Caddy is a project of [ZeroSSL](https://zerossl.com), a Stack Holdings company.
 
 Debian package repository hosting is graciously provided by [Cloudsmith](https://cloudsmith.com). Cloudsmith is the only fully hosted, cloud-native, universal package management solution, that enables your organization to create, store and share packages in any format, to any place, with total confidence.

admin.go

@@ -17,6 +17,10 @@ package caddy
 import (
 	"bytes"
 	"context"
+	"crypto"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/base64"
 	"encoding/json"
 	"errors"
 	"expvar"
@@ -35,12 +39,12 @@ import (
 	"sync"
 	"time"
 
+	"github.com/caddyserver/caddy/v2/notify"
+	"github.com/caddyserver/certmagic"
 	"github.com/prometheus/client_golang/prometheus"
 	"go.uber.org/zap"
 )
 
-// TODO: is there a way to make the admin endpoint so that it can be plugged into the HTTP app? see issue #2833
-
 // AdminConfig configures Caddy's API endpoint, which is used
 // to manage Caddy while it is running.
 type AdminConfig struct {
@@ -58,54 +62,137 @@ type AdminConfig struct {
 	// If true, CORS headers will be emitted, and requests to the
 	// API will be rejected if their `Host` and `Origin` headers
 	// do not match the expected value(s). Use `origins` to
-	// customize which origins/hosts are allowed.If `origins` is
+	// customize which origins/hosts are allowed. If `origins` is
 	// not set, the listen address is the only value allowed by
-	// default.
+	// default. Enforced only on local (plaintext) endpoint.
 	EnforceOrigin bool `json:"enforce_origin,omitempty"`
 
 	// The list of allowed origins/hosts for API requests. Only needed
 	// if accessing the admin endpoint from a host different from the
 	// socket's network interface or if `enforce_origin` is true. If not
 	// set, the listener address will be the default value. If set but
-	// empty, no origins will be allowed.
+	// empty, no origins will be allowed. Enforced only on local
+	// (plaintext) endpoint.
 	Origins []string `json:"origins,omitempty"`
 
-	// Options related to configuration management.
+	// Options pertaining to configuration management.
 	Config *ConfigSettings `json:"config,omitempty"`
+
+	// Options that establish this server's identity. Identity refers to
+	// credentials which can be used to uniquely identify and authenticate
+	// this server instance. This is required if remote administration is
+	// enabled (but does not require remote administration to be enabled).
+	// Default: no identity management.
+	Identity *IdentityConfig `json:"identity,omitempty"`
+
+	// Options pertaining to remote administration. By default, remote
+	// administration is disabled. If enabled, identity management must
+	// also be configured, as that is how the endpoint is secured.
+	// See the neighboring "identity" object.
+	//
+	// EXPERIMENTAL: This feature is subject to change.
+	Remote *RemoteAdmin `json:"remote,omitempty"`
 }
 
-// ConfigSettings configures the, uh, configuration... and
-// management thereof.
+// ConfigSettings configures the management of configuration.
 type ConfigSettings struct {
 	// Whether to keep a copy of the active config on disk. Default is true.
+	// Note that "pulled" dynamic configs (using the neighboring "load" module)
+	// are not persisted; only configs that are pushed to Caddy get persisted.
 	Persist *bool `json:"persist,omitempty"`
+
+	// Loads a configuration to use. This is helpful if your configs are
+	// managed elsewhere, and you want Caddy to pull its config dynamically
+	// when it starts. The pulled config completely replaces the current
+	// one, just like any other config load. It is an error if a pulled
+	// config is configured to pull another config.
+	//
+	// EXPERIMENTAL: Subject to change.
+	LoadRaw json.RawMessage `json:"load,omitempty" caddy:"namespace=caddy.config_loaders inline_key=module"`
+
+	// The interval to pull config. With a non-zero value, will pull config
+	// from config loader (eg. a http loader) with given interval.
+	//
+	// EXPERIMENTAL: Subject to change.
+	LoadInterval Duration `json:"load_interval,omitempty"`
 }
 
-// listenAddr extracts a singular listen address from ac.Listen,
-// returning the network and the address of the listener.
-func (admin AdminConfig) listenAddr() (NetworkAddress, error) {
-	input := admin.Listen
-	if input == "" {
-		input = DefaultAdminListen
-	}
-	listenAddr, err := ParseNetworkAddress(input)
-	if err != nil {
-		return NetworkAddress{}, fmt.Errorf("parsing admin listener address: %v", err)
-	}
-	if listenAddr.PortRangeSize() != 1 {
-		return NetworkAddress{}, fmt.Errorf("admin endpoint must have exactly one address; cannot listen on %v", listenAddr)
-	}
-	return listenAddr, nil
+// IdentityConfig configures management of this server's identity. An identity
+// consists of credentials that uniquely verify this instance; for example,
+// TLS certificates (public + private key pairs).
+type IdentityConfig struct {
+	// List of names or IP addresses which refer to this server.
+	// Certificates will be obtained for these identifiers so
+	// secure TLS connections can be made using them.
+	Identifiers []string `json:"identifiers,omitempty"`
+
+	// Issuers that can provide this admin endpoint its identity
+	// certificate(s). Default: ACME issuers configured for
+	// ZeroSSL and Let's Encrypt. Be sure to change this if you
+	// require credentials for private identifiers.
+	IssuersRaw []json.RawMessage `json:"issuers,omitempty" caddy:"namespace=tls.issuance inline_key=module"`
+
+	issuers []certmagic.Issuer
+}
+
+// RemoteAdmin enables and configures remote administration. If enabled,
+// a secure listener enforcing mutual TLS authentication will be started
+// on a different port from the standard plaintext admin server.
+//
+// This endpoint is secured using identity management, which must be
+// configured separately (because identity management does not depend
+// on remote administration). See the admin/identity config struct.
+//
+// EXPERIMENTAL: Subject to change.
+type RemoteAdmin struct {
+	// The address on which to start the secure listener.
+	// Default: :2021
+	Listen string `json:"listen,omitempty"`
+
+	// List of access controls for this secure admin endpoint.
+	// This configures TLS mutual authentication (i.e. authorized
+	// client certificates), but also application-layer permissions
+	// like which paths and methods each identity is authorized for.
+	AccessControl []*AdminAccess `json:"access_control,omitempty"`
+}
+
+// AdminAccess specifies what permissions an identity or group
+// of identities are granted.
+type AdminAccess struct {
+	// Base64-encoded DER certificates containing public keys to accept.
+	// (The contents of PEM certificate blocks are base64-encoded DER.)
+	// Any of these public keys can appear in any part of a verified chain.
+	PublicKeys []string `json:"public_keys,omitempty"`
+
+	// Limits what the associated identities are allowed to do.
+	// If unspecified, all permissions are granted.
+	Permissions []AdminPermissions `json:"permissions,omitempty"`
+
+	publicKeys []crypto.PublicKey
+}
+
+// AdminPermissions specifies what kinds of requests are allowed
+// to be made to the admin endpoint.
+type AdminPermissions struct {
+	// The API paths allowed. Paths are simple prefix matches.
+	// Any subpath of the specified paths will be allowed.
+	Paths []string `json:"paths,omitempty"`
+
+	// The HTTP methods allowed for the given paths.
+	Methods []string `json:"methods,omitempty"`
 }
 
 // newAdminHandler reads admin's config and returns an http.Handler suitable
 // for use in an admin endpoint server, which will be listening on listenAddr.
-func (admin AdminConfig) newAdminHandler(addr NetworkAddress) adminHandler {
-	muxWrap := adminHandler{
-		enforceOrigin:  admin.EnforceOrigin,
-		enforceHost:    !addr.isWildcardInterface(),
-		allowedOrigins: admin.allowedOrigins(addr),
-		mux:            http.NewServeMux(),
+func (admin AdminConfig) newAdminHandler(addr NetworkAddress, remote bool) adminHandler {
+	muxWrap := adminHandler{mux: http.NewServeMux()}
+
+	// secure the local or remote endpoint respectively
+	if remote {
+		muxWrap.remoteControl = admin.Remote
+	} else {
+		muxWrap.enforceHost = !addr.isWildcardInterface()
+		muxWrap.allowedOrigins = admin.allowedOrigins(addr)
 	}
 
 	addRouteWithMetrics := func(pattern string, handlerLabel string, h http.Handler) {
@@ -197,18 +284,18 @@ func (admin AdminConfig) allowedOrigins(addr NetworkAddress) []string {
 	return allowed
 }
 
-// replaceAdmin replaces the running admin server according
-// to the relevant configuration in cfg. If no configuration
-// for the admin endpoint exists in cfg, a default one is
-// used, so that there is always an admin server (unless it
-// is explicitly configured to be disabled).
-func replaceAdmin(cfg *Config) error {
+// replaceLocalAdminServer replaces the running local admin server
+// according to the relevant configuration in cfg. If no configuration
+// for the admin endpoint exists in cfg, a default one is used, so
+// that there is always an admin server (unless it is explicitly
+// configured to be disabled).
+func replaceLocalAdminServer(cfg *Config) error {
 	// always be sure to close down the old admin endpoint
 	// as gracefully as possible, even if the new one is
 	// disabled -- careful to use reference to the current
 	// (old) admin endpoint since it will be different
 	// when the function returns
-	oldAdminServer := adminServer
+	oldAdminServer := localAdminServer
 	defer func() {
 		// do the shutdown asynchronously so that any
 		// current API request gets a response; this
@@ -236,29 +323,35 @@ func replaceAdmin(cfg *Config) error {
 	}
 
 	// extract a singular listener address
-	addr, err := adminConfig.listenAddr()
+	addr, err := parseAdminListenAddr(adminConfig.Listen, DefaultAdminListen)
 	if err != nil {
 		return err
 	}
 
-	handler := adminConfig.newAdminHandler(addr)
+	handler := adminConfig.newAdminHandler(addr, false)
 
 	ln, err := Listen(addr.Network, addr.JoinHostPort(0))
 	if err != nil {
 		return err
 	}
 
-	adminServer = &http.Server{
+	serverMu.Lock()
+	localAdminServer = &http.Server{
+		Addr:              addr.String(), // for logging purposes only
 		Handler:           handler,
 		ReadTimeout:       10 * time.Second,
 		ReadHeaderTimeout: 5 * time.Second,
 		IdleTimeout:       60 * time.Second,
 		MaxHeaderBytes:    1024 * 64,
 	}
+	serverMu.Unlock()
 
 	adminLogger := Log().Named("admin")
 	go func() {
-		if err := adminServer.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+		serverMu.Lock()
+		server := localAdminServer
+		serverMu.Unlock()
+		if err := server.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
 			adminLogger.Error("admin server shutdown for unknown reason", zap.Error(err))
 		}
 	}()
@@ -276,6 +369,257 @@ func replaceAdmin(cfg *Config) error {
 	return nil
 }
 
+// manageIdentity sets up automated identity management for this server.
+func manageIdentity(ctx Context, cfg *Config) error {
+	if cfg == nil || cfg.Admin == nil || cfg.Admin.Identity == nil {
+		return nil
+	}
+
+	// set default issuers; this is pretty hacky because we can't
+	// import the caddytls package -- but it works
+	if cfg.Admin.Identity.IssuersRaw == nil {
+		cfg.Admin.Identity.IssuersRaw = []json.RawMessage{
+			json.RawMessage(`{"module": "zerossl"}`),
+			json.RawMessage(`{"module": "acme"}`),
+		}
+	}
+
+	// load and provision issuer modules
+	if cfg.Admin.Identity.IssuersRaw != nil {
+		val, err := ctx.LoadModule(cfg.Admin.Identity, "IssuersRaw")
+		if err != nil {
+			return fmt.Errorf("loading identity issuer modules: %s", err)
+		}
+		for _, issVal := range val.([]interface{}) {
+			cfg.Admin.Identity.issuers = append(cfg.Admin.Identity.issuers, issVal.(certmagic.Issuer))
+		}
+	}
+
+	// we'll make a new cache when we make the CertMagic config, so stop any previous cache
+	if identityCertCache != nil {
+		identityCertCache.Stop()
+	}
+
+	logger := Log().Named("admin.identity")
+	cmCfg := cfg.Admin.Identity.certmagicConfig(logger, true)
+
+	// issuers have circular dependencies with the configs because,
+	// as explained in the caddytls package, they need access to the
+	// correct storage and cache to solve ACME challenges
+	for _, issuer := range cfg.Admin.Identity.issuers {
+		// avoid import cycle with caddytls package, so manually duplicate the interface here, yuck
+		if annoying, ok := issuer.(interface{ SetConfig(cfg *certmagic.Config) }); ok {
+			annoying.SetConfig(cmCfg)
+		}
+	}
+
+	// obtain and renew server identity certificate(s)
+	return cmCfg.ManageAsync(ctx, cfg.Admin.Identity.Identifiers)
+}
+
+// replaceRemoteAdminServer replaces the running remote admin server
+// according to the relevant configuration in cfg. It stops any previous
+// remote admin server and only starts a new one if configured.
+func replaceRemoteAdminServer(ctx Context, cfg *Config) error {
+	if cfg == nil {
+		return nil
+	}
+
+	remoteLogger := Log().Named("admin.remote")
+
+	oldAdminServer := remoteAdminServer
+	defer func() {
+		if oldAdminServer != nil {
+			go func(oldAdminServer *http.Server) {
+				err := stopAdminServer(oldAdminServer)
+				if err != nil {
+					Log().Named("admin").Error("stopping current secure admin endpoint", zap.Error(err))
+				}
+			}(oldAdminServer)
+		}
+	}()
+
+	if cfg.Admin == nil || cfg.Admin.Remote == nil {
+		return nil
+	}
+
+	addr, err := parseAdminListenAddr(cfg.Admin.Remote.Listen, DefaultRemoteAdminListen)
+	if err != nil {
+		return err
+	}
+
+	// make the HTTP handler but disable Host/Origin enforcement
+	// because we are using TLS authentication instead
+	handler := cfg.Admin.newAdminHandler(addr, true)
+
+	// create client certificate pool for TLS mutual auth, and extract public keys
+	// so that we can enforce access controls at the application layer
+	clientCertPool := x509.NewCertPool()
+	for i, accessControl := range cfg.Admin.Remote.AccessControl {
+		for j, certBase64 := range accessControl.PublicKeys {
+			cert, err := decodeBase64DERCert(certBase64)
+			if err != nil {
+				return fmt.Errorf("access control %d public key %d: parsing base64 certificate DER: %v", i, j, err)
+			}
+			accessControl.publicKeys = append(accessControl.publicKeys, cert.PublicKey)
+			clientCertPool.AddCert(cert)
+		}
+	}
+
+	// create TLS config that will enforce mutual authentication
+	cmCfg := cfg.Admin.Identity.certmagicConfig(remoteLogger, false)
+	tlsConfig := cmCfg.TLSConfig()
+	tlsConfig.NextProtos = nil // this server does not solve ACME challenges
+	tlsConfig.ClientAuth = tls.RequireAndVerifyClientCert
+	tlsConfig.ClientCAs = clientCertPool
+
+	// convert logger to stdlib so it can be used by HTTP server
+	serverLogger, err := zap.NewStdLogAt(remoteLogger, zap.DebugLevel)
+	if err != nil {
+		return err
+	}
+
+	serverMu.Lock()
+	// create secure HTTP server
+	remoteAdminServer = &http.Server{
+		Addr:              addr.String(), // for logging purposes only
+		Handler:           handler,
+		TLSConfig:         tlsConfig,
+		ReadTimeout:       10 * time.Second,
+		ReadHeaderTimeout: 5 * time.Second,
+		IdleTimeout:       60 * time.Second,
+		MaxHeaderBytes:    1024 * 64,
+		ErrorLog:          serverLogger,
+	}
+	serverMu.Unlock()
+
+	// start listener
+	ln, err := Listen(addr.Network, addr.JoinHostPort(0))
+	if err != nil {
+		return err
+	}
+	ln = tls.NewListener(ln, tlsConfig)
+
+	go func() {
+		serverMu.Lock()
+		server := remoteAdminServer
+		serverMu.Unlock()
+		if err := server.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
+			remoteLogger.Error("admin remote server shutdown for unknown reason", zap.Error(err))
+		}
+	}()
+
+	remoteLogger.Info("secure admin remote control endpoint started",
+		zap.String("address", addr.String()))
+
+	return nil
+}
+
+func (ident *IdentityConfig) certmagicConfig(logger *zap.Logger, makeCache bool) *certmagic.Config {
+	if ident == nil {
+		// user might not have configured identity; that's OK, we can still make a
+		// certmagic config, although it'll be mostly useless for remote management
+		ident = new(IdentityConfig)
+	}
+	cmCfg := &certmagic.Config{
+		Storage: DefaultStorage, // do not act as part of a cluster (this is for the server's local identity)
+		Logger:  logger,
+		Issuers: ident.issuers,
+	}
+	if makeCache {
+		identityCertCache = certmagic.NewCache(certmagic.CacheOptions{
+			GetConfigForCert: func(certmagic.Certificate) (*certmagic.Config, error) {
+				return cmCfg, nil
+			},
+		})
+	}
+	return certmagic.New(identityCertCache, *cmCfg)
+}
+
+// IdentityCredentials returns this instance's configured, managed identity credentials
+// that can be used in TLS client authentication.
+func (ctx Context) IdentityCredentials(logger *zap.Logger) ([]tls.Certificate, error) {
+	if ctx.cfg == nil || ctx.cfg.Admin == nil || ctx.cfg.Admin.Identity == nil {
+		return nil, fmt.Errorf("no server identity configured")
+	}
+	ident := ctx.cfg.Admin.Identity
+	if len(ident.Identifiers) == 0 {
+		return nil, fmt.Errorf("no identifiers configured")
+	}
+	if logger == nil {
+		logger = Log()
+	}
+	magic := ident.certmagicConfig(logger, false)
+	return magic.ClientCredentials(ctx, ident.Identifiers)
+}
+
+// enforceAccessControls enforces application-layer access controls for r based on remote.
+// It expects that the TLS server has already established at least one verified chain of
+// trust, and then looks for a matching, authorized public key that is allowed to access
+// the defined path(s) using the defined method(s).
+func (remote RemoteAdmin) enforceAccessControls(r *http.Request) error {
+	for _, chain := range r.TLS.VerifiedChains {
+		for _, peerCert := range chain {
+			for _, adminAccess := range remote.AccessControl {
+				for _, allowedKey := range adminAccess.publicKeys {
+					// see if we found a matching public key; the TLS server already verified the chain
+					// so we know the client possesses the associated private key; this handy interface
+					// doesn't appear to be defined anywhere in the std lib, but was implemented here:
+					// https://github.com/golang/go/commit/b5f2c0f50297fa5cd14af668ddd7fd923626cf8c
+					comparer, ok := peerCert.PublicKey.(interface{ Equal(crypto.PublicKey) bool })
+					if !ok || !comparer.Equal(allowedKey) {
+						continue
+					}
+
+					// key recognized; make sure its HTTP request is permitted
+					for _, accessPerm := range adminAccess.Permissions {
+						// verify method
+						methodFound := accessPerm.Methods == nil
+						for _, method := range accessPerm.Methods {
+							if method == r.Method {
+								methodFound = true
+								break
+							}
+						}
+						if !methodFound {
+							return APIError{
+								HTTPStatus: http.StatusForbidden,
+								Message:    "not authorized to use this method",
+							}
+						}
+
+						// verify path
+						pathFound := accessPerm.Paths == nil
+						for _, allowedPath := range accessPerm.Paths {
+							if strings.HasPrefix(r.URL.Path, allowedPath) {
+								pathFound = true
+								break
+							}
+						}
+						if !pathFound {
+							return APIError{
+								HTTPStatus: http.StatusForbidden,
+								Message:    "not authorized to access this path",
+							}
+						}
+					}
+
+					// public key authorized, method and path allowed
+					return nil
+				}
+			}
+		}
+	}
+
+	// in theory, this should never happen; with an unverified chain, the TLS server
+	// should not accept the connection in the first place, and the acceptable cert
+	// pool is configured using the same list of public keys we verify against
+	return APIError{
+		HTTPStatus: http.StatusUnauthorized,
+		Message:    "client identity not authorized",
+	}
+}
+
 func stopAdminServer(srv *http.Server) error {
 	if srv == nil {
 		return fmt.Errorf("no admin server")
@@ -286,7 +630,7 @@ func stopAdminServer(srv *http.Server) error {
 	if err != nil {
 		return fmt.Errorf("shutting down admin server: %v", err)
 	}
-	Log().Named("admin").Info("stopped previous server")
+	Log().Named("admin").Info("stopped previous server", zap.String("address", srv.Addr))
 	return nil
 }
 
@@ -302,10 +646,15 @@ type AdminRoute struct {
 }
 
 type adminHandler struct {
+	mux *http.ServeMux
+
+	// security for local/plaintext) endpoint, on by default
 	enforceOrigin  bool
 	enforceHost    bool
 	allowedOrigins []string
-	mux            *http.ServeMux
+
+	// security for remote/encrypted endpoint
+	remoteControl *RemoteAdmin
 }
 
 // ServeHTTP is the external entry point for API requests.
@@ -318,6 +667,12 @@ func (h adminHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		zap.String("remote_addr", r.RemoteAddr),
 		zap.Reflect("headers", r.Header),
 	)
+	if r.TLS != nil {
+		log = log.With(
+			zap.Bool("secure", true),
+			zap.Int("verified_chains", len(r.TLS.VerifiedChains)),
+		)
+	}
 	if r.RequestURI == "/metrics" {
 		log.Debug("received request")
 	} else {
@@ -330,6 +685,14 @@ func (h adminHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 // be called more than once per request, for example if a request
 // is rewritten (i.e. internal redirect).
 func (h adminHandler) serveHTTP(w http.ResponseWriter, r *http.Request) {
+	if h.remoteControl != nil {
+		// enforce access controls on secure endpoint
+		if err := h.remoteControl.enforceAccessControls(r); err != nil {
+			h.handleError(w, r, err)
+			return
+		}
+	}
+
 	if strings.Contains(r.Header.Get("Upgrade"), "websocket") {
 		// I've never been able demonstrate a vulnerability myself, but apparently
 		// WebSocket connections originating from browsers aren't subject to CORS
@@ -363,8 +726,6 @@ func (h adminHandler) serveHTTP(w http.ResponseWriter, r *http.Request) {
 		w.Header().Set("Access-Control-Allow-Origin", origin)
 	}
 
-	// TODO: authentication & authorization, if configured
-
 	h.mux.ServeHTTP(w, r)
 }
 
@@ -372,7 +733,7 @@ func (h adminHandler) handleError(w http.ResponseWriter, r *http.Request, err er
 	if err == nil {
 		return
 	}
-	if err == ErrInternalRedir {
+	if err == errInternalRedir {
 		h.serveHTTP(w, r)
 		return
 	}
@@ -380,12 +741,12 @@ func (h adminHandler) handleError(w http.ResponseWriter, r *http.Request, err er
 	apiErr, ok := err.(APIError)
 	if !ok {
 		apiErr = APIError{
-			Code: http.StatusInternalServerError,
-			Err:  err,
+			HTTPStatus: http.StatusInternalServerError,
+			Err:        err,
 		}
 	}
-	if apiErr.Code == 0 {
-		apiErr.Code = http.StatusInternalServerError
+	if apiErr.HTTPStatus == 0 {
+		apiErr.HTTPStatus = http.StatusInternalServerError
 	}
 	if apiErr.Message == "" && apiErr.Err != nil {
 		apiErr.Message = apiErr.Err.Error()
@@ -393,11 +754,11 @@ func (h adminHandler) handleError(w http.ResponseWriter, r *http.Request, err er
 
 	Log().Named("admin.api").Error("request error",
 		zap.Error(err),
-		zap.Int("status_code", apiErr.Code),
+		zap.Int("status_code", apiErr.HTTPStatus),
 	)
 
 	w.Header().Set("Content-Type", "application/json")
-	w.WriteHeader(apiErr.Code)
+	w.WriteHeader(apiErr.HTTPStatus)
 	encErr := json.NewEncoder(w).Encode(apiErr)
 	if encErr != nil {
 		Log().Named("admin.api").Error("failed to encode error response", zap.Error(encErr))
@@ -418,8 +779,8 @@ func (h adminHandler) checkHost(r *http.Request) error {
 	}
 	if !allowed {
 		return APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("host not allowed: %s", r.Host),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("host not allowed: %s", r.Host),
 		}
 	}
 	return nil
@@ -433,14 +794,14 @@ func (h adminHandler) checkOrigin(r *http.Request) (string, error) {
 	origin := h.getOriginHost(r)
 	if origin == "" {
 		return origin, APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("missing required Origin header"),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("missing required Origin header"),
 		}
 	}
 	if !h.originAllowed(origin) {
 		return origin, APIError{
-			Code: http.StatusForbidden,
-			Err:  fmt.Errorf("client is not allowed to access from origin %s", origin),
+			HTTPStatus: http.StatusForbidden,
+			Err:        fmt.Errorf("client is not allowed to access from origin %s", origin),
 		}
 	}
 	return origin, nil
@@ -480,7 +841,7 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 
 		err := readConfig(r.URL.Path, w)
 		if err != nil {
-			return APIError{Code: http.StatusBadRequest, Err: err}
+			return APIError{HTTPStatus: http.StatusBadRequest, Err: err}
 		}
 
 		return nil
@@ -495,8 +856,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 		if r.Method != http.MethodDelete {
 			if ct := r.Header.Get("Content-Type"); !strings.Contains(ct, "/json") {
 				return APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("unacceptable content-type: %v; 'application/json' required", ct),
+					HTTPStatus: http.StatusBadRequest,
+					Err:        fmt.Errorf("unacceptable content-type: %v; 'application/json' required", ct),
 				}
 			}
 
@@ -507,8 +868,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 			_, err := io.Copy(buf, r.Body)
 			if err != nil {
 				return APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("reading request body: %v", err),
+					HTTPStatus: http.StatusBadRequest,
+					Err:        fmt.Errorf("reading request body: %v", err),
 				}
 			}
 			body = buf.Bytes()
@@ -523,8 +884,8 @@ func handleConfig(w http.ResponseWriter, r *http.Request) error {
 
 	default:
 		return APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method %s not allowed", r.Method),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method %s not allowed", r.Method),
 		}
 	}
 
@@ -555,46 +916,22 @@ func handleConfigID(w http.ResponseWriter, r *http.Request) error {
 	parts = append([]string{expanded}, parts[3:]...)
 	r.URL.Path = path.Join(parts...)
 
-	return ErrInternalRedir
+	return errInternalRedir
 }
 
 func handleStop(w http.ResponseWriter, r *http.Request) error {
-	err := handleUnload(w, r)
-	if err != nil {
-		Log().Named("admin.api").Error("unload error", zap.Error(err))
-	}
-	if adminServer != nil {
-		// use goroutine so that we can finish responding to API request
-		go func() {
-			err := stopAdminServer(adminServer)
-			var exitCode int
-			if err != nil {
-				exitCode = ExitCodeFailedQuit
-				Log().Named("admin.api").Error("failed to stop admin server gracefully", zap.Error(err))
-			}
-			Log().Named("admin.api").Info("stopping now, bye!! 👋")
-			os.Exit(exitCode)
-		}()
-	}
-	return nil
-}
-
-// handleUnload stops the current configuration that is running.
-// Note that doing this can also be accomplished with DELETE /config/
-// but we leave this function because handleStop uses it.
-func handleUnload(w http.ResponseWriter, r *http.Request) error {
 	if r.Method != http.MethodPost {
 		return APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method not allowed"),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method not allowed"),
 		}
 	}
-	Log().Named("admin.api").Info("unloading")
-	if err := stopAndCleanup(); err != nil {
-		Log().Named("admin.api").Error("error unloading", zap.Error(err))
-	} else {
-		Log().Named("admin.api").Info("unloading completed")
+
+	if err := notify.NotifyStopping(); err != nil {
+		Log().Error("unable to notify stopping to service manager", zap.Error(err))
 	}
+
+	exitProcess(Log().Named("admin.api"))
 	return nil
 }
 
@@ -806,9 +1143,9 @@ func (f AdminHandlerFunc) ServeHTTP(w http.ResponseWriter, r *http.Request) erro
 // and client responses. If Message is unset, then
 // Err.Error() will be serialized in its place.
 type APIError struct {
-	Code    int    `json:"-"`
-	Err     error  `json:"-"`
-	Message string `json:"error"`
+	HTTPStatus int    `json:"-"`
+	Err        error  `json:"-"`
+	Message    string `json:"error"`
 }
 
 func (e APIError) Error() string {
@@ -818,20 +1155,44 @@ func (e APIError) Error() string {
 	return e.Message
 }
 
+// parseAdminListenAddr extracts a singular listen address from either addr
+// or defaultAddr, returning the network and the address of the listener.
+func parseAdminListenAddr(addr string, defaultAddr string) (NetworkAddress, error) {
+	input := addr
+	if input == "" {
+		input = defaultAddr
+	}
+	listenAddr, err := ParseNetworkAddress(input)
+	if err != nil {
+		return NetworkAddress{}, fmt.Errorf("parsing listener address: %v", err)
+	}
+	if listenAddr.PortRangeSize() != 1 {
+		return NetworkAddress{}, fmt.Errorf("must be exactly one listener address; cannot listen on: %s", listenAddr)
+	}
+	return listenAddr, nil
+}
+
+// decodeBase64DERCert base64-decodes, then DER-decodes, certStr.
+func decodeBase64DERCert(certStr string) (*x509.Certificate, error) {
+	derBytes, err := base64.StdEncoding.DecodeString(certStr)
+	if err != nil {
+		return nil, err
+	}
+	return x509.ParseCertificate(derBytes)
+}
+
 var (
-	// DefaultAdminListen is the address for the admin
+	// DefaultAdminListen is the address for the local admin
 	// listener, if none is specified at startup.
 	DefaultAdminListen = "localhost:2019"
 
-	// ErrInternalRedir indicates an internal redirect
-	// and is useful when admin API handlers rewrite
-	// the request; in that case, authentication and
-	// authorization needs to happen again for the
-	// rewritten request.
-	ErrInternalRedir = fmt.Errorf("internal redirect; re-authorization required")
+	// DefaultRemoteAdminListen is the address for the remote
+	// (TLS-authenticated) admin listener, if enabled and not
+	// specified otherwise.
+	DefaultRemoteAdminListen = ":2021"
 
 	// DefaultAdminConfig is the default configuration
-	// for the administration endpoint.
+	// for the local administration endpoint.
 	DefaultAdminConfig = &AdminConfig{
 		Listen: DefaultAdminListen,
 	}
@@ -858,6 +1219,13 @@ var idRegexp = regexp.MustCompile(`(?m),?\s*"` + idKey + `"\s*:\s*(-?[0-9]+(\.[0
 // pidfile is the name of the pidfile, if any.
 var pidfile string
 
+// errInternalRedir indicates an internal redirect
+// and is useful when admin API handlers rewrite
+// the request; in that case, authentication and
+// authorization needs to happen again for the
+// rewritten request.
+var errInternalRedir = fmt.Errorf("internal redirect; re-authorization required")
+
 const (
 	rawConfigKey = "config"
 	idKey        = "@id"
@@ -869,4 +1237,9 @@ var bufPool = sync.Pool{
 	},
 }
 
-var adminServer *http.Server
+// keep a reference to admin endpoint singletons while they're active
+var (
+	serverMu                            sync.Mutex
+	localAdminServer, remoteAdminServer *http.Server
+	identityCertCache                   *certmagic.Cache
+)

admin_test.go

@@ -17,9 +17,28 @@ package caddy
 import (
 	"encoding/json"
 	"reflect"
+	"sync"
 	"testing"
 )
 
+var testCfg = []byte(`{
+			"apps": {
+				"http": {
+					"servers": {
+						"myserver": {
+							"listen": ["tcp/localhost:8080-8084"],
+							"read_timeout": "30s"
+						},
+						"yourserver": {
+							"listen": ["127.0.0.1:5000"],
+							"read_header_timeout": "15s"
+						}
+					}
+				}
+			}
+		}
+		`)
+
 func TestUnsyncedConfigAccess(t *testing.T) {
 	// each test is performed in sequence, so
 	// each change builds on the previous ones;
@@ -108,25 +127,24 @@ func TestUnsyncedConfigAccess(t *testing.T) {
 	}
 }
 
+// TestLoadConcurrent exercises Load under concurrent conditions
+// and is most useful under test with `-race` enabled.
+func TestLoadConcurrent(t *testing.T) {
+	var wg sync.WaitGroup
+
+	for i := 0; i < 100; i++ {
+		wg.Add(1)
+		go func() {
+			_ = Load(testCfg, true)
+			wg.Done()
+		}()
+	}
+
+	wg.Wait()
+}
+
 func BenchmarkLoad(b *testing.B) {
 	for i := 0; i < b.N; i++ {
-		cfg := []byte(`{
-			"apps": {
-				"http": {
-					"servers": {
-						"myserver": {
-							"listen": ["tcp/localhost:8080-8084"],
-							"read_timeout": "30s"
-						},
-						"yourserver": {
-							"listen": ["127.0.0.1:5000"],
-							"read_header_timeout": "15s"
-						}
-					}
-				}
-			}
-		}
-		`)
-		Load(cfg, true)
+		Load(testCfg, true)
 	}
 }

caddy.go

@@ -32,7 +32,9 @@ import (
 	"sync"
 	"time"
 
+	"github.com/caddyserver/caddy/v2/notify"
 	"github.com/caddyserver/certmagic"
+	"github.com/google/uuid"
 	"go.uber.org/zap"
 )
 
@@ -99,6 +101,16 @@ func Run(cfg *Config) error {
 // if it is different from the current config or
 // forceReload is true.
 func Load(cfgJSON []byte, forceReload bool) error {
+	if err := notify.NotifyReloading(); err != nil {
+		Log().Error("unable to notify reloading to service manager", zap.Error(err))
+	}
+
+	defer func() {
+		if err := notify.NotifyReadiness(); err != nil {
+			Log().Error("unable to notify readiness to service manager", zap.Error(err))
+		}
+	}()
+
 	return changeConfig(http.MethodPost, "/"+rawConfigKey, cfgJSON, forceReload)
 }
 
@@ -130,8 +142,8 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 	newCfg, err := json.Marshal(rawCfg[rawConfigKey])
 	if err != nil {
 		return APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("encoding new config: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("encoding new config: %v", err),
 		}
 	}
 
@@ -146,14 +158,14 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 	err = indexConfigObjects(rawCfg[rawConfigKey], "/"+rawConfigKey, idx)
 	if err != nil {
 		return APIError{
-			Code: http.StatusInternalServerError,
-			Err:  fmt.Errorf("indexing config: %v", err),
+			HTTPStatus: http.StatusInternalServerError,
+			Err:        fmt.Errorf("indexing config: %v", err),
 		}
 	}
 
 	// load this new config; if it fails, we need to revert to
 	// our old representation of caddy's actual config
-	err = unsyncedDecodeAndRun(newCfg)
+	err = unsyncedDecodeAndRun(newCfg, true)
 	if err != nil {
 		if len(rawCfgJSON) > 0 {
 			// restore old config state to keep it consistent
@@ -233,8 +245,10 @@ func indexConfigObjects(ptr interface{}, configPath string, index map[string]str
 // it as the new config, replacing any other current config.
 // It does NOT update the raw config state, as this is a
 // lower-level function; most callers will want to use Load
-// instead. A write lock on currentCfgMu is required!
-func unsyncedDecodeAndRun(cfgJSON []byte) error {
+// instead. A write lock on currentCfgMu is required! If
+// allowPersist is false, it will not be persisted to disk,
+// even if it is configured to.
+func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
 	// remove any @id fields from the JSON, which would cause
 	// loading to break since the field wouldn't be recognized
 	strippedCfgJSON := RemoveMetaFields(cfgJSON)
@@ -245,6 +259,20 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 		return err
 	}
 
+	// prevent recursive config loads; that is a user error, and
+	// although frequent config loads should be safe, we cannot
+	// guarantee that in the presence of third party plugins, nor
+	// do we want this error to go unnoticed (we assume it was a
+	// pulled config if we're not allowed to persist it)
+	if !allowPersist &&
+		newCfg != nil &&
+		newCfg.Admin != nil &&
+		newCfg.Admin.Config != nil &&
+		newCfg.Admin.Config.LoadRaw != nil &&
+		newCfg.Admin.Config.LoadInterval <= 0 {
+		return fmt.Errorf("recursive config loading detected: pulled configs cannot pull other configs without positive load_interval")
+	}
+
 	// run the new config and start all its apps
 	err = run(newCfg, true)
 	if err != nil {
@@ -259,7 +287,8 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 	unsyncedStop(oldCfg)
 
 	// autosave a non-nil config, if not disabled
-	if newCfg != nil &&
+	if allowPersist &&
+		newCfg != nil &&
 		(newCfg.Admin == nil ||
 			newCfg.Admin.Config == nil ||
 			newCfg.Admin.Config.Persist == nil ||
@@ -273,7 +302,7 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 		} else {
 			err := ioutil.WriteFile(ConfigAutosavePath, cfgJSON, 0600)
 			if err == nil {
-				Log().Info("autosaved config", zap.String("file", ConfigAutosavePath))
+				Log().Info("autosaved config (load with --resume flag)", zap.String("file", ConfigAutosavePath))
 			} else {
 				Log().Error("unable to autosave config",
 					zap.String("file", ConfigAutosavePath),
@@ -309,21 +338,10 @@ func run(newCfg *Config, start bool) error {
 	// been set by a short assignment
 	var err error
 
-	// start the admin endpoint (and stop any prior one)
-	if start {
-		err = replaceAdmin(newCfg)
-		if err != nil {
-			return fmt.Errorf("starting caddy administration endpoint: %v", err)
-		}
-	}
-
 	if newCfg == nil {
-		return nil
+		newCfg = new(Config)
 	}
 
-	// prepare the new config for use
-	newCfg.apps = make(map[string]App)
-
 	// create a context within which to load
 	// modules - essentially our new config's
 	// execution environment; be sure that
@@ -357,6 +375,17 @@ func run(newCfg *Config, start bool) error {
 		return err
 	}
 
+	// start the admin endpoint (and stop any prior one)
+	if start {
+		err = replaceLocalAdminServer(newCfg)
+		if err != nil {
+			return fmt.Errorf("starting caddy administration endpoint: %v", err)
+		}
+	}
+
+	// prepare the new config for use
+	newCfg.apps = make(map[string]App)
+
 	// set up global storage and make it CertMagic's default storage, too
 	err = func() error {
 		if newCfg.StorageRaw != nil {
@@ -400,7 +429,7 @@ func run(newCfg *Config, start bool) error {
 	}
 
 	// Start
-	return func() error {
+	err = func() error {
 		var started []string
 		for name, a := range newCfg.apps {
 			err := a.Start()
@@ -420,6 +449,83 @@ func run(newCfg *Config, start bool) error {
 		}
 		return nil
 	}()
+	if err != nil {
+		return err
+	}
+
+	// now that the user's config is running, finish setting up anything else,
+	// such as remote admin endpoint, config loader, etc.
+	return finishSettingUp(ctx, newCfg)
+}
+
+// finishSettingUp should be run after all apps have successfully started.
+func finishSettingUp(ctx Context, cfg *Config) error {
+	// establish this server's identity (only after apps are loaded
+	// so that cert management of this endpoint doesn't prevent user's
+	// servers from starting which likely also use HTTP/HTTPS ports;
+	// but before remote management which may depend on these creds)
+	err := manageIdentity(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("provisioning remote admin endpoint: %v", err)
+	}
+
+	// replace any remote admin endpoint
+	err = replaceRemoteAdminServer(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("provisioning remote admin endpoint: %v", err)
+	}
+
+	// if dynamic config is requested, set that up and run it
+	if cfg != nil && cfg.Admin != nil && cfg.Admin.Config != nil && cfg.Admin.Config.LoadRaw != nil {
+		val, err := ctx.LoadModule(cfg.Admin.Config, "LoadRaw")
+		if err != nil {
+			return fmt.Errorf("loading config loader module: %s", err)
+		}
+		runLoadedConfig := func(config []byte) {
+			Log().Info("applying dynamically-loaded config", zap.String("loader_module", val.(Module).CaddyModule().ID.Name()), zap.Int("pull_interval", int(cfg.Admin.Config.LoadInterval)))
+			currentCfgMu.Lock()
+			err := unsyncedDecodeAndRun(config, false)
+			currentCfgMu.Unlock()
+			if err == nil {
+				Log().Info("dynamically-loaded config applied successfully")
+			} else {
+				Log().Error("running dynamically-loaded config failed", zap.Error(err))
+			}
+		}
+		if cfg.Admin.Config.LoadInterval > 0 {
+			go func() {
+				select {
+				// if LoadInterval is positive, will wait for the interval and then run with new config
+				case <-time.After(time.Duration(cfg.Admin.Config.LoadInterval)):
+					loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
+					if err != nil {
+						Log().Error("loading dynamic config failed", zap.Error(err))
+						return
+					}
+					runLoadedConfig(loadedConfig)
+				case <-ctx.Done():
+					return
+				}
+			}()
+		} else {
+			// if no LoadInterval is provided, will load config synchronously
+			loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
+			if err != nil {
+				return fmt.Errorf("loading dynamic config from %T: %v", val, err)
+			}
+			// do this in a goroutine so current config can finish being loaded; otherwise deadlock
+			go runLoadedConfig(loadedConfig)
+		}
+
+	}
+
+	return nil
+}
+
+// ConfigLoader is a type that can load a Caddy config. The
+// returned config must be valid Caddy JSON.
+type ConfigLoader interface {
+	LoadConfig(Context) ([]byte, error)
 }
 
 // Stop stops running the current configuration.
@@ -462,20 +568,6 @@ func unsyncedStop(cfg *Config) {
 	cfg.cancelFunc()
 }
 
-// stopAndCleanup calls stop and cleans up anything
-// else that is expedient. This should only be used
-// when stopping and not replacing with a new config.
-func stopAndCleanup() error {
-	if err := Stop(); err != nil {
-		return err
-	}
-	certmagic.CleanUpOwnLocks()
-	if pidfile != "" {
-		return os.Remove(pidfile)
-	}
-	return nil
-}
-
 // Validate loads, provisions, and validates
 // cfg, but does not start running it.
 func Validate(cfg *Config) error {
@@ -486,6 +578,72 @@ func Validate(cfg *Config) error {
 	return err
 }
 
+// exitProcess exits the process as gracefully as possible,
+// but it always exits, even if there are errors doing so.
+// It stops all apps, cleans up external locks, removes any
+// PID file, and shuts down admin endpoint(s) in a goroutine.
+// Errors are logged along the way, and an appropriate exit
+// code is emitted.
+func exitProcess(logger *zap.Logger) {
+	if logger == nil {
+		logger = Log()
+	}
+	logger.Warn("exiting; byeee!! 👋")
+
+	exitCode := ExitCodeSuccess
+
+	// stop all apps
+	if err := Stop(); err != nil {
+		logger.Error("failed to stop apps", zap.Error(err))
+		exitCode = ExitCodeFailedQuit
+	}
+
+	// clean up certmagic locks
+	certmagic.CleanUpOwnLocks(logger)
+
+	// remove pidfile
+	if pidfile != "" {
+		err := os.Remove(pidfile)
+		if err != nil {
+			logger.Error("cleaning up PID file:",
+				zap.String("pidfile", pidfile),
+				zap.Error(err))
+			exitCode = ExitCodeFailedQuit
+		}
+	}
+
+	// shut down admin endpoint(s) in goroutines so that
+	// if this function was called from an admin handler,
+	// it has a chance to return gracefully
+	// use goroutine so that we can finish responding to API request
+	go func() {
+		defer func() {
+			logger = logger.With(zap.Int("exit_code", exitCode))
+			if exitCode == ExitCodeSuccess {
+				logger.Info("shutdown complete")
+			} else {
+				logger.Error("unclean shutdown")
+			}
+			os.Exit(exitCode)
+		}()
+
+		if remoteAdminServer != nil {
+			err := stopAdminServer(remoteAdminServer)
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				logger.Error("failed to stop remote admin server gracefully", zap.Error(err))
+			}
+		}
+		if localAdminServer != nil {
+			err := stopAdminServer(localAdminServer)
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				logger.Error("failed to stop local admin server gracefully", zap.Error(err))
+			}
+		}
+	}()
+}
+
 // Duration can be an integer or a string. An integer is
 // interpreted as nanoseconds. If a string, it is a Go
 // time.Duration value such as `300ms`, `1.5h`, or `2h45m`;
@@ -536,6 +694,26 @@ func ParseDuration(s string) (time.Duration, error) {
 	return time.ParseDuration(s)
 }
 
+// InstanceID returns the UUID for this instance, and generates one if it
+// does not already exist. The UUID is stored in the local data directory,
+// regardless of storage configuration, since each instance is intended to
+// have its own unique ID.
+func InstanceID() (uuid.UUID, error) {
+	uuidFilePath := filepath.Join(AppDataDir(), "instance.uuid")
+	uuidFileBytes, err := ioutil.ReadFile(uuidFilePath)
+	if os.IsNotExist(err) {
+		uuid, err := uuid.NewRandom()
+		if err != nil {
+			return uuid, err
+		}
+		err = ioutil.WriteFile(uuidFilePath, []byte(uuid.String()), 0600)
+		return uuid, err
+	} else if err != nil {
+		return [16]byte{}, err
+	}
+	return uuid.ParseBytes(uuidFileBytes)
+}
+
 // GoModule returns the build info of this Caddy
 // build from debug.BuildInfo (requires Go modules).
 // If no version information is available, a non-nil

caddyconfig/caddyfile/adapter.go

@@ -15,6 +15,7 @@
 package caddyfile
 
 import (
+	"bytes"
 	"encoding/json"
 	"fmt"
 
@@ -51,15 +52,46 @@ func (a Adapter) Adapt(body []byte, options map[string]interface{}) ([]byte, []c
 		return nil, warnings, err
 	}
 
-	marshalFunc := json.Marshal
-	if options["pretty"] == "true" {
-		marshalFunc = caddyconfig.JSONIndent
+	// lint check: see if input was properly formatted; sometimes messy files files parse
+	// successfully but result in logical errors (the Caddyfile is a bad format, I'm sorry)
+	if warning, different := formattingDifference(filename, body); different {
+		warnings = append(warnings, warning)
 	}
-	result, err := marshalFunc(cfg)
+
+	result, err := json.Marshal(cfg)
 
 	return result, warnings, err
 }
 
+// formattingDifference returns a warning and true if the formatted version
+// is any different from the input; empty warning and false otherwise.
+// TODO: also perform this check on imported files
+func formattingDifference(filename string, body []byte) (caddyconfig.Warning, bool) {
+	// replace windows-style newlines to normalize comparison
+	normalizedBody := bytes.Replace(body, []byte("\r\n"), []byte("\n"), -1)
+
+	formatted := Format(normalizedBody)
+	if bytes.Equal(formatted, normalizedBody) {
+		return caddyconfig.Warning{}, false
+	}
+
+	// find where the difference is
+	line := 1
+	for i, ch := range normalizedBody {
+		if i >= len(formatted) || ch != formatted[i] {
+			break
+		}
+		if ch == '\n' {
+			line++
+		}
+	}
+	return caddyconfig.Warning{
+		File:    filename,
+		Line:    line,
+		Message: "input is not formatted with 'caddy fmt'",
+	}, true
+}
+
 // Unmarshaler is a type that can unmarshal
 // Caddyfile tokens to set itself up for a
 // JSON encoding. The goal of an unmarshaler
@@ -87,5 +119,31 @@ type ServerType interface {
 	Setup([]ServerBlock, map[string]interface{}) (*caddy.Config, []caddyconfig.Warning, error)
 }
 
+// UnmarshalModule instantiates a module with the given ID and invokes
+// UnmarshalCaddyfile on the new value using the immediate next segment
+// of d as input. In other words, d's next token should be the first
+// token of the module's Caddyfile input.
+//
+// This function is used when the next segment of Caddyfile tokens
+// belongs to another Caddy module. The returned value is often
+// type-asserted to the module's associated type for practical use
+// when setting up a config.
+func UnmarshalModule(d *Dispenser, moduleID string) (Unmarshaler, error) {
+	mod, err := caddy.GetModule(moduleID)
+	if err != nil {
+		return nil, d.Errf("getting module named '%s': %v", moduleID, err)
+	}
+	inst := mod.New()
+	unm, ok := inst.(Unmarshaler)
+	if !ok {
+		return nil, d.Errf("module %s is not a Caddyfile unmarshaler; is %T", mod.ID, inst)
+	}
+	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
+	if err != nil {
+		return nil, err
+	}
+	return unm, nil
+}
+
 // Interface guard
 var _ caddyconfig.Adapter = (*Adapter)(nil)

caddyconfig/caddyfile/dispenser.go

@@ -345,13 +345,13 @@ func (d *Dispenser) EOFErr() error {
 
 // Err generates a custom parse-time error with a message of msg.
 func (d *Dispenser) Err(msg string) error {
-	msg = fmt.Sprintf("%s:%d - Error during parsing: %s", d.File(), d.Line(), msg)
-	return errors.New(msg)
+	return d.Errf(msg)
 }
 
 // Errf is like Err, but for formatted error messages
 func (d *Dispenser) Errf(format string, args ...interface{}) error {
-	return d.Err(fmt.Sprintf(format, args...))
+	err := fmt.Errorf(format, args...)
+	return fmt.Errorf("%s:%d - Error during parsing: %w", d.File(), d.Line(), err)
 }
 
 // Delete deletes the current token and returns the updated slice

caddyconfig/caddyfile/dispenser_test.go

@@ -15,6 +15,7 @@
 package caddyfile
 
 import (
+	"errors"
 	"reflect"
 	"strings"
 	"testing"
@@ -303,4 +304,10 @@ func TestDispenser_ArgErr_Err(t *testing.T) {
 	if !strings.Contains(err.Error(), "foobar") {
 		t.Errorf("Expected error message with custom message in it ('foobar'); got '%v'", err)
 	}
+
+	var ErrBarIsFull = errors.New("bar is full")
+	bookingError := d.Errf("unable to reserve: %w", ErrBarIsFull)
+	if !errors.Is(bookingError, ErrBarIsFull) {
+		t.Errorf("Errf(): should be able to unwrap the error chain")
+	}
 }

caddyconfig/caddyfile/formatter.go

@@ -78,6 +78,7 @@ func Format(input []byte) []byte {
 		if comment {
 			if ch == '\n' {
 				comment = false
+				space = true
 				nextLine()
 				continue
 			} else {

caddyconfig/caddyfile/formatter_fuzz.go

@@ -0,0 +1,27 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// +build gofuzz
+
+package caddyfile
+
+import "bytes"
+
+func FuzzFormat(input []byte) int {
+	formatted := Format(input)
+	if bytes.Equal(formatted, Format(formatted)) {
+		return 1
+	}
+	return 0
+}

caddyconfig/caddyfile/formatter_test.go

@@ -321,6 +321,44 @@ baz`,
 	foo
 }`,
 		},
+		{
+			description: "matthewpi/vscode-caddyfile-support#13",
+			input: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+			expect: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+		},
+		{
+			description: "matthewpi/vscode-caddyfile-support#13 - bad formatting",
+			input: `{
+	email {$ACMEEMAIL}
+	#debug
+	}
+
+	block {
+	}
+`,
+			expect: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+		},
 	} {
 		// the formatter should output a trailing newline,
 		// even if the tests aren't written to expect that

caddyconfig/caddyfile/importgraph.go

@@ -0,0 +1,127 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"fmt"
+)
+
+type adjacency map[string][]string
+
+type importGraph struct {
+	nodes map[string]bool
+	edges adjacency
+}
+
+func (i *importGraph) addNode(name string) {
+	if i.nodes == nil {
+		i.nodes = make(map[string]bool)
+	}
+	if _, exists := i.nodes[name]; exists {
+		return
+	}
+	i.nodes[name] = true
+}
+func (i *importGraph) addNodes(names []string) {
+	for _, name := range names {
+		i.addNode(name)
+	}
+}
+
+func (i *importGraph) removeNode(name string) {
+	delete(i.nodes, name)
+}
+func (i *importGraph) removeNodes(names []string) {
+	for _, name := range names {
+		i.removeNode(name)
+	}
+}
+
+func (i *importGraph) addEdge(from, to string) error {
+	if !i.exists(from) || !i.exists(to) {
+		return fmt.Errorf("one of the nodes does not exist")
+	}
+
+	if i.willCycle(to, from) {
+		return fmt.Errorf("a cycle of imports exists between %s and %s", from, to)
+	}
+
+	if i.areConnected(from, to) {
+		// if connected, there's nothing to do
+		return nil
+	}
+
+	if i.nodes == nil {
+		i.nodes = make(map[string]bool)
+	}
+	if i.edges == nil {
+		i.edges = make(adjacency)
+	}
+
+	i.edges[from] = append(i.edges[from], to)
+	return nil
+}
+func (i *importGraph) addEdges(from string, tos []string) error {
+	for _, to := range tos {
+		err := i.addEdge(from, to)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (i *importGraph) areConnected(from, to string) bool {
+	al, ok := i.edges[from]
+	if !ok {
+		return false
+	}
+	for _, v := range al {
+		if v == to {
+			return true
+		}
+	}
+	return false
+}
+
+func (i *importGraph) willCycle(from, to string) bool {
+	collector := make(map[string]bool)
+
+	var visit func(string)
+	visit = func(start string) {
+		if !collector[start] {
+			collector[start] = true
+			for _, v := range i.edges[start] {
+				visit(v)
+			}
+		}
+	}
+
+	for _, v := range i.edges[from] {
+		visit(v)
+	}
+	for k := range collector {
+		if to == k {
+			return true
+		}
+	}
+
+	return false
+}
+
+func (i *importGraph) exists(key string) bool {
+	_, exists := i.nodes[key]
+	return exists
+}

caddyconfig/caddyfile/lexer.go

@@ -35,9 +35,11 @@ type (
 
 	// Token represents a single parsable unit.
 	Token struct {
-		File string
-		Line int
-		Text string
+		File        string
+		Line        int
+		Text        string
+		inSnippet   bool
+		snippetName string
 	}
 )
 

caddyconfig/caddyfile/lexer_fuzz.go

@@ -0,0 +1,28 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// +build gofuzz
+
+package caddyfile
+
+func FuzzTokenize(input []byte) int {
+	tokens, err := Tokenize(input, "Caddyfile")
+	if err != nil {
+		return 0
+	}
+	if len(tokens) == 0 {
+		return -1
+	}
+	return 1
+}

caddyconfig/caddyfile/parse.go

@@ -16,6 +16,7 @@ package caddyfile
 
 import (
 	"bytes"
+	"fmt"
 	"io/ioutil"
 	"log"
 	"os"
@@ -40,7 +41,13 @@ func Parse(filename string, input []byte) ([]ServerBlock, error) {
 	if err != nil {
 		return nil, err
 	}
-	p := parser{Dispenser: NewDispenser(tokens)}
+	p := parser{
+		Dispenser: NewDispenser(tokens),
+		importGraph: importGraph{
+			nodes: make(map[string]bool),
+			edges: make(adjacency),
+		},
+	}
 	return p.parseAll()
 }
 
@@ -110,6 +117,7 @@ type parser struct {
 	eof             bool        // if we encounter a valid EOF in a hard place
 	definedSnippets map[string][]Token
 	nesting         int
+	importGraph     importGraph
 }
 
 func (p *parser) parseAll() ([]ServerBlock, error) {
@@ -165,6 +173,15 @@ func (p *parser) begin() error {
 		if err != nil {
 			return err
 		}
+		// Just as we need to track which file the token comes from, we need to
+		// keep track of which snippets do the tokens come from. This is helpful
+		// in tracking import cycles across files/snippets by namespacing them. Without
+		// this we end up with false-positives in cycle-detection.
+		for k, v := range tokens {
+			v.inSnippet = true
+			v.snippetName = name
+			tokens[k] = v
+		}
 		p.definedSnippets[name] = tokens
 		// empty block keys so we don't save this block as a real server.
 		p.block.Keys = nil
@@ -194,9 +211,20 @@ func (p *parser) addresses() error {
 			if expectingAnother {
 				return p.Errf("Expected another address but had '%s' - check for extra comma", tkn)
 			}
+			// Mark this server block as being defined with braces.
+			// This is used to provide a better error message when
+			// the user may have tried to define two server blocks
+			// without having used braces, which are required in
+			// that case.
+			p.block.HasBraces = true
 			break
 		}
 
+		// Users commonly forget to place a space between the address and the '{'
+		if strings.HasSuffix(tkn, "{") {
+			return p.Errf("Site addresses cannot end with a curly brace: '%s' - put a space between the token and the brace", tkn)
+		}
+
 		if tkn != "" { // empty token possible if user typed ""
 			// Trailing comma indicates another address will follow, which
 			// may possibly be on the next line
@@ -207,6 +235,13 @@ func (p *parser) addresses() error {
 				expectingAnother = false // but we may still see another one on this line
 			}
 
+			// If there's a comma here, it's probably because they didn't use a space
+			// between their two domains, e.g. "foo.com,bar.com", which would not be
+			// parsed as two separate site addresses.
+			if strings.Contains(tkn, ",") {
+				return p.Errf("Site addresses cannot contain a comma ',': '%s' - put a space after the comma to separate site addresses", tkn)
+			}
+
 			p.block.Keys = append(p.block.Keys, tkn)
 		}
 
@@ -304,7 +339,7 @@ func (p *parser) doImport() error {
 	args := p.RemainingArgs()
 
 	// add args to the replacer
-	repl := caddy.NewReplacer()
+	repl := caddy.NewEmptyReplacer()
 	for index, arg := range args {
 		repl.Set("args."+strconv.Itoa(index), arg)
 	}
@@ -314,10 +349,15 @@ func (p *parser) doImport() error {
 	tokensBefore := p.tokens[:p.cursor-1-len(args)]
 	tokensAfter := p.tokens[p.cursor+1:]
 	var importedTokens []Token
+	var nodes []string
 
 	// first check snippets. That is a simple, non-recursive replacement
 	if p.definedSnippets != nil && p.definedSnippets[importPattern] != nil {
 		importedTokens = p.definedSnippets[importPattern]
+		if len(importedTokens) > 0 {
+			// just grab the first one
+			nodes = append(nodes, fmt.Sprintf("%s:%s", importedTokens[0].File, importedTokens[0].snippetName))
+		}
 	} else {
 		// make path relative to the file of the _token_ being processed rather
 		// than current working directory (issue #867) and then use glob to get
@@ -353,7 +393,6 @@ func (p *parser) doImport() error {
 		}
 
 		// collect all the imported tokens
-
 		for _, importFile := range matches {
 			newTokens, err := p.doSingleImport(importFile)
 			if err != nil {
@@ -361,6 +400,18 @@ func (p *parser) doImport() error {
 			}
 			importedTokens = append(importedTokens, newTokens...)
 		}
+		nodes = matches
+	}
+
+	nodeName := p.File()
+	if p.Token().inSnippet {
+		nodeName += fmt.Sprintf(":%s", p.Token().snippetName)
+	}
+	p.importGraph.addNode(nodeName)
+	p.importGraph.addNodes(nodes)
+	if err := p.importGraph.addEdges(nodeName, nodes); err != nil {
+		p.importGraph.removeNodes(nodes)
+		return err
 	}
 
 	// copy the tokens so we don't overwrite p.definedSnippets
@@ -526,8 +577,9 @@ func (p *parser) snippetTokens() ([]Token, error) {
 // head of the server block with tokens, which are
 // grouped by segments.
 type ServerBlock struct {
-	Keys     []string
-	Segments []Segment
+	HasBraces bool
+	Keys      []string
+	Segments  []Segment
 }
 
 // DispenseDirective returns a dispenser that contains

caddyconfig/caddyfile/parse_test.go

@@ -160,6 +160,10 @@ func TestParseOneAndImport(t *testing.T) {
 			"localhost",
 		}, []int{}},
 
+		{`localhost{
+		    dir1
+		  }`, true, []string{}, []int{}},
+
 		{`localhost
 		  dir1 {
 		    nested {
@@ -444,6 +448,28 @@ func TestParseAll(t *testing.T) {
 
 		{`import notfound/*`, false, [][]string{}},        // glob needn't error with no matches
 		{`import notfound/file.conf`, true, [][]string{}}, // but a specific file should
+
+		// recursive self-import
+		{`import testdata/import_recursive0.txt`, true, [][]string{}},
+		{`import testdata/import_recursive3.txt
+		import testdata/import_recursive1.txt`, true, [][]string{}},
+
+		// cyclic imports
+		{`(A) {
+			import A
+		}
+		:80
+		import A
+		`, true, [][]string{}},
+		{`(A) {
+			import B
+		}
+		(B) {
+			import A
+		}
+		:80
+		import A
+		`, true, [][]string{}},
 	} {
 		p := testParser(test.input)
 		blocks, err := p.parseAll()

caddyconfig/caddyfile/testdata/import_recursive0.txt

@@ -0,0 +1 @@
+import import_recursive0.txt

caddyconfig/caddyfile/testdata/import_recursive1.txt

@@ -0,0 +1 @@
+import import_recursive2.txt

caddyconfig/caddyfile/testdata/import_recursive2.txt

@@ -0,0 +1 @@
+import import_recursive3.txt

caddyconfig/caddyfile/testdata/import_recursive3.txt

@@ -0,0 +1 @@
+import import_recursive1.txt

caddyconfig/configadapters.go

@@ -35,6 +35,14 @@ type Warning struct {
 	Message   string `json:"message,omitempty"`
 }
 
+func (w Warning) String() string {
+	var directive string
+	if w.Directive != "" {
+		directive = fmt.Sprintf(" (%s)", w.Directive)
+	}
+	return fmt.Sprintf("%s:%d%s: %s", w.File, w.Line, directive, w.Message)
+}
+
 // JSON encodes val as JSON, returning it as a json.RawMessage. Any
 // marshaling errors (which are highly unlikely with correct code)
 // are converted to warnings. This is convenient when filling config
@@ -93,12 +101,6 @@ func JSONModuleObject(val interface{}, fieldName, fieldVal string, warnings *[]W
 	return result
 }
 
-// JSONIndent is used to JSON-marshal the final resulting Caddy
-// configuration in a consistent, human-readable way.
-func JSONIndent(val interface{}) ([]byte, error) {
-	return json.MarshalIndent(val, "", "\t")
-}
-
 // RegisterAdapter registers a config adapter with the given name.
 // This should usually be done at init-time. It panics if the
 // adapter cannot be registered successfully.

caddyconfig/httpcaddyfile/builtins.go

@@ -22,6 +22,7 @@ import (
 	"io/ioutil"
 	"net/http"
 	"reflect"
+	"strconv"
 	"strings"
 
 	"github.com/caddyserver/caddy/v2"
@@ -40,6 +41,8 @@ func init() {
 	RegisterHandlerDirective("root", parseRoot)
 	RegisterHandlerDirective("redir", parseRedir)
 	RegisterHandlerDirective("respond", parseRespond)
+	RegisterHandlerDirective("abort", parseAbort)
+	RegisterHandlerDirective("error", parseError)
 	RegisterHandlerDirective("route", parseRoute)
 	RegisterHandlerDirective("handle", parseHandle)
 	RegisterDirective("handle_errors", parseHandleErrors)
@@ -87,6 +90,7 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 	var folderLoader caddytls.FolderLoader
 	var certSelector caddytls.CustomCertSelectionPolicy
 	var acmeIssuer *caddytls.ACMEIssuer
+	var keyType string
 	var internalIssuer *caddytls.InternalIssuer
 	var issuers []certmagic.Issuer
 	var onDemand bool
@@ -122,10 +126,10 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 			// must load each cert only once; otherwise, they each get a
 			// different tag... since a cert loaded twice has the same
 			// bytes, it will overwrite the first one in the cache, and
-			// only the last cert (and its tag) will survive, so a any conn
-			// policy that is looking for any tag but the last one to be
-			// loaded won't find it, and TLS handshakes will fail (see end)
-			// of issue #3004)
+			// only the last cert (and its tag) will survive, so any conn
+			// policy that is looking for any tag other than the last one
+			// to be loaded won't find it, and TLS handshakes will fail
+			// (see end of issue #3004)
 			//
 			// tlsCertTags maps certificate filenames to their tag.
 			// This is used to remember which tag is used for each
@@ -267,6 +271,13 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 				}
 				acmeIssuer.CA = arg[0]
 
+			case "key_type":
+				arg := h.RemainingArgs()
+				if len(arg) != 1 {
+					return nil, h.ArgErr()
+				}
+				keyType = arg[0]
+
 			case "eab":
 				arg := h.RemainingArgs()
 				if len(arg) != 2 {
@@ -285,21 +296,14 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 					return nil, h.ArgErr()
 				}
 				modName := h.Val()
-				mod, err := caddy.GetModule("tls.issuance." + modName)
-				if err != nil {
-					return nil, h.Errf("getting issuer module '%s': %v", modName, err)
-				}
-				unm, ok := mod.New().(caddyfile.Unmarshaler)
-				if !ok {
-					return nil, h.Errf("issuer module '%s' is not a Caddyfile unmarshaler", mod.ID)
-				}
-				err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+				modID := "tls.issuance." + modName
+				unm, err := caddyfile.UnmarshalModule(h.Dispenser, modID)
 				if err != nil {
 					return nil, err
 				}
 				issuer, ok := unm.(certmagic.Issuer)
 				if !ok {
-					return nil, h.Errf("module %s is not a certmagic.Issuer", mod.ID)
+					return nil, h.Errf("module %s (%T) is not a certmagic.Issuer", modID, unm)
 				}
 				issuers = append(issuers, issuer)
 
@@ -313,20 +317,32 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 				}
 				if acmeIssuer.Challenges == nil {
 					acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
+				}
+				if acmeIssuer.Challenges.DNS == nil {
 					acmeIssuer.Challenges.DNS = new(caddytls.DNSChallengeConfig)
 				}
-				dnsProvModule, err := caddy.GetModule("dns.providers." + provName)
+				modID := "dns.providers." + provName
+				unm, err := caddyfile.UnmarshalModule(h.Dispenser, modID)
 				if err != nil {
-					return nil, h.Errf("getting DNS provider module named '%s': %v", provName, err)
+					return nil, err
 				}
-				dnsProvModuleInstance := dnsProvModule.New()
-				if unm, ok := dnsProvModuleInstance.(caddyfile.Unmarshaler); ok {
-					err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
-					if err != nil {
-						return nil, err
-					}
+				acmeIssuer.Challenges.DNS.ProviderRaw = caddyconfig.JSONModuleObject(unm, "name", provName, h.warnings)
+
+			case "resolvers":
+				args := h.RemainingArgs()
+				if len(args) == 0 {
+					return nil, h.ArgErr()
 				}
-				acmeIssuer.Challenges.DNS.ProviderRaw = caddyconfig.JSONModuleObject(dnsProvModuleInstance, "name", provName, h.warnings)
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
+				if acmeIssuer.Challenges == nil {
+					acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
+				}
+				if acmeIssuer.Challenges.DNS == nil {
+					acmeIssuer.Challenges.DNS = new(caddytls.DNSChallengeConfig)
+				}
+				acmeIssuer.Challenges.DNS.Resolvers = args
 
 			case "ca_root":
 				arg := h.RemainingArgs()
@@ -372,31 +388,64 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 		})
 	}
 
+	// some tls subdirectives are shortcuts that implicitly configure issuers, and the
+	// user can also configure issuers explicitly using the issuer subdirective; the
+	// logic to support both would likely be complex, or at least unintuitive
 	if len(issuers) > 0 && (acmeIssuer != nil || internalIssuer != nil) {
-		// some tls subdirectives are shortcuts that implicitly configure issuers, and the
-		// user can also configure issuers explicitly using the issuer subdirective; the
-		// logic to support both would likely be complex, or at least unintuitive
 		return nil, h.Err("cannot mix issuer subdirective (explicit issuers) with other issuer-specific subdirectives (implicit issuers)")
 	}
-	for _, issuer := range issuers {
-		configVals = append(configVals, ConfigValue{
-			Class: "tls.cert_issuer",
-			Value: issuer,
-		})
+	if acmeIssuer != nil && internalIssuer != nil {
+		return nil, h.Err("cannot create both ACME and internal certificate issuers")
 	}
-	if acmeIssuer != nil {
-		configVals = append(configVals, ConfigValue{
-			Class: "tls.cert_issuer",
-			Value: disambiguateACMEIssuer(acmeIssuer),
-		})
-	}
-	if internalIssuer != nil {
+
+	// now we should either have: explicitly-created issuers, or an implicitly-created
+	// ACME or internal issuer, or no issuers at all
+	switch {
+	case len(issuers) > 0:
+		for _, issuer := range issuers {
+			configVals = append(configVals, ConfigValue{
+				Class: "tls.cert_issuer",
+				Value: issuer,
+			})
+		}
+
+	case acmeIssuer != nil:
+		// implicit ACME issuers (from various subdirectives) - use defaults; there might be more than one
+		defaultIssuers := caddytls.DefaultIssuers()
+
+		// if a CA endpoint was set, override multiple implicit issuers since it's a specific one
+		if acmeIssuer.CA != "" {
+			defaultIssuers = []certmagic.Issuer{acmeIssuer}
+		}
+
+		for _, issuer := range defaultIssuers {
+			switch iss := issuer.(type) {
+			case *caddytls.ACMEIssuer:
+				issuer = acmeIssuer
+			case *caddytls.ZeroSSLIssuer:
+				iss.ACMEIssuer = acmeIssuer
+			}
+			configVals = append(configVals, ConfigValue{
+				Class: "tls.cert_issuer",
+				Value: issuer,
+			})
+		}
+
+	case internalIssuer != nil:
 		configVals = append(configVals, ConfigValue{
 			Class: "tls.cert_issuer",
 			Value: internalIssuer,
 		})
 	}
 
+	// certificate key type
+	if keyType != "" {
+		configVals = append(configVals, ConfigValue{
+			Class: "tls.key_type",
+			Value: keyType,
+		})
+	}
+
 	// on-demand TLS
 	if onDemand {
 		configVals = append(configVals, ConfigValue{
@@ -459,14 +508,14 @@ func parseRedir(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	if h.NextArg() {
 		code = h.Val()
 	}
-	if code == "permanent" {
-		code = "301"
-	}
-	if code == "temporary" || code == "" {
-		code = "302"
-	}
+
 	var body string
-	if code == "html" {
+	switch code {
+	case "permanent":
+		code = "301"
+	case "temporary", "":
+		code = "302"
+	case "html":
 		// Script tag comes first since that will better imitate a redirect in the browser's
 		// history, but the meta tag is a fallback for most non-JS clients.
 		const metaRedir = `<!DOCTYPE html>
@@ -481,6 +530,15 @@ func parseRedir(h Helper) (caddyhttp.MiddlewareHandler, error) {
 `
 		safeTo := html.EscapeString(to)
 		body = fmt.Sprintf(metaRedir, safeTo, safeTo, safeTo, safeTo)
+		code = "302"
+	default:
+		codeInt, err := strconv.Atoi(code)
+		if err != nil {
+			return nil, h.Errf("Not a supported redir code type or not valid integer: '%s'", code)
+		}
+		if codeInt < 300 || codeInt > 399 {
+			return nil, h.Errf("Redir code not in the 3xx range: '%v'", codeInt)
+		}
 	}
 
 	return caddyhttp.StaticResponse{
@@ -500,6 +558,25 @@ func parseRespond(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	return sr, nil
 }
 
+// parseAbort parses the abort directive.
+func parseAbort(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	h.Next() // consume directive
+	for h.Next() || h.NextBlock(0) {
+		return nil, h.ArgErr()
+	}
+	return &caddyhttp.StaticResponse{Abort: true}, nil
+}
+
+// parseError parses the error directive.
+func parseError(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	se := new(caddyhttp.StaticError)
+	err := se.UnmarshalCaddyfile(h.Dispenser)
+	if err != nil {
+		return nil, err
+	}
+	return se, nil
+}
+
 // parseRoute parses the route directive.
 func parseRoute(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	sr := new(caddyhttp.Subroute)
@@ -553,11 +630,50 @@ func parseHandleErrors(h Helper) ([]ConfigValue, error) {
 //     }
 //
 func parseLog(h Helper) ([]ConfigValue, error) {
+	return parseLogHelper(h, nil)
+}
+
+// parseLogHelper is used both for the parseLog directive within Server Blocks,
+// as well as the global "log" option for configuring loggers at the global
+// level. The parseAsGlobalOption parameter is used to distinguish any differing logic
+// between the two.
+func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue, error) {
+	// When the globalLogNames parameter is passed in, we make
+	// modifications to the parsing behavior.
+	parseAsGlobalOption := globalLogNames != nil
+
 	var configValues []ConfigValue
 	for h.Next() {
-		// log does not currently support any arguments
-		if h.NextArg() {
-			return nil, h.ArgErr()
+		// Logic below expects that a name is always present when a
+		// global option is being parsed.
+		var globalLogName string
+		if parseAsGlobalOption {
+			if h.NextArg() {
+				globalLogName = h.Val()
+
+				// Only a single argument is supported.
+				if h.NextArg() {
+					return nil, h.ArgErr()
+				}
+			} else {
+				// If there is no log name specified, we
+				// reference the default logger. See the
+				// setupNewDefault function in the logging
+				// package for where this is configured.
+				globalLogName = "default"
+			}
+
+			// Verify this name is unused.
+			_, used := globalLogNames[globalLogName]
+			if used {
+				return nil, h.Err("duplicate global log option for: " + globalLogName)
+			}
+			globalLogNames[globalLogName] = struct{}{}
+		} else {
+			// No arguments are supported for the server block log directive
+			if h.NextArg() {
+				return nil, h.ArgErr()
+			}
 		}
 
 		cl := new(caddy.CustomLog)
@@ -583,21 +699,15 @@ func parseLog(h Helper) ([]ConfigValue, error) {
 				case "discard":
 					wo = caddy.DiscardWriter{}
 				default:
-					mod, err := caddy.GetModule("caddy.logging.writers." + moduleName)
-					if err != nil {
-						return nil, h.Errf("getting log writer module named '%s': %v", moduleName, err)
-					}
-					unm, ok := mod.New().(caddyfile.Unmarshaler)
-					if !ok {
-						return nil, h.Errf("log writer module '%s' is not a Caddyfile unmarshaler", mod)
-					}
-					err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+					modID := "caddy.logging.writers." + moduleName
+					unm, err := caddyfile.UnmarshalModule(h.Dispenser, modID)
 					if err != nil {
 						return nil, err
 					}
+					var ok bool
 					wo, ok = unm.(caddy.WriterOpener)
 					if !ok {
-						return nil, h.Errf("module %s is not a WriterOpener", mod)
+						return nil, h.Errf("module %s (%T) is not a WriterOpener", modID, unm)
 					}
 				}
 				cl.WriterRaw = caddyconfig.JSONModuleObject(wo, "output", moduleName, h.warnings)
@@ -607,21 +717,14 @@ func parseLog(h Helper) ([]ConfigValue, error) {
 					return nil, h.ArgErr()
 				}
 				moduleName := h.Val()
-				mod, err := caddy.GetModule("caddy.logging.encoders." + moduleName)
-				if err != nil {
-					return nil, h.Errf("getting log encoder module named '%s': %v", moduleName, err)
-				}
-				unm, ok := mod.New().(caddyfile.Unmarshaler)
-				if !ok {
-					return nil, h.Errf("log encoder module '%s' is not a Caddyfile unmarshaler", mod)
-				}
-				err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+				moduleID := "caddy.logging.encoders." + moduleName
+				unm, err := caddyfile.UnmarshalModule(h.Dispenser, moduleID)
 				if err != nil {
 					return nil, err
 				}
 				enc, ok := unm.(zapcore.Encoder)
 				if !ok {
-					return nil, h.Errf("module %s is not a zapcore.Encoder", mod)
+					return nil, h.Errf("module %s (%T) is not a zapcore.Encoder", moduleID, unm)
 				}
 				cl.EncoderRaw = caddyconfig.JSONModuleObject(enc, "format", moduleName, h.warnings)
 
@@ -634,22 +737,48 @@ func parseLog(h Helper) ([]ConfigValue, error) {
 					return nil, h.ArgErr()
 				}
 
+			case "include":
+				// This configuration is only allowed in the global options
+				if !parseAsGlobalOption {
+					return nil, h.ArgErr()
+				}
+				for h.NextArg() {
+					cl.Include = append(cl.Include, h.Val())
+				}
+
+			case "exclude":
+				// This configuration is only allowed in the global options
+				if !parseAsGlobalOption {
+					return nil, h.ArgErr()
+				}
+				for h.NextArg() {
+					cl.Exclude = append(cl.Exclude, h.Val())
+				}
+
 			default:
 				return nil, h.Errf("unrecognized subdirective: %s", h.Val())
 			}
 		}
 
 		var val namedCustomLog
+		// Skip handling of empty logging configs
 		if !reflect.DeepEqual(cl, new(caddy.CustomLog)) {
-			logCounter, ok := h.State["logCounter"].(int)
-			if !ok {
-				logCounter = 0
+			if parseAsGlobalOption {
+				// Use indicated name for global log options
+				val.name = globalLogName
+				val.log = cl
+			} else {
+				// Construct a log name for server log streams
+				logCounter, ok := h.State["logCounter"].(int)
+				if !ok {
+					logCounter = 0
+				}
+				val.name = fmt.Sprintf("log%d", logCounter)
+				cl.Include = []string{"http.log.access." + val.name}
+				val.log = cl
+				logCounter++
+				h.State["logCounter"] = logCounter
 			}
-			val.name = fmt.Sprintf("log%d", logCounter)
-			cl.Include = []string{"http.log.access." + val.name}
-			val.log = cl
-			logCounter++
-			h.State["logCounter"] = logCounter
 		}
 		configValues = append(configValues, ConfigValue{
 			Class: "custom_log",

caddyconfig/httpcaddyfile/builtins_test.go

@@ -10,7 +10,7 @@ import (
 func TestLogDirectiveSyntax(t *testing.T) {
 	for i, tc := range []struct {
 		input       string
-		expectWarn  bool
+		output      string
 		expectError bool
 	}{
 		{
@@ -18,7 +18,7 @@ func TestLogDirectiveSyntax(t *testing.T) {
 				log
 			}
 			`,
-			expectWarn:  false,
+			output:      `{"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{}}}}}}`,
 			expectError: false,
 		},
 		{
@@ -28,17 +28,35 @@ func TestLogDirectiveSyntax(t *testing.T) {
 				}
 			}
 			`,
-			expectWarn:  false,
+			output:      `{"logging":{"logs":{"default":{"exclude":["http.log.access.log0"]},"log0":{"writer":{"filename":"foo.log","output":"file"},"include":["http.log.access.log0"]}}},"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{"default_logger_name":"log0"}}}}}}`,
 			expectError: false,
 		},
 		{
 			input: `:8080 {
-				log /foo {
+				log {
+					format filter {
+						wrap console
+						fields {
+							common_log delete
+							request>remote_addr ip_mask {
+								ipv4 24
+								ipv6 32
+							}
+						}
+					}
+				}
+			}
+			`,
+			output:      `{"logging":{"logs":{"default":{"exclude":["http.log.access.log0"]},"log0":{"encoder":{"fields":{"common_log":{"filter":"delete"},"request\u003eremote_addr":{"filter":"ip_mask","ipv4_cidr":24,"ipv6_cidr":32}},"format":"filter","wrap":{"format":"console"}},"include":["http.log.access.log0"]}}},"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{"default_logger_name":"log0"}}}}}}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log invalid {
 					output file foo.log
 				}
 			}
 			`,
-			expectWarn:  false,
 			expectError: true,
 		},
 	} {
@@ -47,13 +65,134 @@ func TestLogDirectiveSyntax(t *testing.T) {
 			ServerType: ServerType{},
 		}
 
-		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
+		out, _, err := adapter.Adapt([]byte(tc.input), nil)
 
-		if len(warnings) > 0 != tc.expectWarn {
-			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
 			continue
 		}
 
+		if string(out) != tc.output {
+			t.Errorf("Test %d error output mismatch Expected: %s, got %s", i, tc.output, out)
+		}
+	}
+}
+
+func TestRedirDirectiveSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectError bool
+	}{
+		{
+			input: `:8080 {
+				redir :8081
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir * :8081
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /api/* :8081 300
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir :8081 300
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /api/* :8081 399
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir :8081 399
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html temporary
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir https://example.com{uri} permanent
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html permanent
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html html
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html htlm
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 200
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 400
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 temp
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 perm
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 php
+			}`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
 		if err != nil != tc.expectError {
 			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
 			continue

caddyconfig/httpcaddyfile/directives.go

@@ -44,9 +44,9 @@ var directiveOrder = []string{
 	"request_body",
 
 	"redir",
-	"rewrite",
 
 	// URI manipulation
+	"rewrite",
 	"uri",
 	"try_files",
 
@@ -54,15 +54,17 @@ var directiveOrder = []string{
 	"basicauth",
 	"request_header",
 	"encode",
+	"push",
 	"templates",
 
 	// special routing & dispatching directives
 	"handle",
 	"handle_path",
 	"route",
-	"push",
 
 	// handlers that typically respond to requests
+	"abort",
+	"error",
 	"respond",
 	"metrics",
 	"reverse_proxy",
@@ -263,6 +265,13 @@ func (h Helper) NewBindAddresses(addrs []string) []ConfigValue {
 	return []ConfigValue{{Class: "bind", Value: addrs}}
 }
 
+// WithDispenser returns a new instance based on d. All others Helper
+// fields are copied, so typically maps are shared with this new instance.
+func (h Helper) WithDispenser(d *caddyfile.Dispenser) Helper {
+	h.Dispenser = d
+	return h
+}
+
 // ParseSegmentAsSubroute parses the segment such that its subdirectives
 // are themselves treated as directives, from which a subroute is built
 // and returned.
@@ -320,7 +329,7 @@ func parseSegmentAsConfig(h Helper) ([]ConfigValue, error) {
 			dir := seg.Directive()
 			dirFunc, ok := registeredDirectives[dir]
 			if !ok {
-				return nil, h.Errf("unrecognized directive: %s", dir)
+				return nil, h.Errf("unrecognized directive: %s - are you sure your Caddyfile structure (nesting and braces) is correct?", dir)
 			}
 
 			subHelper := h
@@ -469,6 +478,27 @@ func (sb serverBlock) hostsFromKeys(loggerMode bool) []string {
 	return sblockHosts
 }
 
+func (sb serverBlock) hostsFromKeysNotHTTP(httpPort string) []string {
+	// ensure each entry in our list is unique
+	hostMap := make(map[string]struct{})
+	for _, addr := range sb.keys {
+		if addr.Host == "" {
+			continue
+		}
+		if addr.Scheme != "http" && addr.Port != httpPort {
+			hostMap[addr.Host] = struct{}{}
+		}
+	}
+
+	// convert map to slice
+	sblockHosts := make([]string, 0, len(hostMap))
+	for host := range hostMap {
+		sblockHosts = append(sblockHosts, host)
+	}
+
+	return sblockHosts
+}
+
 // hasHostCatchAllKey returns true if sb has a key that
 // omits a host portion, i.e. it "catches all" hosts.
 func (sb serverBlock) hasHostCatchAllKey() bool {
@@ -498,9 +528,10 @@ type (
 	UnmarshalHandlerFunc func(h Helper) (caddyhttp.MiddlewareHandler, error)
 
 	// UnmarshalGlobalFunc is a function which can unmarshal Caddyfile
-	// tokens into a global option config value using a Helper type.
-	// These are passed in a call to RegisterGlobalOption.
-	UnmarshalGlobalFunc func(d *caddyfile.Dispenser) (interface{}, error)
+	// tokens from a global option. It is passed the tokens to parse and
+	// existing value from the previous instance of this global option
+	// (if any). It returns the value to associate with this global option.
+	UnmarshalGlobalFunc func(d *caddyfile.Dispenser, existingVal interface{}) (interface{}, error)
 )
 
 var registeredDirectives = make(map[string]UnmarshalFunc)

caddyconfig/httpcaddyfile/httptype.go

@@ -17,6 +17,7 @@ package httpcaddyfile
 import (
 	"encoding/json"
 	"fmt"
+	"log"
 	"reflect"
 	"regexp"
 	"sort"
@@ -27,6 +28,7 @@ import (
 	"github.com/caddyserver/caddy/v2/caddyconfig"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
+	"github.com/caddyserver/caddy/v2/modules/caddypki"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
 )
 
@@ -34,6 +36,17 @@ func init() {
 	caddyconfig.RegisterAdapter("caddyfile", caddyfile.Adapter{ServerType: ServerType{}})
 }
 
+// App represents the configuration for a non-standard
+// Caddy app module (e.g. third-party plugin) which was
+// parsed from a global options block.
+type App struct {
+	// The JSON key for the app being configured
+	Name string
+
+	// The raw app config as JSON
+	Value json.RawMessage
+}
+
 // ServerType can set up a config from an HTTP Caddyfile.
 type ServerType struct {
 }
@@ -100,6 +113,7 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 		"{tls_client_serial}", "{http.request.tls.client.serial}",
 		"{tls_client_subject}", "{http.request.tls.client.subject}",
 		"{tls_client_certificate_pem}", "{http.request.tls.client.certificate_pem}",
+		"{upstream_hostport}", "{http.reverse_proxy.upstream.hostport}",
 	)
 
 	// these are placeholders that allow a user-defined final
@@ -156,7 +170,11 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 			dirFunc, ok := registeredDirectives[dir]
 			if !ok {
 				tkn := segment[0]
-				return nil, warnings, fmt.Errorf("%s:%d: unrecognized directive: %s", tkn.File, tkn.Line, dir)
+				message := "%s:%d: unrecognized directive: %s"
+				if !sb.block.HasBraces {
+					message += "\nDid you mean to define a second site? If so, you must use curly braces around each site to separate their configurations."
+				}
+				return nil, warnings, fmt.Errorf(message, tkn.File, tkn.Line, dir)
 			}
 
 			h := Helper{
@@ -207,9 +225,10 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 
 	// now that each server is configured, make the HTTP app
 	httpApp := caddyhttp.App{
-		HTTPPort:  tryInt(options["http_port"], &warnings),
-		HTTPSPort: tryInt(options["https_port"], &warnings),
-		Servers:   servers,
+		HTTPPort:    tryInt(options["http_port"], &warnings),
+		HTTPSPort:   tryInt(options["https_port"], &warnings),
+		GracePeriod: tryDuration(options["grace_period"], &warnings),
+		Servers:     servers,
 	}
 
 	// then make the TLS app
@@ -218,23 +237,38 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 		return nil, warnings, err
 	}
 
+	// then make the PKI app
+	pkiApp, warnings, err := st.buildPKIApp(pairings, options, warnings)
+	if err != nil {
+		return nil, warnings, err
+	}
+
 	// extract any custom logs, and enforce configured levels
 	var customLogs []namedCustomLog
 	var hasDefaultLog bool
+	addCustomLog := func(ncl namedCustomLog) {
+		if ncl.name == "" {
+			return
+		}
+		if ncl.name == "default" {
+			hasDefaultLog = true
+		}
+		if _, ok := options["debug"]; ok && ncl.log.Level == "" {
+			ncl.log.Level = "DEBUG"
+		}
+		customLogs = append(customLogs, ncl)
+	}
+	// Apply global log options, when set
+	if options["log"] != nil {
+		for _, logValue := range options["log"].([]ConfigValue) {
+			addCustomLog(logValue.Value.(namedCustomLog))
+		}
+	}
+	// Apply server-specific log options
 	for _, p := range pairings {
 		for _, sb := range p.serverBlocks {
 			for _, clVal := range sb.pile["custom_log"] {
-				ncl := clVal.Value.(namedCustomLog)
-				if ncl.name == "" {
-					continue
-				}
-				if ncl.name == "default" {
-					hasDefaultLog = true
-				}
-				if _, ok := options["debug"]; ok && ncl.log.Level == "" {
-					ncl.log.Level = "DEBUG"
-				}
-				customLogs = append(customLogs, ncl)
+				addCustomLog(clVal.Value.(namedCustomLog))
 			}
 		}
 	}
@@ -252,12 +286,26 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 
 	// annnd the top-level config, then we're done!
 	cfg := &caddy.Config{AppsRaw: make(caddy.ModuleMap)}
+
+	// loop through the configured options, and if any of
+	// them are an httpcaddyfile App, then we insert them
+	// into the config as raw Caddy apps
+	for _, opt := range options {
+		if app, ok := opt.(App); ok {
+			cfg.AppsRaw[app.Name] = app.Value
+		}
+	}
+
+	// insert the standard Caddy apps into the config
 	if len(httpApp.Servers) > 0 {
 		cfg.AppsRaw["http"] = caddyconfig.JSON(httpApp, &warnings)
 	}
 	if !reflect.DeepEqual(tlsApp, &caddytls.TLS{CertificatesRaw: make(caddy.ModuleMap)}) {
 		cfg.AppsRaw["tls"] = caddyconfig.JSON(tlsApp, &warnings)
 	}
+	if !reflect.DeepEqual(pkiApp, &caddypki.PKI{CAs: make(map[string]*caddypki.CA)}) {
+		cfg.AppsRaw["pki"] = caddyconfig.JSON(pkiApp, &warnings)
+	}
 	if storageCvtr, ok := options["storage"].(caddy.StorageConverter); ok {
 		cfg.StorageRaw = caddyconfig.JSONModuleObject(storageCvtr,
 			"module",
@@ -280,7 +328,7 @@ func (st ServerType) Setup(inputServerBlocks []caddyfile.ServerBlock,
 			// most users seem to prefer not writing access logs
 			// to the default log when they are directed to a
 			// file or have any other special customization
-			if len(ncl.log.Include) > 0 {
+			if ncl.name != "default" && len(ncl.log.Include) > 0 {
 				defaultLog, ok := cfg.Logging.Logs["default"]
 				if !ok {
 					defaultLog = new(caddy.CustomLog)
@@ -315,7 +363,7 @@ func (ServerType) evaluateGlobalOptionsBlock(serverBlocks []serverBlock, options
 			return nil, fmt.Errorf("%s:%d: unrecognized global option: %s", tkn.File, tkn.Line, opt)
 		}
 
-		val, err = optFunc(disp)
+		val, err = optFunc(disp, options[opt])
 		if err != nil {
 			return nil, fmt.Errorf("parsing caddyfile tokens for '%s': %v", opt, err)
 		}
@@ -329,11 +377,25 @@ func (ServerType) evaluateGlobalOptionsBlock(serverBlocks []serverBlock, options
 			}
 			serverOpts, ok := val.(serverOptions)
 			if !ok {
-				return nil, fmt.Errorf("unexpected type from 'servers' global options")
+				return nil, fmt.Errorf("unexpected type from 'servers' global options: %T", val)
 			}
 			options[opt] = append(existingOpts, serverOpts)
 			continue
 		}
+		// Additionally, fold multiple "log" options together into an
+		// array so that multiple loggers can be configured.
+		if opt == "log" {
+			existingOpts, ok := options[opt].([]ConfigValue)
+			if !ok {
+				existingOpts = []ConfigValue{}
+			}
+			logOpts, ok := val.([]ConfigValue)
+			if !ok {
+				return nil, fmt.Errorf("unexpected type from 'log' global options: %T", val)
+			}
+			options[opt] = append(existingOpts, logOpts...)
+			continue
+		}
 
 		options[opt] = val
 	}
@@ -395,6 +457,9 @@ func (st *ServerType) serversFromPairings(
 			if autoHTTPS == "disable_redirects" {
 				srv.AutoHTTPS.DisableRedir = true
 			}
+			if autoHTTPS == "ignore_loaded_certs" {
+				srv.AutoHTTPS.IgnoreLoadedCerts = true
+			}
 		}
 
 		// sort server blocks by their keys; this is important because
@@ -409,7 +474,7 @@ func (st *ServerType) serversFromPairings(
 			var iLongestHost, jLongestHost string
 			var iWildcardHost, jWildcardHost bool
 			for _, addr := range p.serverBlocks[i].keys {
-				if strings.Contains(addr.Host, "*.") {
+				if strings.Contains(addr.Host, "*") || addr.Host == "" {
 					iWildcardHost = true
 				}
 				if specificity(addr.Host) > specificity(iLongestHost) {
@@ -420,7 +485,7 @@ func (st *ServerType) serversFromPairings(
 				}
 			}
 			for _, addr := range p.serverBlocks[j].keys {
-				if strings.Contains(addr.Host, "*.") {
+				if strings.Contains(addr.Host, "*") || addr.Host == "" {
 					jWildcardHost = true
 				}
 				if specificity(addr.Host) > specificity(jLongestHost) {
@@ -430,9 +495,12 @@ func (st *ServerType) serversFromPairings(
 					jLongestPath = addr.Path
 				}
 			}
+			// catch-all blocks (blocks with no hostname) should always go
+			// last, even after blocks with wildcard hosts
+			if specificity(iLongestHost) == 0 {
+				return false
+			}
 			if specificity(jLongestHost) == 0 {
-				// catch-all blocks (blocks with no hostname) should always go
-				// last, even after blocks with wildcard hosts
 				return true
 			}
 			if iWildcardHost != jWildcardHost {
@@ -459,6 +527,16 @@ func (st *ServerType) serversFromPairings(
 			}
 		}
 
+		// if needed, the ServerLogConfig is initialized beforehand so
+		// that all server blocks can populate it with data, even when not
+		// coming with a log directive
+		for _, sblock := range p.serverBlocks {
+			if len(sblock.pile["custom_log"]) != 0 {
+				srv.Logs = new(caddyhttp.ServerLogConfig)
+				break
+			}
+		}
+
 		// create a subroute for each site in the server block
 		for _, sblock := range p.serverBlocks {
 			matcherSetsEnc, err := st.compileEncodedMatcherSets(sblock)
@@ -468,6 +546,13 @@ func (st *ServerType) serversFromPairings(
 
 			hosts := sblock.hostsFromKeys(false)
 
+			// emit warnings if user put unspecified IP addresses; they probably want the bind directive
+			for _, h := range hosts {
+				if h == "0.0.0.0" || h == "::" {
+					log.Printf("[WARNING] Site block has unspecified IP address %s which only matches requests having that Host header; you probably want the 'bind' directive to configure the socket", h)
+				}
+			}
+
 			// tls: connection policies
 			if cpVals, ok := sblock.pile["tls.connection_policy"]; ok {
 				// tls connection policies
@@ -500,16 +585,20 @@ func (st *ServerType) serversFromPairings(
 			}
 
 			for _, addr := range sblock.keys {
-				// exclude any hosts that were defined explicitly with "http://"
-				// in the key from automated cert management (issue #2998)
-				if addr.Scheme == "http" && addr.Host != "" {
-					if srv.AutoHTTPS == nil {
-						srv.AutoHTTPS = new(caddyhttp.AutoHTTPSConfig)
-					}
-					if !sliceContains(srv.AutoHTTPS.Skip, addr.Host) {
-						srv.AutoHTTPS.Skip = append(srv.AutoHTTPS.Skip, addr.Host)
+				// if server only uses HTTPS port, auto-HTTPS will not apply
+				if listenersUseAnyPortOtherThan(srv.Listen, httpPort) {
+					// exclude any hosts that were defined explicitly with "http://"
+					// in the key from automated cert management (issue #2998)
+					if addr.Scheme == "http" && addr.Host != "" {
+						if srv.AutoHTTPS == nil {
+							srv.AutoHTTPS = new(caddyhttp.AutoHTTPSConfig)
+						}
+						if !sliceContains(srv.AutoHTTPS.Skip, addr.Host) {
+							srv.AutoHTTPS.Skip = append(srv.AutoHTTPS.Skip, addr.Host)
+						}
 					}
 				}
+
 				// we'll need to remember if the address qualifies for auto-HTTPS, so we
 				// can add a TLS conn policy if necessary
 				if addr.Scheme == "https" ||
@@ -562,9 +651,6 @@ func (st *ServerType) serversFromPairings(
 			sblockLogHosts := sblock.hostsFromKeys(true)
 			for _, cval := range sblock.pile["custom_log"] {
 				ncl := cval.Value.(namedCustomLog)
-				if srv.Logs == nil {
-					srv.Logs = new(caddyhttp.ServerLogConfig)
-				}
 				if sblock.hasHostCatchAllKey() {
 					// all requests for hosts not able to be listed should use
 					// this log because it's a catch-all-hosts server block
@@ -1173,6 +1259,14 @@ func tryString(val interface{}, warnings *[]caddyconfig.Warning) string {
 	return stringVal
 }
 
+func tryDuration(val interface{}, warnings *[]caddyconfig.Warning) caddy.Duration {
+	durationVal, ok := val.(caddy.Duration)
+	if val != nil && !ok && warnings != nil {
+		*warnings = append(*warnings, caddyconfig.Warning{Message: "not a duration type"})
+	}
+	return durationVal
+}
+
 // sliceContains returns true if needle is in haystack.
 func sliceContains(haystack []string, needle string) bool {
 	for _, s := range haystack {
@@ -1183,6 +1277,26 @@ func sliceContains(haystack []string, needle string) bool {
 	return false
 }
 
+// listenersUseAnyPortOtherThan returns true if there are any
+// listeners in addresses that use a port which is not otherPort.
+// Mostly borrowed from unexported method in caddyhttp package.
+func listenersUseAnyPortOtherThan(addresses []string, otherPort string) bool {
+	otherPortInt, err := strconv.Atoi(otherPort)
+	if err != nil {
+		return false
+	}
+	for _, lnAddr := range addresses {
+		laddrs, err := caddy.ParseNetworkAddress(lnAddr)
+		if err != nil {
+			continue
+		}
+		if uint(otherPortInt) > laddrs.EndPort || uint(otherPortInt) < laddrs.StartPort {
+			return true
+		}
+	}
+	return false
+}
+
 // specificity returns len(s) minus any wildcards (*) and
 // placeholders ({...}). Basically, it's a length count
 // that penalizes the use of wildcards and placeholders.

caddyconfig/httpcaddyfile/httptype_test.go

@@ -9,7 +9,6 @@ import (
 func TestMatcherSyntax(t *testing.T) {
 	for i, tc := range []struct {
 		input       string
-		expectWarn  bool
 		expectError bool
 	}{
 		{
@@ -18,7 +17,6 @@ func TestMatcherSyntax(t *testing.T) {
 				query showdebug=1
 			}
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -27,7 +25,6 @@ func TestMatcherSyntax(t *testing.T) {
 				query bad format
 			}
 			`,
-			expectWarn:  false,
 			expectError: true,
 		},
 		{
@@ -38,7 +35,6 @@ func TestMatcherSyntax(t *testing.T) {
 				}
 			}
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -47,14 +43,12 @@ func TestMatcherSyntax(t *testing.T) {
 				not path /somepath*
 			}
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
 			input: `http://localhost
 			@debug not path /somepath*
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -63,7 +57,6 @@ func TestMatcherSyntax(t *testing.T) {
 			}
 			http://localhost
 			`,
-			expectWarn:  false,
 			expectError: true,
 		},
 	} {
@@ -72,12 +65,7 @@ func TestMatcherSyntax(t *testing.T) {
 			ServerType: ServerType{},
 		}
 
-		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
-
-		if len(warnings) > 0 != tc.expectWarn {
-			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
-			continue
-		}
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
 
 		if err != nil != tc.expectError {
 			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
@@ -119,7 +107,6 @@ func TestSpecificity(t *testing.T) {
 func TestGlobalOptions(t *testing.T) {
 	for i, tc := range []struct {
 		input       string
-		expectWarn  bool
 		expectError bool
 	}{
 		{
@@ -129,7 +116,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -139,7 +125,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -149,7 +134,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -161,7 +145,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: true,
 		},
 		{
@@ -174,7 +157,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -187,7 +169,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: false,
 		},
 		{
@@ -200,7 +181,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: true,
 		},
 		{
@@ -213,7 +193,6 @@ func TestGlobalOptions(t *testing.T) {
 				}
 				:80
 			`,
-			expectWarn:  false,
 			expectError: true,
 		},
 	} {
@@ -222,12 +201,7 @@ func TestGlobalOptions(t *testing.T) {
 			ServerType: ServerType{},
 		}
 
-		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
-
-		if len(warnings) > 0 != tc.expectWarn {
-			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
-			continue
-		}
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
 
 		if err != nil != tc.expectError {
 			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)

caddyconfig/httpcaddyfile/options.go

@@ -18,6 +18,7 @@ import (
 	"strconv"
 
 	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
 	"github.com/caddyserver/certmagic"
@@ -28,15 +29,17 @@ func init() {
 	RegisterGlobalOption("debug", parseOptTrue)
 	RegisterGlobalOption("http_port", parseOptHTTPPort)
 	RegisterGlobalOption("https_port", parseOptHTTPSPort)
+	RegisterGlobalOption("grace_period", parseOptDuration)
 	RegisterGlobalOption("default_sni", parseOptSingleString)
 	RegisterGlobalOption("order", parseOptOrder)
-	RegisterGlobalOption("experimental_http3", parseOptTrue)
 	RegisterGlobalOption("storage", parseOptStorage)
+	RegisterGlobalOption("storage_clean_interval", parseOptDuration)
 	RegisterGlobalOption("acme_ca", parseOptSingleString)
 	RegisterGlobalOption("acme_ca_root", parseOptSingleString)
-	RegisterGlobalOption("acme_dns", parseOptSingleString)
+	RegisterGlobalOption("acme_dns", parseOptACMEDNS)
 	RegisterGlobalOption("acme_eab", parseOptACMEEAB)
 	RegisterGlobalOption("cert_issuer", parseOptCertIssuer)
+	RegisterGlobalOption("skip_install_trust", parseOptTrue)
 	RegisterGlobalOption("email", parseOptSingleString)
 	RegisterGlobalOption("admin", parseOptAdmin)
 	RegisterGlobalOption("on_demand_tls", parseOptOnDemand)
@@ -44,13 +47,14 @@ func init() {
 	RegisterGlobalOption("key_type", parseOptSingleString)
 	RegisterGlobalOption("auto_https", parseOptAutoHTTPS)
 	RegisterGlobalOption("servers", parseServerOptions)
+	RegisterGlobalOption("ocsp_stapling", parseOCSPStaplingOptions)
+	RegisterGlobalOption("log", parseLogOptions)
+	RegisterGlobalOption("preferred_chains", parseOptPreferredChains)
 }
 
-func parseOptTrue(d *caddyfile.Dispenser) (interface{}, error) {
-	return true, nil
-}
+func parseOptTrue(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) { return true, nil }
 
-func parseOptHTTPPort(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptHTTPPort(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	var httpPort int
 	for d.Next() {
 		var httpPortStr string
@@ -66,7 +70,7 @@ func parseOptHTTPPort(d *caddyfile.Dispenser) (interface{}, error) {
 	return httpPort, nil
 }
 
-func parseOptHTTPSPort(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptHTTPSPort(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	var httpsPort int
 	for d.Next() {
 		var httpsPortStr string
@@ -82,7 +86,7 @@ func parseOptHTTPSPort(d *caddyfile.Dispenser) (interface{}, error) {
 	return httpsPort, nil
 }
 
-func parseOptOrder(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptOrder(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	newOrder := directiveOrder
 
 	for d.Next() {
@@ -158,34 +162,59 @@ func parseOptOrder(d *caddyfile.Dispenser) (interface{}, error) {
 	return newOrder, nil
 }
 
-func parseOptStorage(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptStorage(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	if !d.Next() { // consume option name
 		return nil, d.ArgErr()
 	}
 	if !d.Next() { // get storage module name
 		return nil, d.ArgErr()
 	}
-	modName := d.Val()
-	mod, err := caddy.GetModule("caddy.storage." + modName)
-	if err != nil {
-		return nil, d.Errf("getting storage module '%s': %v", modName, err)
-	}
-	unm, ok := mod.New().(caddyfile.Unmarshaler)
-	if !ok {
-		return nil, d.Errf("storage module '%s' is not a Caddyfile unmarshaler", mod.ID)
-	}
-	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
+	modID := "caddy.storage." + d.Val()
+	unm, err := caddyfile.UnmarshalModule(d, modID)
 	if err != nil {
 		return nil, err
 	}
 	storage, ok := unm.(caddy.StorageConverter)
 	if !ok {
-		return nil, d.Errf("module %s is not a StorageConverter", mod.ID)
+		return nil, d.Errf("module %s is not a caddy.StorageConverter", modID)
 	}
 	return storage, nil
 }
 
-func parseOptACMEEAB(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptDuration(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	if !d.Next() { // consume option name
+		return nil, d.ArgErr()
+	}
+	if !d.Next() { // get duration value
+		return nil, d.ArgErr()
+	}
+	dur, err := caddy.ParseDuration(d.Val())
+	if err != nil {
+		return nil, err
+	}
+	return caddy.Duration(dur), nil
+}
+
+func parseOptACMEDNS(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	if !d.Next() { // consume option name
+		return nil, d.ArgErr()
+	}
+	if !d.Next() { // get DNS module name
+		return nil, d.ArgErr()
+	}
+	modID := "dns.providers." + d.Val()
+	unm, err := caddyfile.UnmarshalModule(d, modID)
+	if err != nil {
+		return nil, err
+	}
+	prov, ok := unm.(certmagic.ACMEDNSProvider)
+	if !ok {
+		return nil, d.Errf("module %s (%T) is not a certmagic.ACMEDNSProvider", modID, unm)
+	}
+	return prov, nil
+}
+
+func parseOptACMEEAB(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	eab := new(acme.EAB)
 	for d.Next() {
 		if d.NextArg() {
@@ -213,34 +242,30 @@ func parseOptACMEEAB(d *caddyfile.Dispenser) (interface{}, error) {
 	return eab, nil
 }
 
-func parseOptCertIssuer(d *caddyfile.Dispenser) (interface{}, error) {
-	if !d.Next() { // consume option name
-		return nil, d.ArgErr()
+func parseOptCertIssuer(d *caddyfile.Dispenser, existing interface{}) (interface{}, error) {
+	var issuers []certmagic.Issuer
+	if existing != nil {
+		issuers = existing.([]certmagic.Issuer)
 	}
-	if !d.Next() { // get issuer module name
-		return nil, d.ArgErr()
+	for d.Next() { // consume option name
+		if !d.Next() { // get issuer module name
+			return nil, d.ArgErr()
+		}
+		modID := "tls.issuance." + d.Val()
+		unm, err := caddyfile.UnmarshalModule(d, modID)
+		if err != nil {
+			return nil, err
+		}
+		iss, ok := unm.(certmagic.Issuer)
+		if !ok {
+			return nil, d.Errf("module %s (%T) is not a certmagic.Issuer", modID, unm)
+		}
+		issuers = append(issuers, iss)
 	}
-	modName := d.Val()
-	mod, err := caddy.GetModule("tls.issuance." + modName)
-	if err != nil {
-		return nil, d.Errf("getting issuer module '%s': %v", modName, err)
-	}
-	unm, ok := mod.New().(caddyfile.Unmarshaler)
-	if !ok {
-		return nil, d.Errf("issuer module '%s' is not a Caddyfile unmarshaler", mod.ID)
-	}
-	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
-	if err != nil {
-		return nil, err
-	}
-	iss, ok := unm.(certmagic.Issuer)
-	if !ok {
-		return nil, d.Errf("module %s is not a certmagic.Issuer", mod.ID)
-	}
-	return iss, nil
+	return issuers, nil
 }
 
-func parseOptSingleString(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptSingleString(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	d.Next() // consume parameter name
 	if !d.Next() {
 		return "", d.ArgErr()
@@ -252,7 +277,7 @@ func parseOptSingleString(d *caddyfile.Dispenser) (interface{}, error) {
 	return val, nil
 }
 
-func parseOptAdmin(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptAdmin(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	adminCfg := new(caddy.AdminConfig)
 	for d.Next() {
 		if d.NextArg() {
@@ -288,7 +313,7 @@ func parseOptAdmin(d *caddyfile.Dispenser) (interface{}, error) {
 	return adminCfg, nil
 }
 
-func parseOptOnDemand(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptOnDemand(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	var ond *caddytls.OnDemandConfig
 	for d.Next() {
 		if d.NextArg() {
@@ -348,7 +373,7 @@ func parseOptOnDemand(d *caddyfile.Dispenser) (interface{}, error) {
 	return ond, nil
 }
 
-func parseOptAutoHTTPS(d *caddyfile.Dispenser) (interface{}, error) {
+func parseOptAutoHTTPS(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	d.Next() // consume parameter name
 	if !d.Next() {
 		return "", d.ArgErr()
@@ -357,12 +382,79 @@ func parseOptAutoHTTPS(d *caddyfile.Dispenser) (interface{}, error) {
 	if d.Next() {
 		return "", d.ArgErr()
 	}
-	if val != "off" && val != "disable_redirects" {
-		return "", d.Errf("auto_https must be either 'off' or 'disable_redirects'")
+	if val != "off" && val != "disable_redirects" && val != "ignore_loaded_certs" {
+		return "", d.Errf("auto_https must be one of 'off', 'disable_redirects' or 'ignore_loaded_certs'")
 	}
 	return val, nil
 }
 
-func parseServerOptions(d *caddyfile.Dispenser) (interface{}, error) {
+func parseServerOptions(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
 	return unmarshalCaddyfileServerOptions(d)
 }
+
+func parseOCSPStaplingOptions(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	d.Next() // consume option name
+	var val string
+	if !d.AllArgs(&val) {
+		return nil, d.ArgErr()
+	}
+	if val != "off" {
+		return nil, d.Errf("invalid argument '%s'", val)
+	}
+	return certmagic.OCSPConfig{
+		DisableStapling: val == "off",
+	}, nil
+}
+
+// parseLogOptions parses the global log option. Syntax:
+//
+//     log [name] {
+//         output  <writer_module> ...
+//         format  <encoder_module> ...
+//         level   <level>
+//         include <namespaces...>
+//         exclude <namespaces...>
+//     }
+//
+// When the name argument is unspecified, this directive modifies the default
+// logger.
+//
+func parseLogOptions(d *caddyfile.Dispenser, existingVal interface{}) (interface{}, error) {
+	currentNames := make(map[string]struct{})
+	if existingVal != nil {
+		innerVals, ok := existingVal.([]ConfigValue)
+		if !ok {
+			return nil, d.Errf("existing log values of unexpected type: %T", existingVal)
+		}
+		for _, rawVal := range innerVals {
+			val, ok := rawVal.Value.(namedCustomLog)
+			if !ok {
+				return nil, d.Errf("existing log value of unexpected type: %T", existingVal)
+			}
+			currentNames[val.name] = struct{}{}
+		}
+	}
+
+	var warnings []caddyconfig.Warning
+	// Call out the same parser that handles server-specific log configuration.
+	configValues, err := parseLogHelper(
+		Helper{
+			Dispenser: d,
+			warnings:  &warnings,
+		},
+		currentNames,
+	)
+	if err != nil {
+		return nil, err
+	}
+	if len(warnings) > 0 {
+		return nil, d.Errf("warnings found in parsing global log options: %+v", warnings)
+	}
+
+	return configValues, nil
+}
+
+func parseOptPreferredChains(d *caddyfile.Dispenser, _ interface{}) (interface{}, error) {
+	d.Next()
+	return caddytls.ParseCaddyfilePreferredChainsOptions(d)
+}

caddyconfig/httpcaddyfile/options_test.go

@@ -0,0 +1,64 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	_ "github.com/caddyserver/caddy/v2/modules/logging"
+)
+
+func TestGlobalLogOptionSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		output      string
+		expectError bool
+	}{
+		// NOTE: Additional test cases of successful Caddyfile parsing
+		// are present in: caddytest/integration/caddyfile_adapt/
+		{
+			input: `{
+				log default
+			}
+			`,
+			output:      `{}`,
+			expectError: false,
+		},
+		{
+			input: `{
+				log example {
+					output file foo.log
+				}
+				log example {
+					format json
+				}
+			}
+			`,
+			expectError: true,
+		},
+		{
+			input: `{
+				log example /foo {
+					output file foo.log
+				}
+			}
+			`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		out, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %v", i, tc.expectError, err)
+			continue
+		}
+
+		if string(out) != tc.output {
+			t.Errorf("Test %d error output mismatch Expected: %s, got %s", i, tc.output, out)
+		}
+	}
+}

caddyconfig/httpcaddyfile/pkiapp.go

@@ -0,0 +1,61 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package httpcaddyfile
+
+import (
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/modules/caddypki"
+)
+
+func (st ServerType) buildPKIApp(
+	pairings []sbAddrAssociation,
+	options map[string]interface{},
+	warnings []caddyconfig.Warning,
+) (*caddypki.PKI, []caddyconfig.Warning, error) {
+
+	pkiApp := &caddypki.PKI{CAs: make(map[string]*caddypki.CA)}
+
+	skipInstallTrust := false
+	if _, ok := options["skip_install_trust"]; ok {
+		skipInstallTrust = true
+	}
+	falseBool := false
+
+	for _, p := range pairings {
+		for _, sblock := range p.serverBlocks {
+			// find all the CAs that were defined and add them to the app config
+			// i.e. from any "acme_server" directives
+			for _, caCfgValue := range sblock.pile["pki.ca"] {
+				ca := caCfgValue.Value.(*caddypki.CA)
+				if skipInstallTrust {
+					ca.InstallTrust = &falseBool
+				}
+				pkiApp.CAs[ca.ID] = ca
+			}
+		}
+	}
+
+	// if there was no CAs defined in any of the servers,
+	// and we were requested to not install trust, then
+	// add one for the default/local CA to do so
+	if len(pkiApp.CAs) == 0 && skipInstallTrust {
+		ca := new(caddypki.CA)
+		ca.ID = caddypki.DefaultCAID
+		ca.InstallTrust = &falseBool
+		pkiApp.CAs[ca.ID] = ca
+	}
+
+	return pkiApp, warnings, nil
+}

caddyconfig/httpcaddyfile/serveroptions.go

@@ -57,21 +57,14 @@ func unmarshalCaddyfileServerOptions(d *caddyfile.Dispenser) (interface{}, error
 			switch d.Val() {
 			case "listener_wrappers":
 				for nesting := d.Nesting(); d.NextBlock(nesting); {
-					mod, err := caddy.GetModule("caddy.listeners." + d.Val())
-					if err != nil {
-						return nil, fmt.Errorf("finding listener module '%s': %v", d.Val(), err)
-					}
-					unm, ok := mod.New().(caddyfile.Unmarshaler)
-					if !ok {
-						return nil, fmt.Errorf("listener module '%s' is not a Caddyfile unmarshaler", mod)
-					}
-					err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
+					modID := "caddy.listeners." + d.Val()
+					unm, err := caddyfile.UnmarshalModule(d, modID)
 					if err != nil {
 						return nil, err
 					}
 					listenerWrapper, ok := unm.(caddy.ListenerWrapper)
 					if !ok {
-						return nil, fmt.Errorf("module %s is not a listener wrapper", mod)
+						return nil, fmt.Errorf("module %s (%T) is not a listener wrapper", modID, unm)
 					}
 					jsonListenerWrapper := caddyconfig.JSONModuleObject(
 						listenerWrapper,

caddyconfig/httpcaddyfile/tlsapp.go

@@ -40,6 +40,10 @@ func (st ServerType) buildTLSApp(
 	tlsApp := &caddytls.TLS{CertificatesRaw: make(caddy.ModuleMap)}
 	var certLoaders []caddytls.CertificateLoader
 
+	httpPort := strconv.Itoa(caddyhttp.DefaultHTTPPort)
+	if hp, ok := options["http_port"].(int); ok {
+		httpPort = strconv.Itoa(hp)
+	}
 	httpsPort := strconv.Itoa(caddyhttp.DefaultHTTPSPort)
 	if hsp, ok := options["https_port"].(int); ok {
 		httpsPort = strconv.Itoa(hsp)
@@ -50,7 +54,7 @@ func (st ServerType) buildTLSApp(
 	// a hostless key, so that they don't get forgotten/omitted
 	// by auto-HTTPS (since they won't appear in route matchers)
 	var serverBlocksWithTLSHostlessKey int
-	hostsSharedWithHostlessKey := make(map[string]struct{})
+	httpsHostsSharedWithHostlessKey := make(map[string]struct{})
 	for _, pair := range pairings {
 		for _, sb := range pair.serverBlocks {
 			for _, addr := range sb.keys {
@@ -66,8 +70,8 @@ func (st ServerType) buildTLSApp(
 						if otherAddr.Original == addr.Original {
 							continue
 						}
-						if otherAddr.Host != "" {
-							hostsSharedWithHostlessKey[otherAddr.Host] = struct{}{}
+						if otherAddr.Host != "" && otherAddr.Scheme != "http" && otherAddr.Port != httpPort {
+							httpsHostsSharedWithHostlessKey[otherAddr.Host] = struct{}{}
 						}
 					}
 					break
@@ -91,6 +95,11 @@ func (st ServerType) buildTLSApp(
 	}
 
 	for _, p := range pairings {
+		// avoid setting up TLS automation policies for a server that is HTTP-only
+		if !listenersUseAnyPortOtherThan(p.addresses, httpPort) {
+			continue
+		}
+
 		for _, sblock := range p.serverBlocks {
 			// get values that populate an automation policy for this block
 			ap, err := newBaseAutomationPolicy(options, warnings, true)
@@ -108,15 +117,20 @@ func (st ServerType) buildTLSApp(
 				ap.OnDemand = true
 			}
 
+			if keyTypeVals, ok := sblock.pile["tls.key_type"]; ok {
+				ap.KeyType = keyTypeVals[0].Value.(string)
+			}
+
 			// certificate issuers
 			if issuerVals, ok := sblock.pile["tls.cert_issuer"]; ok {
 				var issuers []certmagic.Issuer
 				for _, issuerVal := range issuerVals {
-					ap.Issuers = append(ap.Issuers, issuerVal.Value.(certmagic.Issuer))
+					issuers = append(issuers, issuerVal.Value.(certmagic.Issuer))
 				}
 				if ap == catchAllAP && !reflect.DeepEqual(ap.Issuers, issuers) {
 					return nil, warnings, fmt.Errorf("automation policy from site block is also default/catch-all policy because of key without hostname, and the two are in conflict: %#v != %#v", ap.Issuers, issuers)
 				}
+				ap.Issuers = issuers
 			}
 
 			// custom bind host
@@ -175,7 +189,7 @@ func (st ServerType) buildTLSApp(
 			}
 
 			// associate our new automation policy with this server block's hosts
-			ap.Subjects = sblockHosts
+			ap.Subjects = sblock.hostsFromKeysNotHTTP(httpPort)
 			sort.Strings(ap.Subjects) // solely for deterministic test results
 
 			// if a combination of public and internal names were given
@@ -197,7 +211,7 @@ func (st ServerType) buildTLSApp(
 					// it that we would need to check here) since the hostname is known at handshake;
 					// and it is unexpected to switch to internal issuer when the user wants to get
 					// regular certificates on-demand for a class of certs like *.*.tld.
-					if !certmagic.SubjectIsIP(s) && !certmagic.SubjectIsInternal(s) && (strings.Count(s, "*.") < 2 || ap.OnDemand) {
+					if subjectQualifiesForPublicCert(ap, s) {
 						external = append(external, s)
 					} else {
 						internal = append(internal, s)
@@ -264,6 +278,14 @@ func (st ServerType) buildTLSApp(
 		tlsApp.Automation.OnDemand = onDemand
 	}
 
+	// set the storage clean interval if configured
+	if storageCleanInterval, ok := options["storage_clean_interval"].(caddy.Duration); ok {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.StorageCleanInterval = storageCleanInterval
+	}
+
 	// if any hostnames appear on the same server block as a key with
 	// no host, they will not be used with route matchers because the
 	// hostless key matches all hosts, therefore, it wouldn't be
@@ -275,7 +297,7 @@ func (st ServerType) buildTLSApp(
 	internalAP := &caddytls.AutomationPolicy{
 		IssuersRaw: []json.RawMessage{json.RawMessage(`{"module":"internal"}`)},
 	}
-	for h := range hostsSharedWithHostlessKey {
+	for h := range httpsHostsSharedWithHostlessKey {
 		al = append(al, h)
 		if !certmagic.SubjectQualifiesForPublicCert(h) {
 			internalAP.Subjects = append(internalAP.Subjects, h)
@@ -299,17 +321,24 @@ func (st ServerType) buildTLSApp(
 		globalACMECARoot := options["acme_ca_root"]
 		globalACMEDNS := options["acme_dns"]
 		globalACMEEAB := options["acme_eab"]
-		hasGlobalACMEDefaults := globalEmail != nil || globalACMECA != nil || globalACMECARoot != nil || globalACMEDNS != nil || globalACMEEAB != nil
+		globalPreferredChains := options["preferred_chains"]
+		hasGlobalACMEDefaults := globalEmail != nil || globalACMECA != nil || globalACMECARoot != nil || globalACMEDNS != nil || globalACMEEAB != nil || globalPreferredChains != nil
 		if hasGlobalACMEDefaults {
-			for _, ap := range tlsApp.Automation.Policies {
-				if len(ap.Issuers) == 0 {
-					acme, zerosslACME := new(caddytls.ACMEIssuer), new(caddytls.ACMEIssuer)
-					zerossl := &caddytls.ZeroSSLIssuer{ACMEIssuer: zerosslACME}
-					ap.Issuers = []certmagic.Issuer{acme, zerossl} // TODO: keep this in sync with Caddy's other issuer defaults elsewhere, like in caddytls/automation.go (DefaultIssuers).
+			// for _, ap := range tlsApp.Automation.Policies {
+			for i := 0; i < len(tlsApp.Automation.Policies); i++ {
+				ap := tlsApp.Automation.Policies[i]
+				if len(ap.Issuers) == 0 && automationPolicyHasAllPublicNames(ap) {
+					// for public names, create default issuers which will later be filled in with configured global defaults
+					// (internal names will implicitly use the internal issuer at auto-https time)
+					ap.Issuers = caddytls.DefaultIssuers()
 
-					// if a non-ZeroSSL endpoint is specified, we assume we can't use the ZeroSSL issuer successfully
-					if globalACMECA != nil && !strings.Contains(globalACMECA.(string), "zerossl") {
-						ap.Issuers = []certmagic.Issuer{acme}
+					// if a specific endpoint is configured, can't use multiple default issuers
+					if globalACMECA != nil {
+						if strings.Contains(globalACMECA.(string), "zerossl") {
+							ap.Issuers = []certmagic.Issuer{&caddytls.ZeroSSLIssuer{ACMEIssuer: new(caddytls.ACMEIssuer)}}
+						} else {
+							ap.Issuers = []certmagic.Issuer{new(caddytls.ACMEIssuer)}
+						}
 					}
 				}
 			}
@@ -381,6 +410,7 @@ func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]interf
 	globalACMECARoot := options["acme_ca_root"]
 	globalACMEDNS := options["acme_dns"]
 	globalACMEEAB := options["acme_eab"]
+	globalPreferredChains := options["preferred_chains"]
 
 	if globalEmail != nil && acmeIssuer.Email == "" {
 		acmeIssuer.Email = globalEmail.(string)
@@ -392,20 +422,18 @@ func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]interf
 		acmeIssuer.TrustedRootsPEMFiles = append(acmeIssuer.TrustedRootsPEMFiles, globalACMECARoot.(string))
 	}
 	if globalACMEDNS != nil && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.DNS == nil) {
-		provName := globalACMEDNS.(string)
-		dnsProvModule, err := caddy.GetModule("dns.providers." + provName)
-		if err != nil {
-			return fmt.Errorf("getting DNS provider module named '%s': %v", provName, err)
-		}
 		acmeIssuer.Challenges = &caddytls.ChallengesConfig{
 			DNS: &caddytls.DNSChallengeConfig{
-				ProviderRaw: caddyconfig.JSONModuleObject(dnsProvModule.New(), "name", provName, nil),
+				ProviderRaw: caddyconfig.JSONModuleObject(globalACMEDNS, "name", globalACMEDNS.(caddy.Module).CaddyModule().ID.Name(), nil),
 			},
 		}
 	}
 	if globalACMEEAB != nil && acmeIssuer.ExternalAccount == nil {
 		acmeIssuer.ExternalAccount = globalACMEEAB.(*acme.EAB)
 	}
+	if globalPreferredChains != nil && acmeIssuer.PreferredChains == nil {
+		acmeIssuer.PreferredChains = globalPreferredChains.(*caddytls.ChainPreference)
+	}
 	return nil
 }
 
@@ -415,11 +443,12 @@ func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]interf
 // returned if there are no default/global options. However, if always is
 // true, a non-nil value will always be returned (unless there is an error).
 func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddyconfig.Warning, always bool) (*caddytls.AutomationPolicy, error) {
-	issuer, hasIssuer := options["cert_issuer"]
+	issuers, hasIssuers := options["cert_issuer"]
 	_, hasLocalCerts := options["local_certs"]
 	keyType, hasKeyType := options["key_type"]
+	ocspStapling, hasOCSPStapling := options["ocsp_stapling"]
 
-	hasGlobalAutomationOpts := hasIssuer || hasLocalCerts || hasKeyType
+	hasGlobalAutomationOpts := hasIssuers || hasLocalCerts || hasKeyType || hasOCSPStapling
 
 	// if there are no global options related to automation policies
 	// set, then we can just return right away
@@ -435,30 +464,23 @@ func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddycon
 		ap.KeyType = keyType.(string)
 	}
 
-	if hasIssuer && hasLocalCerts {
+	if hasIssuers && hasLocalCerts {
 		return nil, fmt.Errorf("global options are ambiguous: local_certs is confusing when combined with cert_issuer, because local_certs is also a specific kind of issuer")
 	}
 
-	if hasIssuer {
-		ap.Issuers = []certmagic.Issuer{issuer.(certmagic.Issuer)}
+	if hasIssuers {
+		ap.Issuers = issuers.([]certmagic.Issuer)
 	} else if hasLocalCerts {
 		ap.Issuers = []certmagic.Issuer{new(caddytls.InternalIssuer)}
 	}
 
-	return ap, nil
-}
-
-// disambiguateACMEIssuer returns an issuer based on the properties of acmeIssuer.
-// If acmeIssuer implicitly configures a certain kind of ACMEIssuer (for example,
-// ZeroSSL), the proper wrapper over acmeIssuer will be returned instead.
-func disambiguateACMEIssuer(acmeIssuer *caddytls.ACMEIssuer) certmagic.Issuer {
-	// as a special case, we integrate with ZeroSSL's ACME endpoint if it looks like an
-	// implicit ZeroSSL configuration (this requires a  wrapper type over ACMEIssuer
-	// because of the EAB generation; if EAB is provided, we can use plain ACMEIssuer)
-	if strings.Contains(acmeIssuer.CA, "acme.zerossl.com") && acmeIssuer.ExternalAccount == nil {
-		return &caddytls.ZeroSSLIssuer{ACMEIssuer: acmeIssuer}
+	if hasOCSPStapling {
+		ocspConfig := ocspStapling.(certmagic.OCSPConfig)
+		ap.DisableOCSPStapling = ocspConfig.DisableStapling
+		ap.OCSPOverrides = ocspConfig.ResponderOverrides
 	}
-	return acmeIssuer
+
+	return ap, nil
 }
 
 // consolidateAutomationPolicies combines automation policies that are the same,
@@ -475,25 +497,39 @@ func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls
 		return len(aps[i].Subjects) > len(aps[j].Subjects)
 	})
 
-	// remove any empty policies (except subjects, of course)
+	emptyAPCount := 0
+	origLenAPs := len(aps)
+	// compute the number of empty policies (disregarding subjects) - see #4128
 	emptyAP := new(caddytls.AutomationPolicy)
 	for i := 0; i < len(aps); i++ {
 		emptyAP.Subjects = aps[i].Subjects
 		if reflect.DeepEqual(aps[i], emptyAP) {
-			aps = append(aps[:i], aps[i+1:]...)
-			i--
+			emptyAPCount++
+			if !automationPolicyHasAllPublicNames(aps[i]) {
+				// if this automation policy has internal names, we might as well remove it
+				// so auto-https can implicitly use the internal issuer
+				aps = append(aps[:i], aps[i+1:]...)
+				i--
+			}
 		}
 	}
+	// If all policies are empty, we can return nil, as there is no need to set any policy
+	if emptyAPCount == origLenAPs {
+		return nil
+	}
 
 	// remove or combine duplicate policies
+outer:
 	for i := 0; i < len(aps); i++ {
 		// compare only with next policies; we sorted by specificity so we must not delete earlier policies
 		for j := i + 1; j < len(aps); j++ {
 			// if they're exactly equal in every way, just keep one of them
 			if reflect.DeepEqual(aps[i], aps[j]) {
 				aps = append(aps[:j], aps[j+1:]...)
+				// must re-evaluate current i against next j; can't skip it!
+				// even if i decrements to -1, will be incremented to 0 immediately
 				i--
-				break
+				continue outer
 			}
 
 			// if the policy is the same, we can keep just one, but we have
@@ -518,6 +554,8 @@ func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls
 					// '*.com', which might be different (yes we've seen this happen)
 					if automationPolicyShadows(i, aps) >= j {
 						aps = append(aps[:i], aps[i+1:]...)
+						i--
+						continue outer
 					}
 				} else {
 					// avoid repeated subjects
@@ -574,3 +612,21 @@ func automationPolicyShadows(i int, aps []*caddytls.AutomationPolicy) int {
 	}
 	return -1
 }
+
+// subjectQualifiesForPublicCert is like certmagic.SubjectQualifiesForPublicCert() except
+// that this allows domains with multiple wildcard levels like '*.*.example.com' to qualify
+// if the automation policy has OnDemand enabled (i.e. this function is more lenient).
+func subjectQualifiesForPublicCert(ap *caddytls.AutomationPolicy, subj string) bool {
+	return !certmagic.SubjectIsIP(subj) &&
+		!certmagic.SubjectIsInternal(subj) &&
+		(strings.Count(subj, "*.") < 2 || ap.OnDemand)
+}
+
+func automationPolicyHasAllPublicNames(ap *caddytls.AutomationPolicy) bool {
+	for _, subj := range ap.Subjects {
+		if !subjectQualifiesForPublicCert(ap, subj) {
+			return false
+		}
+	}
+	return true
+}

caddyconfig/httploader.go

@@ -0,0 +1,151 @@
+package caddyconfig
+
+import (
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"time"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func init() {
+	caddy.RegisterModule(HTTPLoader{})
+}
+
+// HTTPLoader can load Caddy configs over HTTP(S). It can adapt the config
+// based on the Content-Type header of the HTTP response.
+type HTTPLoader struct {
+	// The method for the request. Default: GET
+	Method string `json:"method,omitempty"`
+
+	// The URL of the request.
+	URL string `json:"url,omitempty"`
+
+	// HTTP headers to add to the request.
+	Headers http.Header `json:"header,omitempty"`
+
+	// Maximum time allowed for a complete connection and request.
+	Timeout caddy.Duration `json:"timeout,omitempty"`
+
+	TLS *struct {
+		// Present this instance's managed remote identity credentials to the server.
+		UseServerIdentity bool `json:"use_server_identity,omitempty"`
+
+		// PEM-encoded client certificate filename to present to the server.
+		ClientCertificateFile string `json:"client_certificate_file,omitempty"`
+
+		// PEM-encoded key to use with the client certificate.
+		ClientCertificateKeyFile string `json:"client_certificate_key_file,omitempty"`
+
+		// List of PEM-encoded CA certificate files to add to the same trust
+		// store as RootCAPool (or root_ca_pool in the JSON).
+		RootCAPEMFiles []string `json:"root_ca_pem_files,omitempty"`
+	} `json:"tls,omitempty"`
+}
+
+// CaddyModule returns the Caddy module information.
+func (HTTPLoader) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "caddy.config_loaders.http",
+		New: func() caddy.Module { return new(HTTPLoader) },
+	}
+}
+
+// LoadConfig loads a Caddy config.
+func (hl HTTPLoader) LoadConfig(ctx caddy.Context) ([]byte, error) {
+	client, err := hl.makeClient(ctx)
+	if err != nil {
+		return nil, err
+	}
+
+	method := hl.Method
+	if method == "" {
+		method = http.MethodGet
+	}
+
+	req, err := http.NewRequest(method, hl.URL, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header = hl.Headers
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode >= 400 {
+		return nil, fmt.Errorf("server responded with HTTP %d", resp.StatusCode)
+	}
+
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	result, warnings, err := adaptByContentType(resp.Header.Get("Content-Type"), body)
+	if err != nil {
+		return nil, err
+	}
+	for _, warn := range warnings {
+		ctx.Logger(hl).Warn(warn.String())
+	}
+
+	return result, nil
+}
+
+func (hl HTTPLoader) makeClient(ctx caddy.Context) (*http.Client, error) {
+	client := &http.Client{
+		Timeout: time.Duration(hl.Timeout),
+	}
+
+	if hl.TLS != nil {
+		var tlsConfig *tls.Config
+
+		// client authentication
+		if hl.TLS.UseServerIdentity {
+			certs, err := ctx.IdentityCredentials(ctx.Logger(hl))
+			if err != nil {
+				return nil, fmt.Errorf("getting server identity credentials: %v", err)
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.Certificates = certs
+		} else if hl.TLS.ClientCertificateFile != "" && hl.TLS.ClientCertificateKeyFile != "" {
+			cert, err := tls.LoadX509KeyPair(hl.TLS.ClientCertificateFile, hl.TLS.ClientCertificateKeyFile)
+			if err != nil {
+				return nil, err
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.Certificates = []tls.Certificate{cert}
+		}
+
+		// trusted server certs
+		if len(hl.TLS.RootCAPEMFiles) > 0 {
+			rootPool := x509.NewCertPool()
+			for _, pemFile := range hl.TLS.RootCAPEMFiles {
+				pemData, err := ioutil.ReadFile(pemFile)
+				if err != nil {
+					return nil, fmt.Errorf("failed reading ca cert: %v", err)
+				}
+				rootPool.AppendCertsFromPEM(pemData)
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.RootCAs = rootPool
+		}
+
+		client.Transport = &http.Transport{TLSClientConfig: tlsConfig}
+	}
+
+	return client, nil
+}
+
+var _ caddy.ConfigLoader = (*HTTPLoader)(nil)

caddyconfig/load.go

@@ -69,8 +69,8 @@ func (al adminLoad) Routes() []caddy.AdminRoute {
 func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	if r.Method != http.MethodPost {
 		return caddy.APIError{
-			Code: http.StatusMethodNotAllowed,
-			Err:  fmt.Errorf("method not allowed"),
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method not allowed"),
 		}
 	}
 
@@ -81,8 +81,8 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	_, err := io.Copy(buf, r.Body)
 	if err != nil {
 		return caddy.APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("reading request body: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("reading request body: %v", err),
 		}
 	}
 	body := buf.Bytes()
@@ -90,45 +90,21 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	// if the config is formatted other than Caddy's native
 	// JSON, we need to adapt it before loading it
 	if ctHeader := r.Header.Get("Content-Type"); ctHeader != "" {
-		ct, _, err := mime.ParseMediaType(ctHeader)
+		result, warnings, err := adaptByContentType(ctHeader, body)
 		if err != nil {
 			return caddy.APIError{
-				Code: http.StatusBadRequest,
-				Err:  fmt.Errorf("invalid Content-Type: %v", err),
+				HTTPStatus: http.StatusBadRequest,
+				Err:        err,
 			}
 		}
-		if !strings.HasSuffix(ct, "/json") {
-			slashIdx := strings.Index(ct, "/")
-			if slashIdx < 0 {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("malformed Content-Type"),
-				}
-			}
-			adapterName := ct[slashIdx+1:]
-			cfgAdapter := GetAdapter(adapterName)
-			if cfgAdapter == nil {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("unrecognized config adapter '%s'", adapterName),
-				}
-			}
-			result, warnings, err := cfgAdapter.Adapt(body, nil)
+		if len(warnings) > 0 {
+			respBody, err := json.Marshal(warnings)
 			if err != nil {
-				return caddy.APIError{
-					Code: http.StatusBadRequest,
-					Err:  fmt.Errorf("adapting config using %s adapter: %v", adapterName, err),
-				}
+				caddy.Log().Named("admin.api.load").Error(err.Error())
 			}
-			if len(warnings) > 0 {
-				respBody, err := json.Marshal(warnings)
-				if err != nil {
-					caddy.Log().Named("admin.api.load").Error(err.Error())
-				}
-				_, _ = w.Write(respBody)
-			}
-			body = result
+			_, _ = w.Write(respBody)
 		}
+		body = result
 	}
 
 	forceReload := r.Header.Get("Cache-Control") == "must-revalidate"
@@ -136,8 +112,8 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	err = caddy.Load(body, forceReload)
 	if err != nil {
 		return caddy.APIError{
-			Code: http.StatusBadRequest,
-			Err:  fmt.Errorf("loading config: %v", err),
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("loading config: %v", err),
 		}
 	}
 
@@ -146,6 +122,47 @@ func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
 	return nil
 }
 
+// adaptByContentType adapts body to Caddy JSON using the adapter specified by contenType.
+// If contentType is empty or ends with "/json", the input will be returned, as a no-op.
+func adaptByContentType(contentType string, body []byte) ([]byte, []Warning, error) {
+	// assume JSON as the default
+	if contentType == "" {
+		return body, nil, nil
+	}
+
+	ct, _, err := mime.ParseMediaType(contentType)
+	if err != nil {
+		return nil, nil, caddy.APIError{
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("invalid Content-Type: %v", err),
+		}
+	}
+
+	// if already JSON, no need to adapt
+	if strings.HasSuffix(ct, "/json") {
+		return body, nil, nil
+	}
+
+	// adapter name should be suffix of MIME type
+	slashIdx := strings.Index(ct, "/")
+	if slashIdx < 0 {
+		return nil, nil, fmt.Errorf("malformed Content-Type")
+	}
+
+	adapterName := ct[slashIdx+1:]
+	cfgAdapter := GetAdapter(adapterName)
+	if cfgAdapter == nil {
+		return nil, nil, fmt.Errorf("unrecognized config adapter '%s'", adapterName)
+	}
+
+	result, warnings, err := cfgAdapter.Adapt(body, nil)
+	if err != nil {
+		return nil, nil, fmt.Errorf("adapting config using %s adapter: %v", adapterName, err)
+	}
+
+	return result, warnings, nil
+}
+
 var bufPool = sync.Pool{
 	New: func() interface{} {
 		return new(bytes.Buffer)

caddytest/caddytest.go

@@ -14,6 +14,7 @@ import (
 	"net/http/cookiejar"
 	"os"
 	"path"
+	"reflect"
 	"regexp"
 	"runtime"
 	"strings"
@@ -98,6 +99,10 @@ func (tc *Tester) InitServer(rawConfig string, configType string) {
 		tc.t.Logf("failed to load config: %s", err)
 		tc.t.Fail()
 	}
+	if err := tc.ensureConfigRunning(rawConfig, configType); err != nil {
+		tc.t.Logf("failed ensurng config is running: %s", err)
+		tc.t.Fail()
+	}
 }
 
 // InitServer this will configure the server with a configurion of a specific
@@ -171,20 +176,57 @@ func (tc *Tester) initServer(rawConfig string, configType string) error {
 	return nil
 }
 
-var hasValidated bool
-var arePrerequisitesValid bool
-
-func validateTestPrerequisites() error {
-
-	if hasValidated {
-		if !arePrerequisitesValid {
-			return errors.New("caddy integration prerequisites failed. see first error")
+func (tc *Tester) ensureConfigRunning(rawConfig string, configType string) error {
+	expectedBytes := []byte(prependCaddyFilePath(rawConfig))
+	if configType != "json" {
+		adapter := caddyconfig.GetAdapter(configType)
+		if adapter == nil {
+			return fmt.Errorf("adapter of config type is missing: %s", configType)
 		}
-		return nil
+		expectedBytes, _, _ = adapter.Adapt([]byte(rawConfig), nil)
 	}
 
-	hasValidated = true
-	arePrerequisitesValid = false
+	var expected interface{}
+	err := json.Unmarshal(expectedBytes, &expected)
+	if err != nil {
+		return err
+	}
+
+	client := &http.Client{
+		Timeout: Default.LoadRequestTimeout,
+	}
+
+	fetchConfig := func(client *http.Client) interface{} {
+		resp, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
+		if err != nil {
+			return nil
+		}
+		defer resp.Body.Close()
+		actualBytes, err := ioutil.ReadAll(resp.Body)
+		if err != nil {
+			return nil
+		}
+		var actual interface{}
+		err = json.Unmarshal(actualBytes, &actual)
+		if err != nil {
+			return nil
+		}
+		return actual
+	}
+
+	for retries := 4; retries > 0; retries-- {
+		if reflect.DeepEqual(expected, fetchConfig(client)) {
+			return nil
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	tc.t.Errorf("POSTed configuration isn't active")
+	return errors.New("EnsureConfigRunning: POSTed configuration isn't active")
+}
+
+// validateTestPrerequisites ensures the certificates are available in the
+// designated path and Caddy sub-process is running.
+func validateTestPrerequisites() error {
 
 	// check certificates are found
 	for _, certName := range Default.Certifcates {
@@ -200,20 +242,14 @@ func validateTestPrerequisites() error {
 			caddycmd.Main()
 		}()
 
-		// wait for caddy to start
-		retries := 4
-		for ; retries > 0 && isCaddyAdminRunning() != nil; retries-- {
+		// wait for caddy to start serving the initial config
+		for retries := 4; retries > 0 && isCaddyAdminRunning() != nil; retries-- {
 			time.Sleep(10 * time.Millisecond)
 		}
 	}
 
-	// assert that caddy is running
-	if err := isCaddyAdminRunning(); err != nil {
-		return err
-	}
-
-	arePrerequisitesValid = true
-	return nil
+	// one more time to return the error
+	return isCaddyAdminRunning()
 }
 
 func isCaddyAdminRunning() error {
@@ -223,7 +259,7 @@ func isCaddyAdminRunning() error {
 	}
 	resp, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
 	if err != nil {
-		return errors.New("caddy integration test caddy server not running. Expected to be listening on localhost:2019")
+		return fmt.Errorf("caddy integration test caddy server not running. Expected to be listening on localhost:%d", Default.AdminPort)
 	}
 	resp.Body.Close()
 
@@ -327,7 +363,7 @@ func (tc *Tester) AssertRedirect(requestURI string, expectedToLocation string, e
 }
 
 // CompareAdapt adapts a config and then compares it against an expected result
-func CompareAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) bool {
+func CompareAdapt(t *testing.T, filename, rawConfig string, adapterName string, expectedResponse string) bool {
 
 	cfgAdapter := caddyconfig.GetAdapter(adapterName)
 	if cfgAdapter == nil {
@@ -336,7 +372,6 @@ func CompareAdapt(t *testing.T, rawConfig string, adapterName string, expectedRe
 	}
 
 	options := make(map[string]interface{})
-	options["pretty"] = "true"
 
 	result, warnings, err := cfgAdapter.Adapt([]byte(rawConfig), options)
 	if err != nil {
@@ -344,9 +379,17 @@ func CompareAdapt(t *testing.T, rawConfig string, adapterName string, expectedRe
 		return false
 	}
 
+	// prettify results to keep tests human-manageable
+	var prettyBuf bytes.Buffer
+	err = json.Indent(&prettyBuf, result, "", "\t")
+	if err != nil {
+		return false
+	}
+	result = prettyBuf.Bytes()
+
 	if len(warnings) > 0 {
 		for _, w := range warnings {
-			t.Logf("warning: directive: %s : %s", w.Directive, w.Message)
+			t.Logf("warning: %s:%d: %s: %s", filename, w.Line, w.Directive, w.Message)
 		}
 	}
 
@@ -381,7 +424,7 @@ func CompareAdapt(t *testing.T, rawConfig string, adapterName string, expectedRe
 
 // AssertAdapt adapts a config and then tests it against an expected result
 func AssertAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) {
-	ok := CompareAdapt(t, rawConfig, adapterName, expectedResponse)
+	ok := CompareAdapt(t, "Caddyfile", rawConfig, adapterName, expectedResponse)
 	if !ok {
 		t.Fail()
 	}

caddytest/integration/autohttps_test.go

@@ -80,3 +80,46 @@ func TestAutoHTTPRedirectsWithHTTPListenerFirstInAddresses(t *testing.T) {
 `, "json")
 	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
 }
+
+func TestAutoHTTPRedirectsInsertedBeforeUserDefinedCatchAll(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+		local_certs
+	}
+	http://:9080 {
+		respond "Foo"
+	}
+	http://baz.localhost:9080 {
+		respond "Baz"
+	}
+	bar.localhost {
+		respond "Bar"
+	}
+  `, "caddyfile")
+	tester.AssertRedirect("http://bar.localhost:9080/", "https://bar.localhost/", http.StatusPermanentRedirect)
+	tester.AssertGetResponse("http://foo.localhost:9080/", 200, "Foo")
+	tester.AssertGetResponse("http://baz.localhost:9080/", 200, "Baz")
+}
+
+func TestAutoHTTPRedirectsInsertedBeforeUserDefinedCatchAllWithNoExplicitHTTPSite(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+		local_certs
+	}
+	http://:9080 {
+		respond "Foo"
+	}
+	bar.localhost {
+		respond "Bar"
+	}
+  `, "caddyfile")
+	tester.AssertRedirect("http://bar.localhost:9080/", "https://bar.localhost/", http.StatusPermanentRedirect)
+	tester.AssertGetResponse("http://foo.localhost:9080/", 200, "Foo")
+	tester.AssertGetResponse("http://baz.localhost:9080/", 200, "Foo")
+}

caddytest/integration/caddyfile_adapt/auto_https_ignore_loaded_certs.txt

@@ -0,0 +1,34 @@
+{
+	auto_https ignore_loaded_certs
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"ignore_loaded_certificates": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/encode_options.txt

@@ -0,0 +1,85 @@
+:80
+
+# All the options
+encode gzip zstd {
+	minimum_length 256
+	match {
+		status 2xx 4xx 500
+		header Content-Type text/*
+		header Content-Type application/json*
+		header Content-Type application/javascript*
+		header Content-Type application/xhtml+xml*
+		header Content-Type application/atom+xml*
+		header Content-Type application/rss+xml*
+		header Content-Type image/svg+xml*
+	}
+}
+
+# Long way with a block for each encoding
+encode {
+	zstd
+	gzip 5
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"encodings": {
+										"gzip": {},
+										"zstd": {}
+									},
+									"handler": "encode",
+									"match": {
+										"headers": {
+											"Content-Type": [
+												"text/*",
+												"application/json*",
+												"application/javascript*",
+												"application/xhtml+xml*",
+												"application/atom+xml*",
+												"application/rss+xml*",
+												"image/svg+xml*"
+											]
+										},
+										"status_code": [
+											2,
+											4,
+											500
+										]
+									},
+									"minimum_length": 256,
+									"prefer": [
+										"gzip",
+										"zstd"
+									]
+								},
+								{
+									"encodings": {
+										"gzip": {
+											"level": 5
+										},
+										"zstd": {}
+									},
+									"handler": "encode",
+									"prefer": [
+										"zstd",
+										"gzip"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/error_example.txt

@@ -0,0 +1,138 @@
+example.com {
+	root * /srv
+
+	# Trigger errors for certain paths
+	error /private* "Unauthorized" 403
+	error /hidden* "Not found" 404
+
+	# Handle the error by serving an HTML page 
+	handle_errors {
+		rewrite * /{http.error.status_code}.html
+		file_server
+	}
+
+	file_server
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "vars",
+													"root": "/srv"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"error": "Unauthorized",
+													"handler": "error",
+													"status_code": 403
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/private*"
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"error": "Not found",
+													"handler": "error",
+													"status_code": 404
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/hidden*"
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "file_server",
+													"hide": [
+														"./Caddyfile"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"errors": {
+						"routes": [
+							{
+								"match": [
+									{
+										"host": [
+											"example.com"
+										]
+									}
+								],
+								"handle": [
+									{
+										"handler": "subroute",
+										"routes": [
+											{
+												"group": "group0",
+												"handle": [
+													{
+														"handler": "rewrite",
+														"uri": "/{http.error.status_code}.html"
+													}
+												]
+											},
+											{
+												"handle": [
+													{
+														"handler": "file_server",
+														"hide": [
+															"./Caddyfile"
+														]
+													}
+												]
+											}
+										]
+									}
+								],
+								"terminal": true
+							}
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_disable_canonical_uris.txt

@@ -0,0 +1,32 @@
+:80
+
+file_server {
+	disable_canonical_uris
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"canonical_uris": false,
+									"handler": "file_server",
+									"hide": [
+										"./Caddyfile"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_precompressed.txt

@@ -0,0 +1,41 @@
+:80
+
+file_server {
+	precompressed zstd br gzip
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "file_server",
+									"hide": [
+										"./Caddyfile"
+									],
+									"precompressed": {
+										"br": {},
+										"gzip": {},
+										"zstd": {}
+									},
+									"precompressed_order": [
+										"zstd",
+										"br",
+										"gzip"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_status.txt

@@ -0,0 +1,112 @@
+localhost
+
+root * /srv
+
+handle /nope* {
+	file_server {
+		status 403
+	}
+}
+
+handle /custom-status* {
+	file_server {
+		status {env.CUSTOM_STATUS}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "vars",
+													"root": "/srv"
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "file_server",
+																	"hide": [
+																		"./Caddyfile"
+																	],
+																	"status_code": "{env.CUSTOM_STATUS}"
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/custom-status*"
+													]
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "file_server",
+																	"hide": [
+																		"./Caddyfile"
+																	],
+																	"status_code": 403
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/nope*"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options.txt

@@ -2,6 +2,7 @@
 	debug
 	http_port 8080
 	https_port 8443
+	grace_period 5s
 	default_sni localhost
 	order root first
 	storage file_system {
@@ -42,6 +43,7 @@
 		"http": {
 			"http_port": 8080,
 			"https_port": 8443,
+			"grace_period": 5000000000,
 			"servers": {
 				"srv0": {
 					"listen": [

caddytest/integration/caddyfile_adapt/global_options_acme.txt

@@ -9,7 +9,7 @@
 	}
 	acme_ca https://example.com
 	acme_eab {
-		key_id  4K2scIVbBpNd-78scadB2g
+		key_id 4K2scIVbBpNd-78scadB2g
 		mac_key abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh
 	}
 	acme_ca_root /path/to/ca.crt
@@ -20,6 +20,7 @@
 		interval 30s
 		burst 20
 	}
+	storage_clean_interval 7d
 
 	key_type ed25519
 }
@@ -80,7 +81,8 @@
 						"burst": 20
 					},
 					"ask": "https://example.com"
-				}
+				},
+				"storage_clean_interval": 604800000000000
 			}
 		}
 	}

caddytest/integration/caddyfile_adapt/global_options_log_and_site.txt

@@ -0,0 +1,77 @@
+{
+	log {
+		output file caddy.log
+		include some-log-source
+		exclude admin.api admin2.api
+	}
+	log custom-logger {
+		output file caddy.log
+		level WARN
+		include custom-log-source
+	}
+}
+
+:8884 {
+	log {
+		format json
+		output file access.log
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"custom-logger": {
+				"writer": {
+					"filename": "caddy.log",
+					"output": "file"
+				},
+				"level": "WARN",
+				"include": [
+					"custom-log-source"
+				]
+			},
+			"default": {
+				"writer": {
+					"filename": "caddy.log",
+					"output": "file"
+				},
+				"include": [
+					"some-log-source"
+				],
+				"exclude": [
+					"admin.api",
+					"admin2.api",
+					"custom-log-source",
+					"http.log.access.log0"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "access.log",
+					"output": "file"
+				},
+				"encoder": {
+					"format": "json"
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"logs": {
+						"default_logger_name": "log0"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_log_basic.txt

@@ -0,0 +1,18 @@
+{
+	log {
+		output file foo.log
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"writer": {
+					"filename": "foo.log",
+					"output": "file"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_log_custom.txt

@@ -0,0 +1,39 @@
+{
+	log custom-logger {
+		format filter {
+			wrap console
+			fields {
+				common_log delete
+				request>remote_addr ip_mask {
+					ipv4 24
+					ipv6 32
+				}
+			}
+		}
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"custom-logger": {
+				"encoder": {
+					"fields": {
+						"common_log": {
+							"filter": "delete"
+						},
+						"request\u003eremote_addr": {
+							"filter": "ip_mask",
+							"ipv4_cidr": 24,
+							"ipv6_cidr": 32
+						}
+					},
+					"format": "filter",
+					"wrap": {
+						"format": "console"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_log_multi.txt

@@ -0,0 +1,26 @@
+{
+	log first {
+		output file foo.log
+	}
+	log second {
+		format json
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"first": {
+				"writer": {
+					"filename": "foo.log",
+					"output": "file"
+				}
+			},
+			"second": {
+				"encoder": {
+					"format": "json"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_preferred_chains.txt

@@ -0,0 +1,56 @@
+{
+	preferred_chains smallest
+}
+
+example.com
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"module": "acme",
+								"preferred_chains": {
+									"smallest": true
+								}
+							},
+							{
+								"module": "zerossl",
+								"preferred_chains": {
+									"smallest": true
+								}
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_skip_install_trust.txt

@@ -0,0 +1,56 @@
+{
+	skip_install_trust
+}
+
+a.example.com {
+	tls internal
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"pki": {
+			"certificate_authorities": {
+				"local": {
+					"install_trust": false
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"a.example.com"
+						],
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_server_options_multi.txt

@@ -16,7 +16,7 @@
 	}
 }
 
-foo.com {	
+foo.com {
 }
 
 http://bar.com {
@@ -64,12 +64,7 @@ http://bar.com {
 							],
 							"terminal": true
 						}
-					],
-					"automatic_https": {
-						"skip": [
-							"bar.com"
-						]
-					}
+					]
 				},
 				"srv2": {
 					"listen": [

caddytest/integration/caddyfile_adapt/global_server_options_single.txt

@@ -18,7 +18,7 @@
 	}
 }
 
-foo.com {	
+foo.com {
 }
 
 ----------

caddytest/integration/caddyfile_adapt/header.txt

@@ -4,7 +4,7 @@
 	header ?John "von Neumann"
 	header -Wolfram
 	header {
-		Grace: "Hopper"  # some users habitually suffix field names with a colon
+		Grace: "Hopper" # some users habitually suffix field names with a colon
 		+Ray "Solomonoff"
 		?Tim "Berners-Lee"
 		defer

caddytest/integration/caddyfile_adapt/http_only_hostnames.txt

@@ -0,0 +1,45 @@
+# https://github.com/caddyserver/caddy/issues/3977
+http://* {
+	respond "Hello, world!"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "Hello, world!",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_domain.txt

@@ -46,12 +46,7 @@ http://a.caddy.localhost {
 							],
 							"terminal": true
 						}
-					],
-					"automatic_https": {
-						"skip": [
-							"a.caddy.localhost"
-						]
-					}
+					]
 				}
 			}
 		}

caddytest/integration/caddyfile_adapt/http_only_on_hostless_block.txt

@@ -0,0 +1,28 @@
+# Issue #4113
+:80, http://example.com {
+	respond "foo"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"body": "foo",
+									"handler": "static_response"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_snippet_env_placeholder.txt

@@ -0,0 +1,31 @@
+(foo) {
+	respond {env.FOO}
+}
+
+:80 {
+	import foo
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"body": "{env.FOO}",
+									"handler": "static_response"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_filters.txt

@@ -5,7 +5,7 @@ log {
 	format filter {
 		wrap console
 		fields {
-			request>headers>Authorization delete
+			request>headers>Authorization replace REDACTED
 			request>headers>Server delete
 			request>remote_addr ip_mask {
 				ipv4 24
@@ -30,7 +30,8 @@ log {
 				"encoder": {
 					"fields": {
 						"request\u003eheaders\u003eAuthorization": {
-							"filter": "delete"
+							"filter": "replace",
+							"value": "REDACTED"
 						},
 						"request\u003eheaders\u003eServer": {
 							"filter": "delete"
@@ -66,4 +67,4 @@ log {
 			}
 		}
 	}
-}
+}

caddytest/integration/caddyfile_adapt/log_skip_hosts.txt

@@ -0,0 +1,75 @@
+one.example.com {
+	log
+}
+
+two.example.com {
+}
+
+three.example.com {
+}
+
+example.com {
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"three.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"one.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"two.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"logs": {
+						"skip_hosts": [
+							"three.example.com",
+							"two.example.com",
+							"example.com"
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/matchers_in_route.txt

@@ -1,10 +1,10 @@
 :80 {
-    route {
-        # unused matchers should not panic
-        # see https://github.com/caddyserver/caddy/issues/3745
-        @matcher1 path /path1
-        @matcher2 path /path2
-    }
+	route {
+		# unused matchers should not panic
+		# see https://github.com/caddyserver/caddy/issues/3745
+		@matcher1 path /path1
+		@matcher2 path /path2
+	}
 }
 ----------
 {

caddytest/integration/caddyfile_adapt/php_fastcgi_subdirectives.txt

@@ -1,15 +1,15 @@
 :8884
 
 php_fastcgi localhost:9000 {
-    # some php_fastcgi-specific subdirectives
-    split .php .php5
-    env VAR1 value1
-    env VAR2 value2
-    root /var/www
-    index index.php5
+	# some php_fastcgi-specific subdirectives
+	split .php .php5
+	env VAR1 value1
+	env VAR2 value2
+	root /var/www
+	index index.php5
 
-    # passed through to reverse_proxy (directive order doesn't matter!)
-    lb_policy random
+	# passed through to reverse_proxy (directive order doesn't matter!)
+	lb_policy random
 }
 ----------
 {

caddytest/integration/caddyfile_adapt/request_body.txt

@@ -1,6 +1,7 @@
 localhost
+
 request_body {
-  max_size 1MB
+	max_size 1MB
 }
 ----------
 {

caddytest/integration/caddyfile_adapt/request_header.txt

@@ -0,0 +1,90 @@
+:80
+
+@matcher path /something*
+request_header @matcher Denis "Ritchie"
+
+request_header +Edsger "Dijkstra"
+request_header -Wolfram
+
+@images path /images/*
+request_header @images Cache-Control "public, max-age=3600, stale-while-revalidate=86400"
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/something*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "headers",
+									"request": {
+										"set": {
+											"Denis": [
+												"Ritchie"
+											]
+										}
+									}
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"path": [
+										"/images/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "headers",
+									"request": {
+										"set": {
+											"Cache-Control": [
+												"public, max-age=3600, stale-while-revalidate=86400"
+											]
+										}
+									}
+								}
+							]
+						},
+						{
+							"handle": [
+								{
+									"handler": "headers",
+									"request": {
+										"add": {
+											"Edsger": [
+												"Dijkstra"
+											]
+										}
+									}
+								},
+								{
+									"handler": "headers",
+									"request": {
+										"delete": [
+											"Wolfram"
+										]
+									}
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_empty_non_http_transport.txt

@@ -1,7 +1,7 @@
 :8884
 
 reverse_proxy 127.0.0.1:65535 {
-    transport fastcgi
+	transport fastcgi
 }
 ----------
 {

caddytest/integration/caddyfile_adapt/reverse_proxy_h2c_shorthand.txt

@@ -1,38 +1,38 @@
-:8884
-
-reverse_proxy h2c://localhost:8080
-----------
-{
-	"apps": {
-		"http": {
-			"servers": {
-				"srv0": {
-					"listen": [
-						":8884"
-					],
-					"routes": [
-						{
-							"handle": [
-								{
-									"handler": "reverse_proxy",
-									"transport": {
-										"protocol": "http",
-										"versions": [
-											"h2c",
-											"2"
-										]
-									},
-									"upstreams": [
-										{
-											"dial": "localhost:8080"
-										}
-									]
-								}
-							]
-						}
-					]
-				}
-			}
-		}
-	}
+:8884
+
+reverse_proxy h2c://localhost:8080
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"transport": {
+										"protocol": "http",
+										"versions": [
+											"h2c",
+											"2"
+										]
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:8080"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
 }

caddytest/integration/caddyfile_adapt/reverse_proxy_handle_response.txt

@@ -0,0 +1,193 @@
+:8884
+
+reverse_proxy 127.0.0.1:65535 {
+	@accel header X-Accel-Redirect *
+	handle_response @accel {
+		respond "Header X-Accel-Redirect!"
+	}
+
+	@another {
+		header X-Another *
+	}
+	handle_response @another {
+		respond "Header X-Another!"
+	}
+
+	@401 status 401
+	handle_response @401 {
+		respond "Status 401!"
+	}
+
+	handle_response {
+		respond "Any! This should be last in the JSON!"
+	}
+
+	@403 {
+		status 403
+	}
+	handle_response @403 {
+		respond "Status 403!"
+	}
+
+	@multi {
+		status 401 403
+		status 404
+		header Foo *
+		header Bar *
+	}
+	handle_response @multi {
+		respond "Headers Foo, Bar AND statuses 401, 403 and 404!"
+	}
+
+	@changeStatus status 500
+	handle_response @changeStatus 400
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handle_response": [
+										{
+											"match": {
+												"headers": {
+													"X-Accel-Redirect": [
+														"*"
+													]
+												}
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Header X-Accel-Redirect!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"headers": {
+													"X-Another": [
+														"*"
+													]
+												}
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Header X-Another!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"status_code": [
+													401
+												]
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Status 401!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"status_code": [
+													403
+												]
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Status 403!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"headers": {
+													"Bar": [
+														"*"
+													],
+													"Foo": [
+														"*"
+													]
+												},
+												"status_code": [
+													401,
+													403,
+													404
+												]
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Headers Foo, Bar AND statuses 401, 403 and 404!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										},
+										{
+											"match": {
+												"status_code": [
+													500
+												]
+											},
+											"status_code": 400
+										},
+										{
+											"routes": [
+												{
+													"handle": [
+														{
+															"body": "Any! This should be last in the JSON!",
+															"handler": "static_response"
+														}
+													]
+												}
+											]
+										}
+									],
+									"handler": "reverse_proxy",
+									"upstreams": [
+										{
+											"dial": "127.0.0.1:65535"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_health_headers.txt

@@ -0,0 +1,57 @@
+:8884
+
+reverse_proxy 127.0.0.1:65535 {
+	health_headers {
+		Host example.com
+		X-Header-Key 95ca39e3cbe7
+		X-Header-Keys VbG4NZwWnipo 335Q9/MhqcNU3s2TO
+		X-Empty-Value
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"health_checks": {
+										"active": {
+											"headers": {
+												"Host": [
+													"example.com"
+												],
+												"X-Empty-Value": [
+													""
+												],
+												"X-Header-Key": [
+													"95ca39e3cbe7"
+												],
+												"X-Header-Keys": [
+													"VbG4NZwWnipo",
+													"335Q9/MhqcNU3s2TO"
+												]
+											}
+										}
+									},
+									"upstreams": [
+										{
+											"dial": "127.0.0.1:65535"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_health_path_query.txt

@@ -0,0 +1,75 @@
+# Health with query in the uri
+:8443 {
+	reverse_proxy localhost:54321 {
+		health_uri /health?ready=1
+		health_status 2xx
+	}
+}
+
+# Health without query in the uri
+:8444 {
+	reverse_proxy localhost:54321 {
+		health_uri /health
+		health_status 200
+	}
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8443"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"health_checks": {
+										"active": {
+											"expect_status": 2,
+											"uri": "/health?ready=1"
+										}
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:54321"
+										}
+									]
+								}
+							]
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":8444"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"health_checks": {
+										"active": {
+											"expect_status": 200,
+											"uri": "/health"
+										}
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:54321"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_options.txt

@@ -21,7 +21,7 @@ https://example.com {
 			versions h2c 2
 			compression off
 			max_conns_per_host 5
-			max_idle_conns_per_host 2
+			keepalive_idle_conns_per_host 2
 		}
 	}
 }
@@ -79,8 +79,10 @@ https://example.com {
 														"dial_fallback_delay": 5000000000,
 														"dial_timeout": 3000000000,
 														"expect_continue_timeout": 9000000000,
+														"keep_alive": {
+															"max_idle_conns_per_host": 2
+														},
 														"max_conns_per_host": 5,
-														"max_idle_conns_per_host": 2,
 														"max_response_header_size": 30000000,
 														"protocol": "http",
 														"read_buffer_size": 10000000,

caddytest/integration/caddyfile_adapt/reverse_proxy_upstream_placeholder.txt

@@ -0,0 +1,102 @@
+:8884 {
+	map {host} {upstream} {
+		foo.example.com 1.2.3.4
+		default 2.3.4.5
+	}
+
+	# Upstream placeholder with a port should retain the port
+	reverse_proxy {upstream}:80
+}
+
+:8885 {
+	map {host} {upstream} {
+		foo.example.com 1.2.3.4:8080
+		default 2.3.4.5:8080
+	}
+
+	# Upstream placeholder with no port should not have a port joined
+	reverse_proxy {upstream}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"defaults": [
+										"2.3.4.5"
+									],
+									"destinations": [
+										"{upstream}"
+									],
+									"handler": "map",
+									"mappings": [
+										{
+											"input": "foo.example.com",
+											"outputs": [
+												"1.2.3.4"
+											]
+										}
+									],
+									"source": "{http.request.host}"
+								},
+								{
+									"handler": "reverse_proxy",
+									"upstreams": [
+										{
+											"dial": "{upstream}:80"
+										}
+									]
+								}
+							]
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":8885"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"defaults": [
+										"2.3.4.5:8080"
+									],
+									"destinations": [
+										"{upstream}"
+									],
+									"handler": "map",
+									"mappings": [
+										{
+											"input": "foo.example.com",
+											"outputs": [
+												"1.2.3.4:8080"
+											]
+										}
+									],
+									"source": "{http.request.host}"
+								},
+								{
+									"handler": "reverse_proxy",
+									"upstreams": [
+										{
+											"dial": "{upstream}"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/site_block_sorting.txt

@@ -0,0 +1,193 @@
+# https://caddy.community/t/caddy-suddenly-directs-my-site-to-the-wrong-directive/11597/2
+abcdef {
+	respond "abcdef"
+}
+
+abcdefg {
+	respond "abcdefg"
+}
+
+abc {
+	respond "abc"
+}
+
+abcde, http://abcde {
+	respond "abcde"
+}
+
+:443, ab {
+	respond "443 or ab"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"abcdefg"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "abcdefg",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"abcdef"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "abcdef",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"abcde"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "abcde",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"abc"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "abc",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "443 or ab",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"abcde"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "abcde",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"certificates": {
+				"automate": [
+					"ab"
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_acme_preferred_chains.txt

@@ -0,0 +1,57 @@
+localhost
+
+tls {
+	issuer acme {
+		preferred_chains {
+			any_common_name "Generic CA 1" "Generic CA 2"
+		}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"localhost"
+						],
+						"issuers": [
+							{
+								"module": "acme",
+								"preferred_chains": {
+									"any_common_name": [
+										"Generic CA 1",
+										"Generic CA 2"
+									]
+								}
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_2.txt

@@ -0,0 +1,92 @@
+# issue #3953
+{
+	cert_issuer zerossl api_key
+}
+
+example.com {
+	tls {
+		on_demand
+		key_type rsa2048
+	}
+}
+
+http://example.net {
+}
+
+:1234 {
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":1234"
+					]
+				},
+				"srv1": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv2": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.net"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"api_key": "api_key",
+								"module": "zerossl"
+							}
+						],
+						"key_type": "rsa2048",
+						"on_demand": true
+					},
+					{
+						"issuers": [
+							{
+								"api_key": "api_key",
+								"module": "zerossl"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_3.txt

@@ -0,0 +1,89 @@
+# https://caddy.community/t/caddyfile-having-individual-sites-differ-from-global-options/11297
+{
+	local_certs
+}
+
+a.example.com {
+	tls internal
+}
+
+b.example.com {
+	tls abc@example.com
+}
+
+c.example.com {
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"b.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"c.example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"b.example.com"
+						],
+						"issuers": [
+							{
+								"email": "abc@example.com",
+								"module": "acme"
+							},
+							{
+								"email": "abc@example.com",
+								"module": "zerossl"
+							}
+						]
+					},
+					{
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_4.txt

@@ -0,0 +1,143 @@
+{
+	email my.email@example.com
+}
+
+:82 {
+	redir https://example.com{uri}
+}
+
+:83 {
+	redir https://example.com{uri}
+}
+
+:84 {
+	redir https://example.com{uri}
+}
+
+abc.de {
+	redir https://example.com{uri}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"abc.de"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"headers": {
+														"Location": [
+															"https://example.com{http.request.uri}"
+														]
+													},
+													"status_code": 302
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":82"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "static_response",
+									"headers": {
+										"Location": [
+											"https://example.com{http.request.uri}"
+										]
+									},
+									"status_code": 302
+								}
+							]
+						}
+					]
+				},
+				"srv2": {
+					"listen": [
+						":83"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "static_response",
+									"headers": {
+										"Location": [
+											"https://example.com{http.request.uri}"
+										]
+									},
+									"status_code": 302
+								}
+							]
+						}
+					]
+				},
+				"srv3": {
+					"listen": [
+						":84"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "static_response",
+									"headers": {
+										"Location": [
+											"https://example.com{http.request.uri}"
+										]
+									},
+									"status_code": 302
+								}
+							]
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuers": [
+							{
+								"email": "my.email@example.com",
+								"module": "acme"
+							},
+							{
+								"email": "my.email@example.com",
+								"module": "zerossl"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_5.txt

@@ -0,0 +1,62 @@
+a.example.com {
+}
+
+b.example.com {
+}
+
+:443 {
+	tls {
+		on_demand
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"b.example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"a.example.com",
+							"b.example.com"
+						]
+					},
+					{
+						"on_demand": true
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_6.txt

@@ -0,0 +1,120 @@
+# (this Caddyfile is contrived, but based on issue #4161)
+
+example.com {
+	tls {
+		ca https://foobar
+	}
+}
+
+example.com:8443 {
+	tls {
+		ca https://foobar
+	}
+}
+
+example.com:8444 {
+	tls {
+		ca https://foobar
+	}
+}
+
+example.com:8445 {
+	tls {
+		ca https://foobar
+	}
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":8443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv2": {
+					"listen": [
+						":8444"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv3": {
+					"listen": [
+						":8445"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"ca": "https://foobar",
+								"module": "acme"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_7.txt

@@ -0,0 +1,68 @@
+# (this Caddyfile is contrived, but based on issues #4176 and #4198)
+
+http://example.com {
+}
+
+https://example.com {
+	tls internal
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				},
+				"srv1": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_automation_policies_global_email_localhost.txt

@@ -0,0 +1,66 @@
+{
+	email foo@bar
+}
+
+localhost {
+}
+
+example.com {
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"example.com"
+						],
+						"issuers": [
+							{
+								"email": "foo@bar",
+								"module": "acme"
+							},
+							{
+								"email": "foo@bar",
+								"module": "zerossl"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_client_auth_cert_file.txt

@@ -3,7 +3,7 @@ localhost
 respond "hello from localhost"
 tls {
 	client_auth {
-		mode request 
+		mode request
 		trusted_ca_cert_file ../caddy.ca.cer
 	}
 }

caddytest/integration/caddyfile_adapt/tls_client_auth_inline_cert.txt

@@ -3,8 +3,8 @@ localhost
 respond "hello from localhost"
 tls {
 	client_auth {
-		mode request 
-		trusted_ca_cert MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkwODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl03WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45twOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNxtdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTUApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAdBgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5uNY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfKD66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEOfG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnkoNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdleIh6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ== 
+		mode request
+		trusted_ca_cert MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkwODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl03WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45twOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNxtdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTUApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAdBgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5uNY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfKD66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEOfG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnkoNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdleIh6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ==
 	}
 }
 ----------

caddytest/integration/caddyfile_adapt/tls_conn_policy_consolidate.txt

@@ -75,12 +75,7 @@ http://b.b https://b.b:8443 {
 							],
 							"terminal": true
 						}
-					],
-					"automatic_https": {
-						"skip": [
-							"b.b"
-						]
-					}
+					]
 				},
 				"srv2": {
 					"listen": [

caddytest/integration/caddyfile_adapt/tls_propagation_timeout.txt

@@ -0,0 +1,70 @@
+localhost
+
+respond "hello from localhost"
+tls {
+	issuer acme {
+		propagation_timeout "10m0s"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"subjects": [
+							"localhost"
+						],
+						"issuers": [
+							{
+								"challenges": {
+									"dns": {
+										"propagation_timeout": 600000000000
+									}
+								},
+								"module": "acme"
+							}
+						]
+					}
+				]
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt_test.go

@@ -1,7 +1,10 @@
 package integration
 
 import (
+	jsonMod "encoding/json"
+	"fmt"
 	"io/ioutil"
+	"path/filepath"
 	"regexp"
 	"strings"
 	"testing"
@@ -32,14 +35,19 @@ func TestCaddyfileAdaptToJSON(t *testing.T) {
 		}
 
 		// split the Caddyfile (first) and JSON (second) parts
+		// (append newline to Caddyfile to match formatter expectations)
 		parts := strings.Split(string(data), "----------")
-		caddyfile, json := strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1])
+		caddyfile, json := strings.TrimSpace(parts[0])+"\n", strings.TrimSpace(parts[1])
 
 		// replace windows newlines in the json with unix newlines
 		json = winNewlines.ReplaceAllString(json, "\n")
 
+		// replace os-specific default path for file_server's hide field
+		replacePath, _ := jsonMod.Marshal(fmt.Sprint(".", string(filepath.Separator), "Caddyfile"))
+		json = strings.ReplaceAll(json, `"./Caddyfile"`, string(replacePath))
+
 		// run the test
-		ok := caddytest.CompareAdapt(t, caddyfile, "caddyfile", json)
+		ok := caddytest.CompareAdapt(t, filename, caddyfile, "caddyfile", json)
 		if !ok {
 			t.Errorf("failed to adapt %s", filename)
 		}

caddytest/integration/map_test.go

@@ -19,7 +19,7 @@ func TestMap(t *testing.T) {
 
 		map {http.request.method} {dest-1} {dest-2} {
 			default unknown1    unknown2
-			~G.T    get-called
+			~G(.)(.)    G${1}${2}-called
 			POST    post-called foobar
 		}
 
@@ -30,7 +30,7 @@ func TestMap(t *testing.T) {
 	`, "caddyfile")
 
 	// act and assert
-	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost get-called unknown2")
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost GET-called unknown2")
 	tester.AssertPostResponseBody("http://localhost:9080/version", []string{}, bytes.NewBuffer([]byte{}), 200, "hello from localhost post-called foobar")
 }
 

caddytest/integration/reverseproxy_test.go

@@ -371,7 +371,7 @@ func TestReverseProxyHealthCheck(t *testing.T) {
 		reverse_proxy {
 			to localhost:2020
 	
-			health_path /health
+			health_uri /health
 			health_port 2021
 			health_interval 2s
 			health_timeout 5s
@@ -426,7 +426,7 @@ func TestReverseProxyHealthCheckUnixSocket(t *testing.T) {
 		reverse_proxy {
 			to unix/%s
 	
-			health_path /health
+			health_uri /health
 			health_port 2021
 			health_interval 2s
 			health_timeout 5s
@@ -436,3 +436,57 @@ func TestReverseProxyHealthCheckUnixSocket(t *testing.T) {
 
 	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")
 }
+
+func TestReverseProxyHealthCheckUnixSocketWithoutPort(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.SkipNow()
+	}
+	tester := caddytest.NewTester(t)
+	f, err := ioutil.TempFile("", "*.sock")
+	if err != nil {
+		t.Errorf("failed to create TempFile: %s", err)
+		return
+	}
+	// a hack to get a file name within a valid path to use as socket
+	socketName := f.Name()
+	os.Remove(f.Name())
+
+	server := http.Server{
+		Handler: http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+			if strings.HasPrefix(req.URL.Path, "/health") {
+				w.Write([]byte("ok"))
+				return
+			}
+			w.Write([]byte("Hello, World!"))
+		}),
+	}
+
+	unixListener, err := net.Listen("unix", socketName)
+	if err != nil {
+		t.Errorf("failed to listen on the socket: %s", err)
+		return
+	}
+	go server.Serve(unixListener)
+	t.Cleanup(func() {
+		server.Close()
+	})
+	runtime.Gosched() // Allow other goroutines to run
+
+	tester.InitServer(fmt.Sprintf(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	http://localhost:9080 {
+		reverse_proxy {
+			to unix/%s
+	
+			health_uri /health
+			health_interval 2s
+			health_timeout 5s
+		}
+	}
+	`, socketName), "caddyfile")
+
+	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")
+}

cmd/caddy/setcap.sh

@@ -0,0 +1,6 @@
+#!/bin/sh
+
+# USAGE: go run -exec ./setcap.sh <args...>
+
+sudo setcap cap_net_bind_service=+ep "$1"
+"$@"

cmd/commandfuncs.go

@@ -19,6 +19,7 @@ import (
 	"context"
 	"crypto/rand"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"io/ioutil"
@@ -27,7 +28,6 @@ import (
 	"net/http"
 	"os"
 	"os/exec"
-	"reflect"
 	"runtime"
 	"runtime/debug"
 	"sort"
@@ -44,6 +44,7 @@ func cmdStart(fl Flags) (int, error) {
 	startCmdConfigAdapterFlag := fl.String("adapter")
 	startCmdPidfileFlag := fl.String("pidfile")
 	startCmdWatchFlag := fl.Bool("watch")
+	startCmdEnvfileFlag := fl.String("envfile")
 
 	// open a listener to which the child process will connect when
 	// it is ready to confirm that it has successfully started
@@ -67,6 +68,9 @@ func cmdStart(fl Flags) (int, error) {
 	if startCmdConfigFlag != "" {
 		cmd.Args = append(cmd.Args, "--config", startCmdConfigFlag)
 	}
+	if startCmdEnvfileFlag != "" {
+		cmd.Args = append(cmd.Args, "--envfile", startCmdEnvfileFlag)
+	}
 	if startCmdConfigAdapterFlag != "" {
 		cmd.Args = append(cmd.Args, "--adapter", startCmdConfigAdapterFlag)
 	}
@@ -116,7 +120,7 @@ func cmdStart(fl Flags) (int, error) {
 		for {
 			conn, err := ln.Accept()
 			if err != nil {
-				if !strings.Contains(err.Error(), "use of closed network connection") {
+				if !errors.Is(err, net.ErrClosed) {
 					log.Println(err)
 				}
 				break
@@ -173,9 +177,6 @@ func cmdRun(fl Flags) (int, error) {
 		printEnvironment()
 	}
 
-	// TODO: This is TEMPORARY, until the RCs
-	moveStorage()
-
 	// load the config, depending on flags
 	var config []byte
 	var err error
@@ -277,7 +278,7 @@ func cmdRun(fl Flags) (int, error) {
 func cmdStop(fl Flags) (int, error) {
 	stopCmdAddrFlag := fl.String("address")
 
-	err := apiRequest(stopCmdAddrFlag, http.MethodPost, "/stop", nil)
+	err := apiRequest(stopCmdAddrFlag, http.MethodPost, "/stop", nil, nil)
 	if err != nil {
 		caddy.Log().Warn("failed using API to stop instance", zap.Error(err))
 		return caddy.ExitCodeFailedStartup, err
@@ -290,6 +291,7 @@ func cmdReload(fl Flags) (int, error) {
 	reloadCmdConfigFlag := fl.String("config")
 	reloadCmdConfigAdapterFlag := fl.String("adapter")
 	reloadCmdAddrFlag := fl.String("address")
+	reloadCmdForceFlag := fl.Bool("force")
 
 	// get the config in caddy's native format
 	config, configFile, err := loadConfig(reloadCmdConfigFlag, reloadCmdConfigAdapterFlag)
@@ -314,7 +316,13 @@ func cmdReload(fl Flags) (int, error) {
 		adminAddr = tmpStruct.Admin.Listen
 	}
 
-	err = apiRequest(adminAddr, http.MethodPost, "/load", bytes.NewReader(config))
+	// optionally force a config reload
+	headers := make(http.Header)
+	if reloadCmdForceFlag {
+		headers.Set("Cache-Control", "must-revalidate")
+	}
+
+	err = apiRequest(adminAddr, http.MethodPost, "/load", headers, bytes.NewReader(config))
 	if err != nil {
 		return caddy.ExitCodeFailedStartup, fmt.Errorf("sending configuration to instance: %v", err)
 	}
@@ -323,19 +331,7 @@ func cmdReload(fl Flags) (int, error) {
 }
 
 func cmdVersion(_ Flags) (int, error) {
-	goModule := caddy.GoModule()
-	fmt.Print(goModule.Version)
-	if goModule.Sum != "" {
-		// a build with a known version will also have a checksum
-		fmt.Printf(" %s", goModule.Sum)
-	}
-	if goModule.Replace != nil {
-		fmt.Printf(" => %s", goModule.Replace.Path)
-		if goModule.Replace.Version != "" {
-			fmt.Printf(" %s", goModule.Replace.Version)
-		}
-	}
-	fmt.Println()
+	fmt.Println(CaddyVersion())
 	return caddy.ExitCodeSuccess, nil
 }
 
@@ -345,8 +341,11 @@ func cmdBuildInfo(fl Flags) (int, error) {
 		return caddy.ExitCodeFailedStartup, fmt.Errorf("no build information")
 	}
 
-	fmt.Printf("path: %s\n", bi.Path)
-	fmt.Printf("main: %s %s %s\n", bi.Main.Path, bi.Main.Version, bi.Main.Sum)
+	fmt.Printf("go_version: %s\n", runtime.Version())
+	fmt.Printf("go_os:      %s\n", runtime.GOOS)
+	fmt.Printf("go_arch:    %s\n", runtime.GOARCH)
+	fmt.Printf("path:       %s\n", bi.Path)
+	fmt.Printf("main:       %s %s %s\n", bi.Main.Path, bi.Main.Version, bi.Main.Sum)
 	fmt.Println("dependencies:")
 
 	for _, goMod := range bi.Deps {
@@ -360,59 +359,60 @@ func cmdBuildInfo(fl Flags) (int, error) {
 }
 
 func cmdListModules(fl Flags) (int, error) {
+	packages := fl.Bool("packages")
 	versions := fl.Bool("versions")
 
-	bi, ok := debug.ReadBuildInfo()
-	if !ok || !versions {
-		// if there's no build information,
-		// just print out the modules
+	printModuleInfo := func(mi moduleInfo) {
+		fmt.Print(mi.caddyModuleID)
+		if versions && mi.goModule != nil {
+			fmt.Print(" " + mi.goModule.Version)
+		}
+		if packages && mi.goModule != nil {
+			fmt.Print(" " + mi.goModule.Path)
+			if mi.goModule.Replace != nil {
+				fmt.Print(" => " + mi.goModule.Replace.Path)
+			}
+		}
+		if mi.err != nil {
+			fmt.Printf(" [%v]", mi.err)
+		}
+		fmt.Println()
+	}
+
+	// organize modules by whether they come with the standard distribution
+	standard, nonstandard, unknown, err := getModules()
+	if err != nil {
+		// oh well, just print the module IDs and exit
 		for _, m := range caddy.Modules() {
 			fmt.Println(m)
 		}
 		return caddy.ExitCodeSuccess, nil
 	}
 
-	for _, modID := range caddy.Modules() {
-		modInfo, err := caddy.GetModule(modID)
-		if err != nil {
-			// that's weird
-			fmt.Println(modID)
-			continue
+	if len(standard) > 0 {
+		for _, mod := range standard {
+			printModuleInfo(mod)
 		}
-
-		// to get the Caddy plugin's version info, we need to know
-		// the package that the Caddy module's value comes from; we
-		// can use reflection but we need a non-pointer value (I'm
-		// not sure why), and since New() should return a pointer
-		// value, we need to dereference it first
-		iface := interface{}(modInfo.New())
-		if rv := reflect.ValueOf(iface); rv.Kind() == reflect.Ptr {
-			iface = reflect.New(reflect.TypeOf(iface).Elem()).Elem().Interface()
-		}
-		modPkgPath := reflect.TypeOf(iface).PkgPath()
-
-		// now we find the Go module that the Caddy module's package
-		// belongs to; we assume the Caddy module package path will
-		// be prefixed by its Go module path, and we will choose the
-		// longest matching prefix in case there are nested modules
-		var matched *debug.Module
-		for _, dep := range bi.Deps {
-			if strings.HasPrefix(modPkgPath, dep.Path) {
-				if matched == nil || len(dep.Path) > len(matched.Path) {
-					matched = dep
-				}
-			}
-		}
-
-		// if we could find no matching module, just print out
-		// the module ID instead
-		if matched == nil {
-			fmt.Println(modID)
-			continue
-		}
-
-		fmt.Printf("%s %s\n", modID, matched.Version)
 	}
+	fmt.Printf("\n  Standard modules: %d\n", len(standard))
+	if len(nonstandard) > 0 {
+		if len(standard) > 0 {
+			fmt.Println()
+		}
+		for _, mod := range nonstandard {
+			printModuleInfo(mod)
+		}
+	}
+	fmt.Printf("\n  Non-standard modules: %d\n", len(nonstandard))
+	if len(unknown) > 0 {
+		if len(standard) > 0 || len(nonstandard) > 0 {
+			fmt.Println()
+		}
+		for _, mod := range unknown {
+			printModuleInfo(mod)
+		}
+	}
+	fmt.Printf("\n  Unknown modules: %d\n", len(unknown))
 
 	return caddy.ExitCodeSuccess, nil
 }
@@ -463,17 +463,25 @@ func cmdAdaptConfig(fl Flags) (int, error) {
 			fmt.Errorf("reading input file: %v", err)
 	}
 
-	opts := make(map[string]interface{})
-	if adaptCmdPrettyFlag {
-		opts["pretty"] = "true"
-	}
-	opts["filename"] = adaptCmdInputFlag
+	opts := map[string]interface{}{"filename": adaptCmdInputFlag}
 
 	adaptedConfig, warnings, err := cfgAdapter.Adapt(input, opts)
 	if err != nil {
 		return caddy.ExitCodeFailedStartup, err
 	}
 
+	if adaptCmdPrettyFlag {
+		var prettyBuf bytes.Buffer
+		err = json.Indent(&prettyBuf, adaptedConfig, "", "\t")
+		if err != nil {
+			return caddy.ExitCodeFailedStartup, err
+		}
+		adaptedConfig = prettyBuf.Bytes()
+	}
+
+	// print result to stdout
+	fmt.Println(string(adaptedConfig))
+
 	// print warnings to stderr
 	for _, warn := range warnings {
 		msg := warn.Message
@@ -483,9 +491,6 @@ func cmdAdaptConfig(fl Flags) (int, error) {
 		fmt.Fprintf(os.Stderr, "[WARNING][%s] %s:%d: %s\n", adaptCmdAdapterFlag, warn.File, warn.Line, msg)
 	}
 
-	// print result to stdout
-	fmt.Println(string(adaptedConfig))
-
 	// validate output if requested
 	if adaptCmdValidateFlag {
 		var cfg *caddy.Config
@@ -631,7 +636,7 @@ commands:
 // apiRequest makes an API request to the endpoint adminAddr with the
 // given HTTP method and request URI. If body is non-nil, it will be
 // assumed to be Content-Type application/json.
-func apiRequest(adminAddr, method, uri string, body io.Reader) error {
+func apiRequest(adminAddr, method, uri string, headers http.Header, body io.Reader) error {
 	// parse the admin address
 	if adminAddr == "" {
 		adminAddr = caddy.DefaultAdminListen
@@ -671,6 +676,9 @@ func apiRequest(adminAddr, method, uri string, body io.Reader) error {
 	if body != nil {
 		req.Header.Set("Content-Type", "application/json")
 	}
+	for k, v := range headers {
+		req.Header[k] = v
+	}
 
 	// make an HTTP client that dials our network type, since admin
 	// endpoints aren't always TCP, which is what the default transport
@@ -700,3 +708,9 @@ func apiRequest(adminAddr, method, uri string, body io.Reader) error {
 
 	return nil
 }
+
+type moduleInfo struct {
+	caddyModuleID string
+	goModule      *debug.Module
+	err           error
+}

cmd/commands.go

@@ -61,6 +61,12 @@ type Command struct {
 // any error that occurred.
 type CommandFunc func(Flags) (int, error)
 
+// Commands returns a list of commands initialised by
+// RegisterCommand
+func Commands() map[string]Command {
+	return commands
+}
+
 var commands = make(map[string]Command)
 
 func init() {
@@ -74,18 +80,22 @@ func init() {
 	RegisterCommand(Command{
 		Name:  "start",
 		Func:  cmdStart,
-		Usage: "[--config <path> [--adapter <name>]] [--watch] [--pidfile <file>]",
+		Usage: "[--config <path> [--adapter <name>]] [--envfile <path>] [--watch] [--pidfile <file>]",
 		Short: "Starts the Caddy process in the background and then returns",
 		Long: `
 Starts the Caddy process, optionally bootstrapped with an initial config file.
 This command unblocks after the server starts running or fails to run.
 
+If --envfile is specified, an environment file with environment variables in
+the KEY=VALUE format will be loaded into the Caddy process.
+
 On Windows, the spawned child process will remain attached to the terminal, so
 closing the window will forcefully stop Caddy; to avoid forgetting this, try
 using 'caddy run' instead to keep it in the foreground.`,
 		Flags: func() *flag.FlagSet {
 			fs := flag.NewFlagSet("start", flag.ExitOnError)
 			fs.String("config", "", "Configuration file")
+			fs.String("envfile", "", "Environment file to load")
 			fs.String("adapter", "", "Name of config adapter to apply")
 			fs.String("pidfile", "", "Path of file to which to write process ID")
 			fs.Bool("watch", false, "Reload changed config file automatically")
@@ -178,6 +188,7 @@ config file; otherwise the default is assumed.`,
 			fs.String("config", "", "Configuration file (required)")
 			fs.String("adapter", "", "Name of config adapter to apply")
 			fs.String("address", "", "Address of the administration listener, if different from config")
+			fs.Bool("force", false, "Force config reload, even if it is the same")
 			return fs
 		}(),
 	})
@@ -191,10 +202,11 @@ config file; otherwise the default is assumed.`,
 	RegisterCommand(Command{
 		Name:  "list-modules",
 		Func:  cmdListModules,
-		Usage: "[--versions]",
+		Usage: "[--packages] [--versions]",
 		Short: "Lists the installed Caddy modules",
 		Flags: func() *flag.FlagSet {
 			fs := flag.NewFlagSet("list-modules", flag.ExitOnError)
+			fs.Bool("packages", false, "Print package paths")
 			fs.Bool("versions", false, "Print version information")
 			return fs
 		}(),
@@ -276,6 +288,39 @@ is always printed to stdout.`,
 		}(),
 	})
 
+	RegisterCommand(Command{
+		Name:  "upgrade",
+		Func:  cmdUpgrade,
+		Short: "Upgrade Caddy (EXPERIMENTAL)",
+		Long: `
+Downloads an updated Caddy binary with the same modules/plugins at the
+latest versions. EXPERIMENTAL: May be changed or removed.`,
+	})
+
+	RegisterCommand(Command{
+		Name:  "add-package",
+		Func:  cmdAddPackage,
+		Usage: "<packages...>",
+		Short: "Adds Caddy packages (EXPERIMENTAL)",
+		Long: `
+Downloads an updated Caddy binary with the specified packages (module/plugin)
+added. Retains existing packages. Returns an error if the any of packages are 
+already included. EXPERIMENTAL: May be changed or removed.
+`,
+	})
+
+	RegisterCommand(Command{
+		Name:  "remove-package",
+		Func:  cmdRemovePackage,
+		Usage: "<packages...>",
+		Short: "Removes Caddy packages (EXPERIMENTAL)",
+		Long: `
+Downloads an updated Caddy binaries without the specified packages (module/plugin). 
+Returns an error if any of the packages are not included. 
+EXPERIMENTAL: May be changed or removed.
+`,
+	})
+
 }
 
 // RegisterCommand registers the command cmd.

cmd/main.go

@@ -49,7 +49,7 @@ func init() {
 }
 
 // Main implements the main function of the caddy command.
-// Call this if Caddy is to be the main() if your program.
+// Call this if Caddy is to be the main() of your program.
 func Main() {
 	switch len(os.Args) {
 	case 0:
@@ -185,7 +185,7 @@ func loadConfig(configFile, adapterName string) ([]byte, string, error) {
 			if warn.Directive != "" {
 				msg = fmt.Sprintf("%s: %s", warn.Directive, warn.Message)
 			}
-			fmt.Printf("[WARNING][%s] %s:%d: %s\n", adapterName, warn.File, warn.Line, msg)
+			caddy.Log().Warn(msg, zap.String("adapter", adapterName), zap.String("file", warn.File), zap.Int("line", warn.Line))
 		}
 		config = adaptedConfig
 	}
@@ -311,7 +311,7 @@ func (f Flags) Int(name string) int {
 
 // Float64 returns the float64 representation of the
 // flag given by name. It returns false if the flag
-// is not a float63 type. It panics if the flag is
+// is not a float64 type. It panics if the flag is
 // not in the flag set.
 func (f Flags) Float64(name string) float64 {
 	val, _ := strconv.ParseFloat(f.String(name), 64)
@@ -361,6 +361,11 @@ func loadEnvFromFile(envFile string) error {
 		}
 	}
 
+	// Update the storage paths to ensure they have the proper
+	// value after loading a specified env file.
+	caddy.ConfigAutosavePath = filepath.Join(caddy.AppConfigDir(), "autosave.json")
+	caddy.DefaultStorage = &certmagic.FileStorage{Path: caddy.AppDataDir()}
+
 	return nil
 }
 
@@ -415,7 +420,7 @@ func printEnvironment() {
 	fmt.Printf("caddy.AppDataDir=%s\n", caddy.AppDataDir())
 	fmt.Printf("caddy.AppConfigDir=%s\n", caddy.AppConfigDir())
 	fmt.Printf("caddy.ConfigAutosavePath=%s\n", caddy.ConfigAutosavePath)
-	fmt.Printf("caddy.Version=%s\n", caddy.GoModule().Version)
+	fmt.Printf("caddy.Version=%s\n", CaddyVersion())
 	fmt.Printf("runtime.GOOS=%s\n", runtime.GOOS)
 	fmt.Printf("runtime.GOARCH=%s\n", runtime.GOARCH)
 	fmt.Printf("runtime.Compiler=%s\n", runtime.Compiler)
@@ -432,70 +437,21 @@ func printEnvironment() {
 	}
 }
 
-// moveStorage moves the old default dataDir to the new default dataDir.
-// TODO: This is TEMPORARY until the release candidates.
-func moveStorage() {
-	// get the home directory (the old way)
-	oldHome := os.Getenv("HOME")
-	if oldHome == "" && runtime.GOOS == "windows" {
-		drive := os.Getenv("HOMEDRIVE")
-		path := os.Getenv("HOMEPATH")
-		oldHome = drive + path
-		if drive == "" || path == "" {
-			oldHome = os.Getenv("USERPROFILE")
+// CaddyVersion returns a detailed version string, if available.
+func CaddyVersion() string {
+	goModule := caddy.GoModule()
+	ver := goModule.Version
+	if goModule.Sum != "" {
+		ver += " " + goModule.Sum
+	}
+	if goModule.Replace != nil {
+		ver += " => " + goModule.Replace.Path
+		if goModule.Replace.Version != "" {
+			ver += "@" + goModule.Replace.Version
+		}
+		if goModule.Replace.Sum != "" {
+			ver += " " + goModule.Replace.Sum
 		}
 	}
-	if oldHome == "" {
-		oldHome = "."
-	}
-	oldDataDir := filepath.Join(oldHome, ".local", "share", "caddy")
-
-	// nothing to do if old data dir doesn't exist
-	_, err := os.Stat(oldDataDir)
-	if os.IsNotExist(err) {
-		return
-	}
-
-	// nothing to do if the new data dir is the same as the old one
-	newDataDir := caddy.AppDataDir()
-	if oldDataDir == newDataDir {
-		return
-	}
-
-	logger := caddy.Log().Named("automigrate").With(
-		zap.String("old_dir", oldDataDir),
-		zap.String("new_dir", newDataDir))
-
-	logger.Info("beginning one-time data directory migration",
-		zap.String("details", "https://github.com/caddyserver/caddy/issues/2955"))
-
-	// if new data directory exists, avoid auto-migration as a conservative safety measure
-	_, err = os.Stat(newDataDir)
-	if !os.IsNotExist(err) {
-		logger.Error("new data directory already exists; skipping auto-migration as conservative safety measure",
-			zap.Error(err),
-			zap.String("instructions", "https://github.com/caddyserver/caddy/issues/2955#issuecomment-570000333"))
-		return
-	}
-
-	// construct the new data directory's parent folder
-	err = os.MkdirAll(filepath.Dir(newDataDir), 0700)
-	if err != nil {
-		logger.Error("unable to make new datadirectory - follow link for instructions",
-			zap.String("instructions", "https://github.com/caddyserver/caddy/issues/2955#issuecomment-570000333"),
-			zap.Error(err))
-		return
-	}
-
-	// folder structure is same, so just try to rename (move) it;
-	// this fails if the new path is on a separate device
-	err = os.Rename(oldDataDir, newDataDir)
-	if err != nil {
-		logger.Error("new data directory already exists; skipping auto-migration as conservative safety measure - follow link for instructions",
-			zap.String("instructions", "https://github.com/caddyserver/caddy/issues/2955#issuecomment-570000333"),
-			zap.Error(err))
-	}
-
-	logger.Info("successfully completed one-time migration of data directory",
-		zap.String("details", "https://github.com/caddyserver/caddy/issues/2955"))
+	return ver
 }

cmd/packagesfuncs.go

@@ -0,0 +1,306 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddycmd
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"os"
+	"os/exec"
+	"reflect"
+	"runtime"
+	"runtime/debug"
+	"strings"
+
+	"github.com/caddyserver/caddy/v2"
+	"go.uber.org/zap"
+)
+
+func cmdUpgrade(_ Flags) (int, error) {
+	_, nonstandard, _, err := getModules()
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("unable to enumerate installed plugins: %v", err)
+	}
+	pluginPkgs, err := getPluginPackages(nonstandard)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	return upgradeBuild(pluginPkgs)
+}
+
+func cmdAddPackage(fl Flags) (int, error) {
+	if len(fl.Args()) == 0 {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("at least one package name must be specified")
+	}
+	_, nonstandard, _, err := getModules()
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("unable to enumerate installed plugins: %v", err)
+	}
+	pluginPkgs, err := getPluginPackages(nonstandard)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	for _, arg := range fl.Args() {
+		if _, ok := pluginPkgs[arg]; ok {
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("package is already added")
+		}
+		pluginPkgs[arg] = struct{}{}
+	}
+
+	return upgradeBuild(pluginPkgs)
+}
+
+func cmdRemovePackage(fl Flags) (int, error) {
+	if len(fl.Args()) == 0 {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("at least one package name must be specified")
+	}
+	_, nonstandard, _, err := getModules()
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("unable to enumerate installed plugins: %v", err)
+	}
+	pluginPkgs, err := getPluginPackages(nonstandard)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	for _, arg := range fl.Args() {
+		if _, ok := pluginPkgs[arg]; !ok {
+			// package does not exist
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("package is not added")
+		}
+		delete(pluginPkgs, arg)
+	}
+
+	return upgradeBuild(pluginPkgs)
+}
+
+func upgradeBuild(pluginPkgs map[string]struct{}) (int, error) {
+	l := caddy.Log()
+
+	thisExecPath, err := os.Executable()
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("determining current executable path: %v", err)
+	}
+	thisExecStat, err := os.Stat(thisExecPath)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("retrieving current executable permission bits: %v", err)
+	}
+	l.Info("this executable will be replaced", zap.String("path", thisExecPath))
+
+	// build the request URL to download this custom build
+	qs := url.Values{
+		"os":   {runtime.GOOS},
+		"arch": {runtime.GOARCH},
+	}
+	for pkg := range pluginPkgs {
+		qs.Add("p", pkg)
+	}
+
+	// initiate the build
+	resp, err := downloadBuild(qs)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("download failed: %v", err)
+	}
+	defer resp.Body.Close()
+
+	// back up the current binary, in case something goes wrong we can replace it
+	backupExecPath := thisExecPath + ".tmp"
+	l.Info("build acquired; backing up current executable",
+		zap.String("current_path", thisExecPath),
+		zap.String("backup_path", backupExecPath))
+	err = os.Rename(thisExecPath, backupExecPath)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("backing up current binary: %v", err)
+	}
+	defer func() {
+		if err != nil {
+			err2 := os.Rename(backupExecPath, thisExecPath)
+			if err2 != nil {
+				l.Error("restoring original executable failed; will need to be restored manually",
+					zap.String("backup_path", backupExecPath),
+					zap.String("original_path", thisExecPath),
+					zap.Error(err2))
+			}
+		}
+	}()
+
+	// download the file; do this in a closure to close reliably before we execute it
+	err = writeCaddyBinary(thisExecPath, &resp.Body, thisExecStat)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup, err
+	}
+
+	l.Info("download successful; displaying new binary details", zap.String("location", thisExecPath))
+
+	// use the new binary to print out version and module info
+	fmt.Print("\nModule versions:\n\n")
+	if err = listModules(thisExecPath); err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to execute: %v", err)
+	}
+	fmt.Println("\nVersion:")
+	if err = showVersion(thisExecPath); err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to execute: %v", err)
+	}
+	fmt.Println()
+
+	// clean up the backup file
+	if err = os.Remove(backupExecPath); err != nil {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("download succeeded, but unable to clean up backup binary: %v", err)
+	}
+	l.Info("upgrade successful; please restart any running Caddy instances", zap.String("executable", thisExecPath))
+
+	return caddy.ExitCodeSuccess, nil
+}
+
+func getModules() (standard, nonstandard, unknown []moduleInfo, err error) {
+	bi, ok := debug.ReadBuildInfo()
+	if !ok {
+		err = fmt.Errorf("no build info")
+		return
+	}
+
+	for _, modID := range caddy.Modules() {
+		modInfo, err := caddy.GetModule(modID)
+		if err != nil {
+			// that's weird, shouldn't happen
+			unknown = append(unknown, moduleInfo{caddyModuleID: modID, err: err})
+			continue
+		}
+
+		// to get the Caddy plugin's version info, we need to know
+		// the package that the Caddy module's value comes from; we
+		// can use reflection but we need a non-pointer value (I'm
+		// not sure why), and since New() should return a pointer
+		// value, we need to dereference it first
+		iface := interface{}(modInfo.New())
+		if rv := reflect.ValueOf(iface); rv.Kind() == reflect.Ptr {
+			iface = reflect.New(reflect.TypeOf(iface).Elem()).Elem().Interface()
+		}
+		modPkgPath := reflect.TypeOf(iface).PkgPath()
+
+		// now we find the Go module that the Caddy module's package
+		// belongs to; we assume the Caddy module package path will
+		// be prefixed by its Go module path, and we will choose the
+		// longest matching prefix in case there are nested modules
+		var matched *debug.Module
+		for _, dep := range bi.Deps {
+			if strings.HasPrefix(modPkgPath, dep.Path) {
+				if matched == nil || len(dep.Path) > len(matched.Path) {
+					matched = dep
+				}
+			}
+		}
+
+		caddyModGoMod := moduleInfo{caddyModuleID: modID, goModule: matched}
+
+		if strings.HasPrefix(modPkgPath, caddy.ImportPath) {
+			standard = append(standard, caddyModGoMod)
+		} else {
+			nonstandard = append(nonstandard, caddyModGoMod)
+		}
+	}
+	return
+}
+
+func listModules(path string) error {
+	cmd := exec.Command(path, "list-modules", "--versions")
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	err := cmd.Run()
+	if err != nil {
+		return fmt.Errorf("download succeeded, but unable to execute: %v", err)
+	}
+	return nil
+}
+
+func showVersion(path string) error {
+	cmd := exec.Command(path, "version")
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	err := cmd.Run()
+	if err != nil {
+		return fmt.Errorf("download succeeded, but unable to execute: %v", err)
+	}
+	return nil
+}
+
+func downloadBuild(qs url.Values) (*http.Response, error) {
+	l := caddy.Log()
+	l.Info("requesting build",
+		zap.String("os", qs.Get("os")),
+		zap.String("arch", qs.Get("arch")),
+		zap.Strings("packages", qs["p"]))
+	resp, err := http.Get(fmt.Sprintf("%s?%s", downloadPath, qs.Encode()))
+	if err != nil {
+		return nil, fmt.Errorf("secure request failed: %v", err)
+	}
+	if resp.StatusCode >= 400 {
+		var details struct {
+			StatusCode int `json:"status_code"`
+			Error      struct {
+				Message string `json:"message"`
+				ID      string `json:"id"`
+			} `json:"error"`
+		}
+		err2 := json.NewDecoder(resp.Body).Decode(&details)
+		if err2 != nil {
+			return nil, fmt.Errorf("download and error decoding failed: HTTP %d: %v", resp.StatusCode, err2)
+		}
+		return nil, fmt.Errorf("download failed: HTTP %d: %s (id=%s)", resp.StatusCode, details.Error.Message, details.Error.ID)
+	}
+	return resp, nil
+}
+
+func getPluginPackages(modules []moduleInfo) (map[string]struct{}, error) {
+	pluginPkgs := make(map[string]struct{})
+	for _, mod := range modules {
+		if mod.goModule.Replace != nil {
+			return nil, fmt.Errorf("cannot auto-upgrade when Go module has been replaced: %s => %s",
+				mod.goModule.Path, mod.goModule.Replace.Path)
+		}
+		pluginPkgs[mod.goModule.Path] = struct{}{}
+	}
+	return pluginPkgs, nil
+}
+
+func writeCaddyBinary(path string, body *io.ReadCloser, fileInfo os.FileInfo) error {
+	l := caddy.Log()
+	destFile, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, fileInfo.Mode())
+	if err != nil {
+		return fmt.Errorf("unable to open destination file: %v", err)
+	}
+	defer destFile.Close()
+
+	l.Info("downloading binary", zap.String("destination", path))
+
+	_, err = io.Copy(destFile, *body)
+	if err != nil {
+		return fmt.Errorf("unable to download file: %v", err)
+	}
+
+	err = destFile.Sync()
+	if err != nil {
+		return fmt.Errorf("syncing downloaded file to device: %v", err)
+	}
+
+	return nil
+}
+
+const downloadPath = "https://caddyserver.com/api/download"

context.go

@@ -430,5 +430,13 @@ func (ctx Context) Storage() certmagic.Storage {
 
 // Logger returns a logger that can be used by mod.
 func (ctx Context) Logger(mod Module) *zap.Logger {
+	if ctx.cfg == nil {
+		// often the case in tests; just use a dev logger
+		l, err := zap.NewDevelopment()
+		if err != nil {
+			panic("config missing, unable to create dev logger: " + err.Error())
+		}
+		return l
+	}
 	return ctx.cfg.Logging.Logger(mod)
 }

duration_fuzz.go

@@ -0,0 +1,25 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// +build gofuzz
+
+package caddy
+
+func FuzzParseDuration(data []byte) int {
+	_, err := ParseDuration(string(data))
+	if err != nil {
+		return 0
+	}
+	return 1
+}