metrics: Always track method label in uppercase (#3742)

* metrics: Always track method label in uppercase

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Just use strings.ToUpper for clarity

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

.github/CONTRIBUTING.md

@@ -0,0 +1,184 @@
+Contributing to Caddy
+=====================
+
+Welcome! Thank you for choosing to be a part of our community. Caddy wouldn't be great without your involvement!
+
+For starters, we invite you to join [the Caddy forum](https://caddy.community) where you can hang out with other Caddy users and developers.
+
+## Common Tasks
+
+- [Contributing code](#contributing-code)
+- [Writing a Caddy module](#writing-a-caddy-module)
+- [Asking or answering questions for help using Caddy](#getting-help-using-caddy)
+- [Reporting a bug](#reporting-bugs)
+- [Suggesting an enhancement or a new feature](#suggesting-features)
+- [Improving documentation](#improving-documentation)
+
+Other menu items:
+
+- [Values](#values)
+- [Coordinated Disclosure](#coordinated-disclosure)
+- [Thank You](#thank-you)
+
+
+### Contributing code
+
+You can have a huge impact on the project by helping with its code. To contribute code to Caddy, open a [pull request](https://github.com/caddyserver/caddy/pulls) (PR). If you're new to our community, that's okay: **we gladly welcome pull requests from anyone, regardless of your native language or coding experience.** You can get familiar with Caddy's code base by using [code search at Sourcegraph](https://sourcegraph.com/github.com/caddyserver/caddy/-/search).
+
+We hold contributions to a high standard for quality :bowtie:, so don't be surprised if we ask for revisions—even if it seems small or insignificant. Please don't take it personally. :blue_heart: If your change is on the right track, we can guide you to make it mergable.
+
+Here are some of the expectations we have of contributors:
+
+- **Open an issue to propose your change first.** This way we can avoid confusion, coordinate what everyone is working on, and ensure that any changes are in-line with the project's goals and the best interests of its users. We can also discuss the best possible implementation. If there's already an issue about it, comment on the existing issue to claim it.
+
+- **Keep pull requests small.** Smaller PRs are more likely to be merged because they are easier to review! We might ask you to break up large PRs into smaller ones. [An example of what we want to avoid.](https://twitter.com/iamdevloper/status/397664295875805184)
+
+- **Keep related commits together in a PR.** We do want pull requests to be small, but you should also keep multiple related commits in the same PR if they rely on each other.
+
+- **Write tests.** Tests are essential! Written properly, they ensure your change works, and that other changes in the future won't break your change. CI checks should pass.
+
+- **Benchmarks should be included for optimizations.** Optimizations sometimes make code harder to read or have changes that are less than obvious. They should be proven with benchmarks or profiling.
+
+- **[Squash](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html) insignificant commits.** Every commit should be significant. Commits which merely rewrite a comment or fix a typo can be combined into another commit that has more substance. Interactive rebase can do this, or a simpler way is `git reset --soft <diverging-commit>` then `git commit -s`.
+
+- **Own your contributions.** Caddy is a growing project, and it's much better when individual contributors help maintain their change after it is merged.
+
+- **Use comments properly.** We expect good godoc comments for package-level functions, types, and values. Comments are also useful whenever the purpose for a line of code is not obvious.
+
+We often grant [collaborator status](#collaborator-instructions) to contributors who author one or more significant, high-quality PRs that are merged into the code base!
+
+
+#### HOW TO MAKE A PULL REQUEST TO CADDY
+
+Contributing to Go projects on GitHub is fun and easy. We recommend the following workflow:
+
+1. [Fork this repo](https://github.com/caddyserver/caddy). This makes a copy of the code you can write to.
+
+2. If you don't already have this repo (caddyserver/caddy.git) repo on your computer, get it with `go get github.com/caddyserver/caddy/v2`.
+
+3. Tell git that it can push the caddyserver/caddy.git repo to your fork by adding a remote: `git remote add myfork https://github.com/<your-username>/caddy.git`
+
+4. Make your changes in the caddyserver/caddy.git repo on your computer.
+
+5. Push your changes to your fork: `git push myfork`
+
+6. [Create a pull request](https://github.com/caddyserver/caddy/pull/new/master) to merge your changes into caddyserver/caddy @ master. (Click "compare across forks" and change the head fork.)
+
+This workflow is nice because you don't have to change import paths. You can get fancier by using different branches if you want.
+
+
+### Writing a Caddy module
+
+Caddy can do more with modules! Anyone can write one. Caddy modules are Go libraries that get compiled into Caddy, extending its feature set. They can add directives to the Caddyfile, add new configuration adapters, and even implement new server types (e.g. HTTP, DNS).
+
+[Learn how to write a module here](https://caddyserver.com/docs/extending-caddy). You should also share and discuss your module idea [on the forums](https://caddy.community) to have people test it out. We don't use the Caddy issue tracker for third-party modules.
+
+
+### Getting help using Caddy
+
+If you have a question about using Caddy, [ask on our forum](https://caddy.community)! There will be more people there who can help you than just the Caddy developers who follow our issue tracker. Issues are not the place for usage questions.
+
+Many people on the forums could benefit from your experience and expertise, too. Once you've been helped, consider giving back by answering other people's questions and participating in other discussions.
+
+
+### Reporting bugs
+
+Like every software, Caddy has its flaws. If you find one, [search the issues](https://github.com/caddyserver/caddy/issues) to see if it has already been reported. If not, [open a new issue](https://github.com/caddyserver/caddy/issues/new) and describe the bug, and somebody will look into it! (This repository is only for Caddy and its standard modules.)
+
+**You can help stop bugs in their tracks!** Speed up the patch by identifying the bug in the code. This can sometimes be done by adding `fmt.Println()` statements (or similar) in relevant code paths to narrow down where the problem may be. It's a good way to [introduce yourself to the Go language](https://tour.golang.org), too.
+
+Please follow the issue template so we have all the needed information. Unredacted&mdash;yes, actual values matter. We need to be able to repeat the bug using your instructions. Please simplify the issue as much as possible. The burden is on you to convince us that it is actually a bug in Caddy. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you!
+
+We suggest reading [How to Report Bugs Effectively](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html).
+
+Please be kind. :smile: Remember that Caddy comes at no cost to you, and you're getting free support when we fix your issues. If we helped you, please consider helping someone else!
+
+#### Bug reporting expectations
+
+Maintainers---or more generally, developers---need three things to act on bugs:
+
+1. To agree or be convinced that it's a bug (reporter's responsibility).
+	- A bug is undesired or surprising behavior which violates documentation or the spec.
+
+2. To be able to understand what is happening (mostly reporter's responsibility).
+	- If the reporter can provide satisfactory instructions such that a developer can reproduce the bug, the developer will likely be able to understand the bug, write a test case, and implement a fix.
+	- Otherwise, the burden is on the reporter to test possible solutions. This is discouraged because it loosens the feedback loop, slows down debugging efforts, obscures the true nature of the problem from the developers, and is unlikely to result in new test cases.
+
+3. A solution, or ideas toward a solution (mostly maintainer's responsibility).
+	- Sometimes the best solution is a documentation change.
+	- Usually the developers have the best domain knowledge for inventing a solution, but reporters may have ideas or preferences for how they would like the software to work.
+	- Security, correctness, and project goals/vision all take priority over a user's preferences.
+	- It's simply good business to yield a solution that satisfies the users, and it's even better business to leave them impressed.
+
+Thus, at the very least, the reporter is expected to:
+
+1. Convince the reader that it's a bug (if it's not obvious).
+2. Reduce the problem down to the minimum specific steps required to reproduce it.
+
+The maintainer is usually able to do the rest; but of course the reporter may invest additional effort to speed up the process.
+
+
+
+### Suggesting features
+
+First, [search to see if your feature has already been requested](https://github.com/caddyserver/caddy/issues). If it has, you can add a :+1: reaction to vote for it. If your feature idea is new, open an issue to request the feature. Please describe your idea thoroughly so that we know how to implement it! Really vague requests may not be helpful or actionable and, without clarification, will have to be closed.
+
+While we really do value your requests and implement many of them, not all features are a good fit for Caddy. Most of those [make good modules](#writing-a-caddy-module), which can be made by anyone! But if a feature is not in the best interest of the Caddy project or its users in general, we may politely decline to implement it into Caddy core.
+
+
+### Improving documentation
+
+Caddy's documentation is available at [https://caddyserver.com/docs](https://caddyserver.com/docs) and its source is in the [website repo](https://github.com/caddyserver/website). If you would like to make a fix to the docs, please submit an issue there describing the change to make.
+
+Note that third-party module documentation is not hosted by the Caddy website, other than basic usage examples. They are managed by the individual module authors, and you will have to contact them to change their documentation.
+
+
+
+## Collaborator Instructions
+
+Collaborators have push rights to the repository. We grant this permission after one or more successful, high-quality PRs are merged! We thank them for their help.The expectations we have of collaborators are:
+
+- **Help review pull requests.** Be meticulous, but also kind. We love our contributors, but we critique the contribution to make it better. Multiple, thorough reviews make for the best contributions! Here are some questions to consider:
+	- Can the change be made more elegant?
+	- Is this a maintenance burden?
+	- What assumptions does the code make?
+	- Is it well-tested?
+	- Is the change a good fit for the project?
+	- Does it actually fix the problem or is it creating a special case instead?
+	- Does the change incur any new dependencies? (Avoid these!)
+
+- **Answer issues.** If every collaborator helped out with issues, we could count the number of open issues on two hands. This means getting involved in the discussion, investigating the code, and yes, debugging it. It's fun. Really! :smile: Please, please help with open issues. Granted, some issues need to be done before others. And of course some are larger than others: you don't have to do it all yourself. Work with other collaborators as a team!
+
+- **Do not merge pull requests until they have been approved by one or two other collaborators.** If a project owner approves the PR, it can be merged (as long as the conversation has finished too).
+
+- **Prefer squashed commits over a messy merge.** If there are many little commits, please [squash the commits](https://stackoverflow.com/a/11732910/1048862) so we don't clutter the commit history.
+
+- **Don't accept new dependencies lightly.** Dependencies can make the world crash and burn, but they are sometimes necessary. Choose carefully. Extremely small dependencies (a few lines of code) can be inlined. The rest may not be needed. For those that are, Caddy uses [go modules](https://github.com/golang/go/wiki/Modules). All external dependencies must be installed as modules, and _Caddy must not export any types defined by those dependencies_. Check this diligently!
+
+- **Be extra careful in some areas of the code.** There are some critical areas in the Caddy code base that we review extra meticulously: the `caddyhttp` and `caddytls` packages especially.
+
+- **Make sure tests test the actual thing.** Double-check that the tests fail without the change, and pass with it. It's important that they assert what they're purported to assert.
+
+- **Recommended reading**
+	- [CodeReviewComments](https://github.com/golang/go/wiki/CodeReviewComments) for an idea of what we look for in good, clean Go code
+	- [Linus Torvalds describes a good commit message](https://gist.github.com/matthewhudson/1475276)
+	- [Best Practices for Maintainers](https://opensource.guide/best-practices/)
+	- [Shrinking Code Review](https://alexgaynor.net/2015/dec/29/shrinking-code-review/)
+
+
+
+## Values
+
+- A person is always more important than code. People don't like being handled "efficiently". But we can still process issues and pull requests efficiently while being kind, patient, and considerate.
+
+- The ends justify the means, if the means are good. A good tree won't produce bad fruit. But if we cut corners or are hasty in our process, the end result will not be good.
+
+
+## Security Policy
+
+If you think you've found a security vulnerability, please refer to our [Security Policy](https://github.com/caddyserver/caddy/security/policy) document.
+
+
+## Thank you
+
+Thanks for your help! Caddy would not be what it is today without your contributions.

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: [mholt] # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.github/SECURITY.md

@@ -0,0 +1,27 @@
+# Security Policy
+
+The Caddy project would like to make sure that it stays on top of all practically-exploitable vulnerabilities.
+
+Some security problems are more the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please report only vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing or BGP hijacks a vulnerability in the Caddy web server).
+
+Please note that we consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems.
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x     | :white_check_mark: |
+| 1.x     | :white_check_mark: (EOL 1 Oct 2020) |
+| < 1.x   | :x:                |
+
+## Reporting a Vulnerability
+
+Please email Matt Holt (the author) directly: matt [at] ardanlabs [dot com].
+
+We'll need enough information to verify the bug and make a patch. It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, resources permitting. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. Thank you for understanding.
+
+Please also understand that due to our nature as an open source project, we do not have a budget to award security bounties. We can only thank you.
+
+If your report is valid and a patch is released, we will not reveal your identity by default. If you wish to be credited, please give us the name to use.
+
+Thanks for responsibly helping Caddy&mdash;and thousands of websites&mdash;be more secure!

.github/workflows/ci.yml

@@ -0,0 +1,175 @@
+# Used as inspiration: https://github.com/mvdan/github-actions-golang
+
+name: Tests
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  test:
+    strategy:
+      # Default is true, cancels jobs for other platforms in the matrix if one fails
+      fail-fast: false
+      matrix:
+        os: [ ubuntu-latest, macos-latest, windows-latest ]
+        go: [ '1.14', '1.15' ]
+
+        # Set some variables per OS, usable via ${{ matrix.VAR }}
+        # CADDY_BIN_PATH: the path to the compiled Caddy binary, for artifact publishing
+        # SUCCESS: the typical value for $? per OS (Windows/pwsh returns 'True')
+        include:
+        - os: ubuntu-latest
+          CADDY_BIN_PATH: ./cmd/caddy/caddy
+          SUCCESS: 0
+
+        - os: macos-latest
+          CADDY_BIN_PATH: ./cmd/caddy/caddy
+          SUCCESS: 0
+
+        - os: windows-latest
+          CADDY_BIN_PATH: ./cmd/caddy/caddy.exe
+          SUCCESS: 'True'
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Install Go
+      uses: actions/setup-go@v2
+      with:
+        go-version: ${{ matrix.go }}
+
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    # These tools would be useful if we later decide to reinvestigate
+    # publishing test/coverage reports to some tool for easier consumption
+    # - name: Install test and coverage analysis tools
+    #   run: |
+    #     go get github.com/axw/gocov/gocov
+    #     go get github.com/AlekSi/gocov-xml
+    #     go get -u github.com/jstemmer/go-junit-report
+    #     echo "::add-path::$(go env GOPATH)/bin"
+
+    - name: Print Go version and environment
+      id: vars
+      run: |
+        printf "Using go at: $(which go)\n"
+        printf "Go version: $(go version)\n"
+        printf "\n\nGo environment:\n\n"
+        go env
+        printf "\n\nSystem environment:\n\n"
+        env
+        # Calculate the short SHA1 hash of the git commit
+        echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
+        echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+    - name: Cache the build cache
+      uses: actions/cache@v2
+      with:
+        path: ${{ steps.vars.outputs.go_cache }}
+        key: ${{ runner.os }}-${{ matrix.go }}-go-ci-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-${{ matrix.go }}-go-ci
+
+    - name: Get dependencies
+      run: |
+        go get -v -t -d ./...
+        # mkdir test-results
+
+    - name: Build Caddy
+      working-directory: ./cmd/caddy
+      env:
+        CGO_ENABLED: 0
+      run: |
+        go build -trimpath -ldflags="-w -s" -v
+
+    - name: Publish Build Artifact
+      uses: actions/upload-artifact@v1
+      with:
+        name: caddy_${{ runner.os }}_go${{ matrix.go }}_${{ steps.vars.outputs.short_sha }}
+        path: ${{ matrix.CADDY_BIN_PATH }}
+
+    # Commented bits below were useful to allow the job to continue
+    # even if the tests fail, so we can publish the report separately
+    # For info about set-output, see https://stackoverflow.com/questions/57850553/github-actions-check-steps-status
+    - name: Run tests
+      # id: step_test
+      # continue-on-error: true
+      run: |
+        # (go test -v -coverprofile=cover-profile.out -race ./... 2>&1) > test-results/test-result.out
+        go test -v -coverprofile="cover-profile.out" -short -race ./...
+        # echo "::set-output name=status::$?"
+
+    # Relevant step if we reinvestigate publishing test/coverage reports
+    # - name: Prepare coverage reports
+    #   run: |
+    #     mkdir coverage
+    #     gocov convert cover-profile.out > coverage/coverage.json
+    #     # Because Windows doesn't work with input redirection like *nix, but output redirection works.
+    #     (cat ./coverage/coverage.json | gocov-xml) > coverage/coverage.xml
+
+    # To return the correct result even though we set 'continue-on-error: true'
+    # - name: Coerce correct build result
+    #   if: matrix.os != 'windows-latest' && steps.step_test.outputs.status != ${{ matrix.SUCCESS }}
+    #   run: |
+    #     echo "step_test ${{ steps.step_test.outputs.status }}\n"
+    #     exit 1
+
+  s390x-test:
+    name: test (s390x on IBM Z)
+    runs-on: ubuntu-latest
+    if: github.event.pull_request.head.repo.full_name == github.repository
+    continue-on-error: true  # August 2020: s390x VM is down due to weather and power issues
+    steps:
+      - name: Checkout code into the Go module directory
+        uses: actions/checkout@v2
+      - name: Run Tests
+        run: |
+          mkdir -p ~/.ssh && echo -e "${SSH_KEY//_/\\n}" > ~/.ssh/id_ecdsa && chmod og-rwx ~/.ssh/id_ecdsa
+
+          # short sha is enough?
+          short_sha=$(git rev-parse --short HEAD)
+
+          # The environment is fresh, so there's no point in keeping accepting and adding the key.
+          rsync -arz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress --delete --exclude '.git' . caddy-ci@ci-s390x.caddyserver.com:/var/tmp/"$short_sha"
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -t caddy-ci@ci-s390x.caddyserver.com "cd /var/tmp/$short_sha; CGO_ENABLED=0 /usr/local/go/bin/go test -v ./..."
+          test_result=$?
+
+          # There's no need leaving the files around
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null caddy-ci@ci-s390x.caddyserver.com "rm -rf /var/tmp/'$short_sha'"
+
+          echo "Test exit code: $test_result"
+          exit $test_result
+        env:
+          SSH_KEY: ${{ secrets.S390X_SSH_KEY }}
+
+  # From https://github.com/reviewdog/action-golangci-lint
+  golangci-lint:
+    name: runner / golangci-lint
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code into the Go module directory
+        uses: actions/checkout@v2
+
+      - name: Run golangci-lint
+        uses: reviewdog/action-golangci-lint@v1
+        # uses: docker://reviewdog/action-golangci-lint:v1 # pre-build docker image
+        with:
+          github_token: ${{ secrets.github_token }}
+
+  goreleaser-check:
+    runs-on: ubuntu-latest
+    steps:
+      - name: checkout
+        uses: actions/checkout@v2
+      - uses: goreleaser/goreleaser-action@v2
+        with:
+          version: latest
+          args: check
+        env:
+          TAG: ${{ steps.vars.outputs.version_tag }}

.github/workflows/cross-build.yml

@@ -0,0 +1,60 @@
+name: Cross-Build
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  cross-build-test:
+    strategy:
+      fail-fast: false
+      matrix:
+        goos: ['android', 'linux', 'solaris', 'illumos', 'dragonfly', 'freebsd', 'openbsd', 'plan9', 'windows', 'darwin', 'netbsd']
+        go: [ '1.14', '1.15' ]
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: ${{ matrix.go }}
+
+      - name: Print Go version and environment
+        id: vars
+        run: |
+          printf "Using go at: $(which go)\n"
+          printf "Go version: $(go version)\n"
+          printf "\n\nGo environment:\n\n"
+          go env
+          printf "\n\nSystem environment:\n\n"
+          env
+          echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+      - name: Cache the build cache
+        uses: actions/cache@v2
+        with:
+          path: ${{ steps.vars.outputs.go_cache }}
+          key: cross-build-go${{ matrix.go }}-${{ matrix.goos }}-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            cross-build-go${{ matrix.go }}-${{ matrix.goos }}
+
+      - name: Checkout code into the Go module directory
+        uses: actions/checkout@v2
+
+      - name: Run Build
+        env:
+          CGO_ENABLED: 0
+          GOOS: ${{ matrix.goos }}
+        shell: bash
+        continue-on-error: true
+        working-directory: ./cmd/caddy
+        run: |
+          GOOS=$GOOS go build -trimpath -o caddy-"$GOOS"-amd64 2> /dev/null
+          if [ $? -ne 0 ]; then
+            echo "::warning ::$GOOS Build Failed"
+            exit 0
+          fi

.github/workflows/fuzzing.yml

@@ -0,0 +1,73 @@
+name: Fuzzing
+
+on:
+  # Daily midnight fuzzing
+  schedule:
+    - cron: '0 0 * * *'
+
+jobs:
+  fuzzing:
+    name: Fuzzing
+
+    strategy:
+      matrix:
+        os: [ ubuntu-latest ]
+        go: [ '1.14' ]
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Install Go
+      uses: actions/setup-go@v2
+      with:
+        go-version: ${{ matrix.go }}
+
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    - name: Download go-fuzz tools and the Fuzzit CLI, move Fuzzit CLI to GOBIN
+      # If we decide we need to prevent this from running on forks, we can use this line:
+      # if: github.repository == 'caddyserver/caddy'
+      run: |
+
+        go get -v github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build
+        wget -q -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v2.4.77/fuzzit_Linux_x86_64
+        chmod a+x fuzzit
+        mv fuzzit $(go env GOPATH)/bin
+        echo "::add-path::$(go env GOPATH)/bin"
+
+    - name: Generate fuzzers & submit them to Fuzzit
+      continue-on-error: true
+      env:
+        FUZZIT_API_KEY: ${{ secrets.FUZZIT_API_KEY }}
+        SYSTEM_PULLREQUEST_SOURCEBRANCH: ${{ github.ref }}
+        BUILD_SOURCEVERSION: ${{ github.sha }}
+      run: |
+        # debug
+        echo "PR Source Branch: $SYSTEM_PULLREQUEST_SOURCEBRANCH"
+        echo "Source version: $BUILD_SOURCEVERSION"
+
+        declare -A fuzzers_funcs=(\
+          ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="FuzzParseAddress" \
+          ["./listeners_fuzz.go"]="FuzzParseNetworkAddress" \
+          ["./replacer_fuzz.go"]="FuzzReplacer" \
+        )
+
+        declare -A fuzzers_targets=(\
+          ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="parse-address" \
+          ["./listeners_fuzz.go"]="parse-network-address" \
+          ["./replacer_fuzz.go"]="replacer" \
+        )
+
+        fuzz_type="fuzzing"
+
+        for f in $(find . -name \*_fuzz.go); do
+          FUZZER_DIRECTORY=$(dirname "$f")
+          
+          echo "go-fuzz-build func ${fuzzers_funcs[$f]} residing in $f"
+          
+          go-fuzz-build -func "${fuzzers_funcs[$f]}" -o "$FUZZER_DIRECTORY/${fuzzers_targets[$f]}.zip" "$FUZZER_DIRECTORY"
+          
+          fuzzit create job --engine go-fuzz caddyserver/"${fuzzers_targets[$f]}" "$FUZZER_DIRECTORY"/"${fuzzers_targets[$f]}.zip" --api-key "${FUZZIT_API_KEY}" --type "${fuzz_type}" --branch "${SYSTEM_PULLREQUEST_SOURCEBRANCH}" --revision "${BUILD_SOURCEVERSION}"
+          
+          echo "Completed $f"
+        done

.github/workflows/release.yml

@@ -0,0 +1,83 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - 'v*.*.*'
+
+jobs:
+  release:
+    name: Release
+    strategy:
+      matrix:
+        os: [ ubuntu-latest ]
+        go: [ '1.15' ]
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Install Go
+      uses: actions/setup-go@v2
+      with:
+        go-version: ${{ matrix.go }}
+
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    # So GoReleaser can generate the changelog properly
+    - name: Unshallowify the repo clone
+      run: git fetch --prune --unshallow
+
+    # https://github.community/t5/GitHub-Actions/How-to-get-just-the-tag-name/m-p/32167/highlight/true#M1027
+    - name: Print Go version and environment
+      id: vars
+      run: |
+        printf "Using go at: $(which go)\n"
+        printf "Go version: $(go version)\n"
+        printf "\n\nGo environment:\n\n"
+        go env
+        printf "\n\nSystem environment:\n\n"
+        env
+        echo "::set-output name=version_tag::${GITHUB_REF/refs\/tags\//}"
+        echo "::set-output name=short_sha::$(git rev-parse --short HEAD)"
+        echo "::set-output name=go_cache::$(go env GOCACHE)"
+
+        # Parse semver
+        TAG=${GITHUB_REF/refs\/tags\//}
+        SEMVER_RE='[^0-9]*\([0-9]*\)[.]\([0-9]*\)[.]\([0-9]*\)\([0-9A-Za-z\.-]*\)'
+        TAG_MAJOR=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\1#"`
+        TAG_MINOR=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\2#"`
+        TAG_PATCH=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\3#"`
+        TAG_SPECIAL=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\4#"`
+        echo "::set-output name=tag_major::${TAG_MAJOR}"
+        echo "::set-output name=tag_minor::${TAG_MINOR}"
+        echo "::set-output name=tag_patch::${TAG_PATCH}"
+        echo "::set-output name=tag_special::${TAG_SPECIAL}"
+
+    - name: Cache the build cache
+      uses: actions/cache@v2
+      with:
+        path: ${{ steps.vars.outputs.go_cache }}
+        key: ${{ runner.os }}-go${{ matrix.go }}-release-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-go${{ matrix.go }}-release
+
+    # GoReleaser will take care of publishing those artifacts into the release
+    - name: Run GoReleaser
+      uses: goreleaser/goreleaser-action@v2
+      with:
+        version: latest
+        args: release --rm-dist
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        TAG: ${{ steps.vars.outputs.version_tag }}
+
+    # Only publish on non-special tags (e.g. non-beta)
+    - name: Publish .deb to Gemfury
+      if: ${{ steps.vars.outputs.tag_special == '' }}
+      env:
+        GEMFURY_PUSH_TOKEN: ${{ secrets.GEMFURY_PUSH_TOKEN }}
+      run: |
+        for filename in dist/*.deb; do
+          curl -F package=@"$filename" https://${GEMFURY_PUSH_TOKEN}:@push.fury.io/caddy/
+        done
+

.github/workflows/release_published.yml

@@ -0,0 +1,34 @@
+name: Release Published
+
+# Event payload: https://developer.github.com/webhooks/event-payloads/#release
+on:
+  release:
+    types: [published]
+
+jobs:
+  release:
+    name: Release Published
+    strategy:
+      matrix:
+        os: [ ubuntu-latest ]
+    runs-on: ${{ matrix.os }}
+
+    steps:
+
+    # See https://github.com/peter-evans/repository-dispatch
+    - name: Trigger event on caddyserver/dist
+      uses: peter-evans/repository-dispatch@v1
+      with:
+        token: ${{ secrets.REPO_DISPATCH_TOKEN }}
+        repository: caddyserver/dist
+        event-type: release-tagged
+        client-payload: '{"tag": "${{ github.event.release.tag_name }}"}'
+
+    - name: Trigger event on caddyserver/caddy-docker
+      uses: peter-evans/repository-dispatch@v1
+      with:
+        token: ${{ secrets.REPO_DISPATCH_TOKEN }}
+        repository: caddyserver/caddy-docker
+        event-type: release-tagged
+        client-payload: '{"tag": "${{ github.event.release.tag_name }}"}'
+

.gitignore

@@ -1,6 +1,7 @@
 _gitignore/
 *.log
 Caddyfile
+!caddyfile/
 
 # artifacts from pprof tooling
 *.prof
@@ -15,3 +16,8 @@ cmd/caddy/caddy.exe
 
 # go modules
 vendor
+
+# goreleaser artifacts
+dist
+caddy-build
+caddy-dist

.golangci.yml

@@ -8,6 +8,8 @@ linters-settings:
 linters:
   enable:
     - bodyclose
+    - prealloc
+    - unconvert
     - errcheck
     - gofmt
     - goimports

.goreleaser.yml

@@ -0,0 +1,117 @@
+before:
+  hooks:
+    # The build is done in this particular way to build Caddy in a designated directory named in .gitignore.
+    # This is so we can run goreleaser on tag without Git complaining of being dirty. The main.go in cmd/caddy directory 
+    # cannot be built within that directory due to changes necessary for the build causing Git to be dirty, which
+    # subsequently causes gorleaser to refuse running.
+    - mkdir -p caddy-build
+    - cp cmd/caddy/main.go caddy-build/main.go
+    - cp ./go.mod caddy-build/go.mod
+    - sed -i.bkp 's|github.com/caddyserver/caddy/v2|caddy|g' ./caddy-build/go.mod
+    # GoReleaser doesn't seem to offer {{.Tag}} at this stage, so we have to embed it into the env
+    # so we run: TAG=$(git describe --abbrev=0) goreleaser release --rm-dist --skip-publish --skip-validate
+    - go mod edit -require=github.com/caddyserver/caddy/v2@{{.Env.TAG}} ./caddy-build/go.mod
+    - git clone --depth 1 https://github.com/caddyserver/dist caddy-dist
+    - go mod download
+
+builds:
+- env:
+  - CGO_ENABLED=0
+  - GO111MODULE=on
+  main: main.go
+  dir: ./caddy-build
+  binary: caddy
+  goos:
+  - darwin
+  - linux
+  - windows
+  - freebsd
+  goarch:
+  - amd64
+  - arm
+  - arm64
+  - s390x
+  - ppc64le
+  goarm:
+  - 5
+  - 6
+  - 7
+  ignore:
+    - goos: darwin
+      goarch: arm
+    - goos: darwin
+      goarch: ppc64le
+    - goos: darwin
+      goarch: s390x
+    - goos: windows
+      goarch: ppc64le
+    - goos: windows
+      goarch: s390x
+    - goos: freebsd
+      goarch: ppc64le
+    - goos: freebsd
+      goarch: s390x
+    - goos: freebsd
+      goarch: arm
+      goarm: 5
+  flags:
+  - -trimpath
+  ldflags:
+  - -s -w
+
+archives:
+  - format_overrides:
+      - goos: windows
+        format: zip
+    replacements:
+      darwin: mac
+checksum:
+  algorithm: sha512
+
+nfpms:
+  - id: default
+    package_name: caddy
+
+    vendor: Light Code Labs
+    homepage: https://caddyserver.com
+    maintainer: Matthew Holt <mholt@users.noreply.github.com>
+    description: |
+      Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go
+    license: Apache 2.0
+
+    formats:
+      - deb
+      # - rpm
+
+    bindir: /usr/bin
+    files:
+      ./caddy-dist/init/caddy.service: /lib/systemd/system/caddy.service
+      ./caddy-dist/init/caddy-api.service: /lib/systemd/system/caddy-api.service
+      ./caddy-dist/welcome/index.html: /usr/share/caddy/index.html
+      ./caddy-dist/scripts/completions/bash-completion: /etc/bash_completion.d/caddy
+    config_files:
+      ./caddy-dist/config/Caddyfile: /etc/caddy/Caddyfile
+
+    scripts:
+      postinstall: ./caddy-dist/scripts/postinstall.sh
+      preremove: ./caddy-dist/scripts/preremove.sh
+      postremove: ./caddy-dist/scripts/postremove.sh
+
+
+release:
+  github:
+    owner: caddyserver
+    name: caddy
+  draft: true
+  prerelease: auto
+
+changelog:
+  sort: asc
+  filters:
+    exclude:
+    - '^chore:'
+    - '^ci:'
+    - '^docs?:'
+    - '^readme:'
+    - '^tests?:'
+    - '^\w+\s+' # a hack to remove commit messages without colons thus don't correspond to a package

README.md

@@ -1,384 +1,151 @@
-Caddy 2 Development Branch
-===========================
+<p align="center">
+	<a href="https://caddyserver.com"><img src="https://user-images.githubusercontent.com/1128849/36338535-05fb646a-136f-11e8-987b-e6901e717d5a.png" alt="Caddy" width="450"></a>
+</p>
+<h3 align="center">Every site on HTTPS</h3>
+<p align="center">Caddy is an extensible server platform that uses TLS by default.</p>
+<p align="center">
+	<a href="https://github.com/caddyserver/caddy/actions?query=workflow%3ACross-Platform"><img src="https://github.com/caddyserver/caddy/workflows/Cross-Platform/badge.svg"></a>
+	<a href="https://pkg.go.dev/github.com/caddyserver/caddy/v2"><img src="https://img.shields.io/badge/godoc-reference-%23007d9c.svg"></a>
+	<br>
+	<a href="https://twitter.com/caddyserver" title="@caddyserver on Twitter"><img src="https://img.shields.io/badge/twitter-@caddyserver-55acee.svg" alt="@caddyserver on Twitter"></a>
+	<a href="https://caddy.community" title="Caddy Forum"><img src="https://img.shields.io/badge/community-forum-ff69b4.svg" alt="Caddy Forum"></a>
+	<a href="https://sourcegraph.com/github.com/caddyserver/caddy?badge" title="Caddy on Sourcegraph"><img src="https://sourcegraph.com/github.com/caddyserver/caddy/-/badge.svg" alt="Caddy on Sourcegraph"></a>
+</p>
+<p align="center">
+	<a href="https://github.com/caddyserver/caddy/releases">Releases</a> ·
+	<a href="https://caddyserver.com/docs/">Documentation</a> ·
+	<a href="https://caddy.community">Get Help</a>
+</p>
 
-[![Build Status](https://dev.azure.com/mholt-dev/Caddy/_apis/build/status/Multiplatform%20Tests?branchName=v2)](https://dev.azure.com/mholt-dev/Caddy/_build/latest?definitionId=5&branchName=v2)
-[![fuzzit](https://app.fuzzit.dev/badge?org_id=caddyserver)](https://app.fuzzit.dev/orgs/caddyserver/dashboard)
-
-This is the development branch for Caddy 2. This code (version 2) is not yet feature-complete or production-ready, but is already being used in production, and we encourage you to deploy it today on sites that are not very visible or important so that it can obtain crucial experience in the field.
-
-Please file issues to propose new features and report bugs, and after the bug or feature has been discussed, submit a pull request! We need your help to build this web server into what you want it to be. (Caddy 2 issues and pull requests receive priority over Caddy 1 issues and pull requests.)
-
-**Caddy 2 is the web server of the Go community.** We are looking for maintainers to represent the community! Please become involved (issues, PRs, [our forum](https://caddy.community) etc.) and express interest if you are committed to being a collaborator on the Caddy project.
 
 
 ### Menu
 
+- [Features](#features)
 - [Build from source](#build-from-source)
-- [Quick Start](#quick-start)
-- [Configuration](#configuration)
-- [Full Documentation](#full-documentation)
-- [List of Improvements](#list-of-improvements)
-- [FAQ](#faq)
+	- [For development](#for-development)
+	- [With version information and/or plugins](#with-version-information-andor-plugins)
+- [Quick start](#quick-start)
+- [Overview](#overview)
+- [Full documentation](#full-documentation)
+- [Getting help](#getting-help)
+- [About](#about)
+
+<p align="center">
+	<b>Powered by</b>
+	<br>
+	<a href="https://github.com/caddyserver/certmagic"><img src="https://user-images.githubusercontent.com/1128849/49704830-49d37200-fbd5-11e8-8385-767e0cd033c3.png" alt="CertMagic" width="250"></a>
+</p>
+
+
+## Features
+
+- **Easy configuration** with the [Caddyfile](https://caddyserver.com/docs/caddyfile)
+- **Powerful configuration** with its [native JSON config](https://caddyserver.com/docs/json/)
+- **Dynamic configuration** with the [JSON API](https://caddyserver.com/docs/api)
+- [**Config adapters**](https://caddyserver.com/docs/config-adapters) if you don't like JSON
+- **Automatic HTTPS** by default
+	- [Let's Encrypt](https://letsencrypt.org) for public sites
+	- Fully-managed local CA for internal names & IPs
+	- Can coordinate with other Caddy instances in a cluster
+- **Stays up when other servers go down** due to TLS/OCSP/certificate-related issues
+- **HTTP/1.1, HTTP/2, and experimental HTTP/3** support
+- **Highly extensible** [modular architecture](https://caddyserver.com/docs/architecture) lets Caddy do anything without bloat
+- **Runs anywhere** with **no external dependencies** (not even libc)
+- Written in Go, a language with higher **memory safety guarantees** than other servers
+- Actually **fun to use**
+- So, so much more to discover
+
 
 
 ## Build from source
 
 Requirements:
 
-- [Go 1.13 or newer](https://golang.org/dl/)
-- Do NOT disable [Go modules](https://github.com/golang/go/wiki/Modules) (`export GO111MODULE=auto`)
-
-Download the `v2` source code:
-
-```bash
-$ git clone -b v2 "https://github.com/caddyserver/caddy.git"
-```
-
-Build:
+- [Go 1.14 or newer](https://golang.org/dl/)
 
+### For development
+ 
 ```bash
+$ git clone "https://github.com/caddyserver/caddy.git"
 $ cd caddy/cmd/caddy/
 $ go build
 ```
 
-That will put a `caddy(.exe)` binary into the current directory. You can move it into your PATH or use `go install` to do that automatically (assuming `$GOPATH/bin` is already in your PATH). You can also use `go run main.go` for quick, temporary builds while developing.
+_**Note:** These steps [will not embed proper version information](https://github.com/golang/go/issues/29228). For that, please follow the instructions below._
 
-The initial build may be slow as dependencies are downloaded. Subsequent builds should be very fast. If you encounter any Go-module-related errors, try clearing your Go module cache (`$GOPATH/pkg/mod`) and Go package cache (`$GOPATH/pkg`) and read [the Go wiki page about modules for help](https://github.com/golang/go/wiki/Modules). If you have issues with Go modules, please consult the Go community for help. But if there is an actual error in Caddy, please report it to us.
+### With version information and/or plugins
 
-
-## Quick Start
-
-(Until the stable 2.0 release, there may be breaking changes in v2, please be aware!)
-
-These instructions assume an executable build of Caddy 2 is named `caddy` in the current folder. If it's in your PATH, you may omit the path to the binary (`./`).
-
-Start Caddy:
-
-```bash
-$ ./caddy start
-```
-
-There are no config files with Caddy 2. Instead, you POST configuration to it:
-
-```bash
-$ curl -X POST "http://localhost:2019/load" \
-    -H "Content-Type: application/json" \
-    -d @- << EOF
-    {
-        "apps": {
-            "http": {
-                "servers": {
-                    "example": {
-                        "listen": ["127.0.0.1:2080"],
-                        "routes": [
-                            {
-                                "handle": [{
-                                    "handler": "file_server",
-                                    "browse": {}
-                                }]
-                            }
-                        ]
-                    }
-                }
-            }
-        }
-    }
-EOF
-```
-
-Now visit http://localhost:2080 in your browser and you will see the contents of the current directory displayed.
-
-To change Caddy's configuration, simply POST a new payload to that endpoint. Config changes are extremely lightweight and efficient, and should be graceful on all platforms -- _even Windows_.
-
-Updating configuration using heredoc can be tedious, so you can still use a config file if you prefer. Put your configuration in any file (`caddy.json` for example) and then POST that instead:
-
-```bash
-$ curl -X POST "http://localhost:2019/load" \
-    -H "Content-Type: application/json" \
-    -d @caddy.json
-```
-
-Or you can tell Caddy to load its configuration from a file in the first place (this simply does the work of the above curl command for you):
-
-```bash
-$ ./caddy start --config caddy.json
-```
-
-To stop Caddy:
-
-```bash
-$ ./caddy stop
-```
-
-Note that this will stop any process named the same as `os.Args[0]`.
-
-For other commands, please see [the Caddy 2 documentation](https://github.com/caddyserver/caddy/wiki/v2:-Documentation).
-
-### Caddyfile
-
-Caddy 2 can be configured with a Caddyfile, much like in v1, for example:
-
-```plain
-example.com
-
-try_files {path}.html {path}
-encode gzip zstd
-reverse_proxy /api  localhost:9005
-php_fastcgi   /blog unix//path/to/socket
-file_server
-```
-
-Instead of being its primary mode of configuration, an internal _config adapter_ adapts the Caddyfile to Caddy's native JSON structure. You can see it in action with the [`adapt` command](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#adapt):
-
-```bash
-$ ./caddy adapt --config path/to/Caddyfile --adapter caddyfile --pretty
-```
-
-If you just want to run Caddy with your Caddyfile directly, the CLI wraps this up for you nicely. Either of the following commands:
-
-```bash
-$ ./caddy start
-$ ./caddy run
-```
-
-will use your Caddyfile if it is called `Caddyfile` in the current directory.
-
-If your Caddyfile is somewhere else, you can still use it:
-
-```bash
-$ ./caddy start|run --config path/to/Caddyfile --adapter caddyfile
-```
-
-[Learn more about the Caddyfile in v2.](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#caddyfile-adapter)
-
-
-## Configuration
-
-Caddy 2 exposes an unprecedented level of control compared to any web server in existence. In Caddy 2, you are usually setting the actual values of the initialized types in memory that power everything from your HTTP handlers and TLS handshakes to your storage medium. Caddy 2 is also ridiculously extensible, with a module system that makes vast improvements over Caddy 1's plugin system.
-
-Nearly all of Caddy 2's configuration is contained in a single config document, rather than being spread across CLI flags and env variables and a configuration file as with other web servers (and Caddy 1).
-
-To wield the power of this design, you need to know how the config document is structured. Please see the [the Caddy 2 documentation in our wiki](https://github.com/caddyserver/caddy/wiki/v2:-Documentation) for details about Caddy's config structure.
-
-Configuration is normally given to Caddy through an API endpoint, which is likewise documented in the wiki pages. However, you can also use config files of various formats with [config adapters](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#config-adapters).
-
-
-## Full Documentation
-
-Caddy 2 is very much in development, so the documentation is an ongoing WIP, but the latest will be in our wiki for now:
-
-**https://github.com/caddyserver/caddy/wiki/v2:-Documentation**
-
-Note that breaking changes are expected until the stable 2.0 release.
-
-
-## List of Improvements
-
-The following is a non-comprehensive list of significant improvements over Caddy 1. Not everything in this list is finished yet, but they will be finished or at least will be possible with Caddy 2:
-
-- Centralized configuration. No more disparate use of environment variables, config files (potentially multiple!), CLI flags, etc.
-- REST API. Control Caddy with HTTP requests to an administration endpoint. Changes are applied immediately and efficiently.
-- Dynamic configuration. Any and all specific config values can be modified directly through the admin API with a REST endpoint.
-    - Change only specific configuration settings instead of needing to specify the whole config each time. This makes it safe and easy to change Caddy's config with manually-crafted curl commands, for example.
-- No configuration files. Except optionally to bootstrap its configuration at startup. You can still use config files if you wish, and we expect that most people will.
-- Export the current Caddy configuration with an API GET request.
-- Silky-smooth graceful reloads. Update the configuration up to dozens of times per second with no dropped requests and very little memory cost. Our unique graceful reload technology is lighter and faster **and works on all platforms, including Windows**.
-- An embedded scripting language! Caddy2 has native Starlark integration. Do things you never thought possible with higher performance than Lua, JavaScript, and other VMs. Starlark is expressive, familiar (dialect of Python), _almost_ Turing-complete, and highly efficient. (We're still improving performance here.)
-- Using [XDG standards](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html#variables) instead of dumping all assets in `$HOME/.caddy`.
-- Caddy plugins are now called "Caddy modules" (although the terms "plugin" and "module" may be used interchangably). Caddy modules are a concept unrelated [Go modules](https://github.com/golang/go/wiki/Modules), except that Caddy modules may be implemented by Go modules. Caddy modules are centrally-registered, properly namespaced, and generically loaded & configured, as opposed to how scattered and unorganized Caddy 1-era plugins are.
-- Modules are easier to write, since they do not have to both deserialize their own configuration from a configuration DSL and provision themselves like plugins did. Modules are initialized pre-configured and have the ability to validate the configuration and perform provisioning steps if necessary.
-- Can specify different storage mechanisms in different parts of the configuration, if more than one is needed.
-- "Top-level" Caddy modules are simply called "apps" because literally any long-lived application can be served by Caddy 2.
-- Even more of Caddy is made of modules, allowing for unparalleled extensibility, flexibility, and control. Caddy 2 is arguably the most flexible, extensible, programmable web server ever made.
-- TLS improvements!
-	- TLS configuration is now centralized and decoupled from specific sites
-	- A single certificate cache is used process-wide, reducing duplication and improving memory use
-	- Customize how to manage each certificate ("automation policies") based on the hostname
-	- Automation policy doesn't have to be limited to just ACME - could be any way to manage certificates
-	- Fine-grained control over TLS handshakes
-	- If an ACME challenge fails, other enabled challenges will be tried (no other web server does this)
-    - TLS Session Ticket Ephemeral Keys (STEKs) can be rotated in a cluster for increased performance (no other web server does this either!)
-    - Ability to select a specific certificate per ClientHello given multiple qualifying certificates
-    - Provide TLS certificates without persisting them to disk; keep private keys entirely in memory
-	- Certificate management at startup is now asynchronous and much easier to use through machine reboots and in unsupervised settings
-- All-new HTTP server core
-	- Listeners can be configured for any network type, address, and port range
-	- Customizable TLS connection policies
-	- HTTP handlers are configured by "routes" which consist of matcher and handler components. Match matches an HTTP request, and handle defines the list of handlers to invoke as a result of the match.
-	- Some matchers are regular expressions, which expose capture groups to placeholders.
-	- New matchers include negation and matching based on remote IP address / CIDR ranges.
-    - Placeholders are vastly improved generally
-	- Placeholders (variables) are more properly namespaced.
-	- Multiple routes may match an HTTP request, creating a "composite route" quickly on the fly.
-	- The actual handler for any given request is its composite route.
-	- User defines the order of middlewares (careful! easy to break things).
-	- Adding middlewares no longer requires changes to Caddy's code base (there is no authoritative list).
-	- Routes may be marked as terminal, meaning no more routes will be matched.
-	- Routes may be grouped so that only the first matching route in a group is applied.
-	- Requests may be "re-handled" if they are modified and need to be sent through the chain again (internal redirect).
-	- Vastly more powerful static file server, with native content-negotiation abilities
-    - Done away with URL-rewriting hacks often needed in Caddy 1
-	- Highly descriptive/traceable errors
-	- Very flexible error handling, with the ability to specify a whole list of routes just for error cases
-	- The proxy has numerous improvements, including dynamic backends and more configurable health checks
-	- FastCGI support integrated with the reverse proxy
-	- More control over automatic HTTPS: disable entirely, disable only HTTP->HTTPS redirects, disable only cert management, and for certain names, etc.
-    - Use Starlark to build custom, dynamic HTTP handlers at request-time
-        - We are finding that -- on average -- Caddy 2's Starlark handlers are ~1.25-2x faster than NGINX+Lua.
-
-And a few major features still being worked on:
-
-- Logging
-- Kubernetes ingress controller (mostly done, just polishing it -- and it's amazing)
-- More config adapters. Caddy's native JSON config structure is powerful and complex. Config adapters upsample various formats to Caddy's native config. There are already adapters for Caddyfile, JSON 5, and JSON-C. Planned are NGINX config, YAML, and TOML. The community might be interested in building Traefik and Apache config adapters!
-
-
-
-## FAQ
-
-### How do I configure Caddy 2?
-
-Caddy's primary mode of configuration is a REST API, which accepts a JSON document. The JSON structure is described [in the wiki](https://github.com/caddyserver/caddy/wiki/v2:-Documentation). The advantages of exposing this low-level structure are 1) it has near-parity with actual memory initialization, 2) it allows us to offer wrappers over this configuration to any degree of convenience that is needed, and 3) it performs very well under rapid config changes.
-
-Basically, you will [start Caddy](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#start), then [POST a JSON config to its API endpoint](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#post-load).
-
-Although this makes Caddy 2 highly programmable, not everyone will want to configure Caddy via JSON with an API. Sometimes we just want to give Caddy a simple, static config file and have it do its thing. That's what **[config adapters](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#config-adapters)** are for! You can configure Caddy more ways than one, depending on your needs and preferences. See the next questions that explain this more.
-
-### Caddy 2 feels harder to use. How is this an improvement over Caddy 1?
-
-Caddy's ease of use is one of the main reasons it is special. We are not taking that away in Caddy 2, but first we had to be sure to tackle the fundamental design limitations with Caddy 1. Usability can then be layered on top. This approach has several advantages which we discuss in the next question.
-
-### What about the Caddyfile; are there easier ways to configure Caddy 2?
-
-Yes! Caddy's native JSON configuration via API is nice when you are automating config changes at scale, but if you just have a simple, static configuration in a file, you can do that too with the [Caddyfile](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#caddyfile-adapter).
-
-The v2 Caddyfile is very similar to the v1 Caddyfile, but they are not compatible. Several improvements have been made to request matching and directives in v2, giving you more power with less complexity and fewer inconsistencies.
-
-Caddy's default _config adapter_ is the Caddyfile adapter. This takes a Caddyfile as input and [outputs the JSON config](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#adapt). You can even run Caddy directly without having to see or think about the underlying JSON config.
-
-The following _config adapters_ are already being built or plan to be built:
-
-- Caddyfile
-- JSON 5
-- JSON-C
-- nginx
-- YAML
-- TOML
-- any others that the community would like to contribute
-
-Config adapters allow you to configure Caddy not just one way but _any_ of these ways. For example, you'll be able to bring your existing NGINX config to Caddy and it will spit out the Caddy config JSON you need (to the best of its ability). How cool is that! You can then easily tweak the resulting config by hand, if necessary.
-
-All config adapters vary in their theoretical expressiveness; that is, if you need more advanced configuration you'll have to drop down to the JSON config, because the Caddyfile or an nginx config may not be expressive enough.
-
-However, we expect that most users will be able to use the Caddyfile (or another easy config adapter) exclusively for their sites.
-
-### Why JSON for configuration? Why not _&lt;any other serialization format&gt;_?
-
-We know there might be strong opinions on this one. Regardless, for Caddy 2, we've decided to go with JSON. If that proves to be a fatal mistake, then Caddy 3 probably won't use JSON.
-
-JSON may not be the fastest, the most compact, the easiest to write, serialization format that exists. But those aren't our goals. It has withstood the test of time and checks all our boxes.
-
-- It is almost entirely ubiquitous. JSON works natively in web browsers and has mature libraries in pretty much every language.
-- It is human-readable (as opposed to a binary format).
-- It is easy to tweak by hand. Although composing raw JSON by hand is not awesome, this will not be mainstream once our config adapters are done.
-- It is generally easy to convert other serializations or config formats into JSON, as opposed to the other way around.
-- Even though JSON deserialization is not fast per-se, that kind of performance is not really a concern since config reloads are not the server's hottest path like HTTP request handling or TLS handshakes are. Even with JSON, Caddy 2 can handle dozens of config changes per second, which is probably plenty for now.
-- It maps almost 1:1 to the actual, in-memory values that power your HTTP handlers and other parts of the server (no need to parse a config file with some arbitrary DSL and do a bunch of extra pre-processing).
-
-Ultimately, we think all these properties are appropriate -- if not ideal -- for a web server configuration.
-
-If you're still not happy with the choice of JSON, feel free to contribute a config adapter of your own choice!
-
-Or just use YAML or TOML, which seamlessly translate to JSON.
-
-### JSON is declarative; what if I need more programmability (i.e. imperative syntax)?
-
-NGINX also realized the need for imperative logic in declarative configs, so they tried "if" statements, [but it was a bad idea](https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/).
-
-We have good news. Caddy 2 can give you the power of imperative logic without the perils of mixing declarative and imperative config such as befell NGINX. We do this by allowing embedded imperative syntax awithin the Caddy's declarative config.
-
-Caddy 2's configuration is declarative because configuration is very much declarative in nature. Configuration is a tricky medium, as it is read and written by both computers and humans. Computers use it, but humans constantly refer to it and update it. Declarative syntaxes are fairly straightforward to make sense of, whereas it is difficult to reason about imperative logic.
-
-However, sometimes computation is useful, and in some cases, the only way to express what you need. This can be illustrated really well in the simple case of trying to decide whether a particular HTTP middleware should be invoked as part of an HTTP request. A lot of the time, such logic is as simple as: "GET requests for any path starting with /foo/bar", which can be expressed declaratively in JSON:
-
-```json
-{
-	"method": "GET",
-	"path": "/foo/bar"
-}
-```
-
-But what if you need to match /foo/bar OR /topaz? How do you express that OR clause? Maybe an array:
-
-```json
-{
-	"method": ["GET"],
-	"path": ["/foo/bar", "/topaz"]
-}
-```
-
-Now what if you need add a NOT or AND clause? JSON quickly tires out. As you learn about Caddy 2's request matching, you will see how we handled this. Caddy 2's JSON gives you the ability to express moderately-complex logic such as:
-
-```js
-// this is not actual Caddy config, just logic pseudocode
-IF (Host = "example.com")
-	OR (Host = "sub.example.com" AND Path != "/foo/bar")
-```
-
-Already, this is more expressive power than most web servers offer with their native config, yet Caddy 2 offers this in JSON.
-
-But in most web servers, to make logic this complex feasible, you'll generally call out to Lua or some extra DSL. For example, in NGINX you could use a Lua module to express this logic. Traefik 2.0 has [yet another kind of clunky-looking custom DSL](https://blog.containo.us/back-to-traefik-2-0-2f9aa17be305#d22e) just for this.
-
-Caddy 2 solves this in a novel way with [Starlark expressions](https://godoc.org/go.starlark.net/starlark#Eval). Starlark is a familiar dialect of Python! So, no new DSLs to learn and no VMs to slow things down:
-
-```python
-req.host == 'example.com' ||
-	(req.host == 'sub.example.com' && req.path != '/foo/bar')
-```
-
-Starlark performs at least as well as NGINX+Lua (more performance tests ongoing, as well as optimizations to make it even faster!) and because it's basically Python, it's familiar and easy to use.
-
-In summary: Caddy 2 config is declarative, but can be imperative where that is useful.
-
-### What is Caddy 2 licensed as?
-
-Caddy 2 is licensed under the Apache 2.0 open source license. There are no official Caddy 2 distributions that are proprietary.
-
-### Does Caddy 2 have telemetry?
-
-No. There was not enough academic interest to continue supporting it. If telemetry does get added later, it will not be on by default or will be vastly reduced in its scope.
-
-## Does Caddy 2 use HTTPS by default?
-
-Yes. HTTPS is automatic and enabled by default when possible, just like in Caddy 1. Basically, if your HTTP routes specify a `host` matcher with qualifying domain names, those names will be enabled for automatic HTTPS. Automatic HTTPS is disabled for domains which match certificates that are manually loaded by your config.
-
-## How do I avoid Let's Encrypt rate limits with Caddy 2?
-
-As you are testing and developing with Caddy 2, you should use test ("staging") certificates from Let's Encrypt to avoid rate limits. By default, Caddy 2 uses Let's Encrypt's production endpoint to get real certificates for your domains, but their [rate limits](https://letsencrypt.org/docs/rate-limits/) forbid testing and development use of this endpoint for good reasons. You can switch to their [staging endpoint](https://letsencrypt.org/docs/staging-environment/) by adding the staging CA to your automation policy in the `tls` app:
-
-```json
-"tls": {
-	"automation": {
-		"policies": [
-			{
-				"management": {
-					"module": "acme",
-					"ca": "https://acme-staging-v02.api.letsencrypt.org/directory"
-				}
-			}
-		]
-	}
-}
-```
-
-Or with the Caddyfile, using a global options block at the top:
+Using [our builder tool, `xcaddy`](https://github.com/caddyserver/xcaddy)...
 
 ```
-{
-	acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
-}
+$ xcaddy build
 ```
 
-## Can we get some access controls on the admin endpoint?
+...the following steps are automated:
 
-Yeah, that's coming. For now, you can use a permissioned unix socket for some basic security.
+1. Create a new folder: `mkdir caddy`
+2. Change into it: `cd caddy`
+3. Copy [Caddy's main.go](https://github.com/caddyserver/caddy/blob/master/cmd/caddy/main.go) into the empty folder. Add imports for any custom plugins you want to add.
+4. Initialize a Go module: `go mod init caddy`
+5. (Optional) Pin Caddy version: `go get github.com/caddyserver/caddy/v2@version` replacing `version` with a git tag or commit.
+6. (Optional) Add plugins by adding their import: `_ "import/path/here"`
+7. Compile: `go build`
+
+
+
+
+## Quick start
+
+The [Caddy website](https://caddyserver.com/docs/) has documentation that includes tutorials, quick-start guides, reference, and more.
+
+**We recommend that all users do our [Getting Started](https://caddyserver.com/docs/getting-started) guide to become familiar with using Caddy.**
+
+If you've only got a minute, [the website has several quick-start tutorials](https://caddyserver.com/docs/quick-starts) to choose from! However, after finishing a quick-start tutorial, please read more documentation to understand how the software works. 🙂
+
+
+
+
+## Overview
+
+Caddy is most often used as an HTTPS server, but it is suitable for any long-running Go program. First and foremost, it is a platform to run Go applications. Caddy "apps" are just Go programs that are implemented as Caddy modules. Two apps -- `tls` and `http` -- ship standard with Caddy.
+
+Caddy apps instantly benefit from [automated documentation](https://caddyserver.com/docs/json/), graceful on-line [config changes via API](https://caddyserver.com/docs/api), and unification with other Caddy apps.
+
+Although [JSON](https://caddyserver.com/docs/json/) is Caddy's native config language, Caddy can accept input from [config adapters](https://caddyserver.com/docs/config-adapters) which can essentially convert any config format of your choice into JSON: Caddyfile, JSON 5, YAML, TOML, NGINX config, and more.
+
+The primary way to configure Caddy is through [its API](https://caddyserver.com/docs/api), but if you prefer config files, the [command-line interface](https://caddyserver.com/docs/command-line) supports those too.
+
+Caddy exposes an unprecedented level of control compared to any web server in existence. In Caddy, you are usually setting the actual values of the initialized types in memory that power everything from your HTTP handlers and TLS handshakes to your storage medium. Caddy is also ridiculously extensible, with a powerful plugin system that makes vast improvements over other web servers.
+
+To wield the power of this design, you need to know how the config document is structured. Please see [our documentation site](https://caddyserver.com/docs/) for details about [Caddy's config structure](https://caddyserver.com/docs/json/).
+
+Nearly all of Caddy's configuration is contained in a single config document, rather than being scattered across CLI flags and env variables and a configuration file as with other web servers. This makes managing your server config more straightforward and reduces hidden variables/factors.
+
+
+## Full documentation
+
+Our website has complete documentation:
+
+**https://caddyserver.com/docs/**
+
+The docs are also open source. You can contribute to them here: https://github.com/caddyserver/website
+
+
+
+## Getting help
+
+- We **strongly recommend** that all professionals or companies using Caddy get a support contract through [Ardan Labs](https://www.ardanlabs.com/my/contact-us?dd=caddy) before help is needed.
+
+- Individuals can exchange help for free on our community forum at https://caddy.community. Remember that people give help out of their spare time and good will. The best way to get help is to give it first!
+
+Please use our [issue tracker](https://github.com/caddyserver/caddy/issues) only for bug reports and feature requests, i.e. actionable development items (support questions will usually be referred to the forums).
+
+
+
+## About
+
+**The name "Caddy" is trademarked.** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". Caddy is a registered trademark of Light Code Labs, LLC.
+
+- _Project on Twitter: [@caddyserver](https://twitter.com/caddyserver)_
+- _Author on Twitter: [@mholt6](https://twitter.com/mholt6)_

admin_test.go

@@ -15,14 +15,102 @@
 package caddy
 
 import (
-	"strings"
+	"encoding/json"
+	"reflect"
 	"testing"
 )
 
+func TestUnsyncedConfigAccess(t *testing.T) {
+	// each test is performed in sequence, so
+	// each change builds on the previous ones;
+	// the config is not reset between tests
+	for i, tc := range []struct {
+		method    string
+		path      string // rawConfigKey will be prepended
+		payload   string
+		expect    string // JSON representation of what the whole config is expected to be after the request
+		shouldErr bool
+	}{
+		{
+			method:  "POST",
+			path:    "",
+			payload: `{"foo": "bar", "list": ["a", "b", "c"]}`, // starting value
+			expect:  `{"foo": "bar", "list": ["a", "b", "c"]}`,
+		},
+		{
+			method:  "POST",
+			path:    "/foo",
+			payload: `"jet"`,
+			expect:  `{"foo": "jet", "list": ["a", "b", "c"]}`,
+		},
+		{
+			method:  "POST",
+			path:    "/bar",
+			payload: `{"aa": "bb", "qq": "zz"}`,
+			expect:  `{"foo": "jet", "bar": {"aa": "bb", "qq": "zz"}, "list": ["a", "b", "c"]}`,
+		},
+		{
+			method: "DELETE",
+			path:   "/bar/qq",
+			expect: `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c"]}`,
+		},
+		{
+			method:  "POST",
+			path:    "/list",
+			payload: `"e"`,
+			expect:  `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "e"]}`,
+		},
+		{
+			method:  "PUT",
+			path:    "/list/3",
+			payload: `"d"`,
+			expect:  `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "d", "e"]}`,
+		},
+		{
+			method: "DELETE",
+			path:   "/list/3",
+			expect: `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "e"]}`,
+		},
+		{
+			method:  "PATCH",
+			path:    "/list/3",
+			payload: `"d"`,
+			expect:  `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "d"]}`,
+		},
+		{
+			method:  "POST",
+			path:    "/list/...",
+			payload: `["e", "f", "g"]`,
+			expect:  `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "d", "e", "f", "g"]}`,
+		},
+	} {
+		err := unsyncedConfigAccess(tc.method, rawConfigKey+tc.path, []byte(tc.payload), nil)
+
+		if tc.shouldErr && err == nil {
+			t.Fatalf("Test %d: Expected error return value, but got: %v", i, err)
+		}
+		if !tc.shouldErr && err != nil {
+			t.Fatalf("Test %d: Should not have had error return value, but got: %v", i, err)
+		}
+
+		// decode the expected config so we can do a convenient DeepEqual
+		var expectedDecoded interface{}
+		err = json.Unmarshal([]byte(tc.expect), &expectedDecoded)
+		if err != nil {
+			t.Fatalf("Test %d: Unmarshaling expected config: %v", i, err)
+		}
+
+		// make sure the resulting config is as we expect it
+		if !reflect.DeepEqual(rawCfg[rawConfigKey], expectedDecoded) {
+			t.Fatalf("Test %d:\nExpected:\n\t%#v\nActual:\n\t%#v",
+				i, expectedDecoded, rawCfg[rawConfigKey])
+		}
+	}
+}
+
 func BenchmarkLoad(b *testing.B) {
 	for i := 0; i < b.N; i++ {
-		r := strings.NewReader(`{
-			"testval": "Yippee!",
+		cfg := []byte(`{
 			"apps": {
 				"http": {
 					"servers": {
@@ -39,6 +127,6 @@ func BenchmarkLoad(b *testing.B) {
 			}
 		}
 		`)
-		Load(r)
+		Load(cfg, true)
 	}
 }

azure-pipelines.yml

@@ -1,243 +0,0 @@
-# Mutilated beyond recognition from the example at:
-# https://docs.microsoft.com/azure/devops/pipelines/languages/go
-
-trigger:
-  - v2
-
-schedules:
-- cron: "0 0 * * *"
-  displayName: Daily midnight fuzzing
-  branches:
-    include:
-    - v2
-  always: true
-
-variables:
-  GOROOT: $(gorootDir)/go
-  GOPATH: $(system.defaultWorkingDirectory)/gopath
-  GOBIN:  $(GOPATH)/bin
-  modulePath: '$(GOPATH)/src/github.com/$(build.repository.name)'
-  # TODO: Remove once it's enabled by default
-  GO111MODULE: on
-
-jobs:
-- job: crossPlatformTest
-  displayName: "Cross-Platform Tests"
-  strategy:
-    matrix:
-      linux:
-        imageName: ubuntu-16.04
-        gorootDir: /usr/local
-      mac:
-        imageName: macos-10.13
-        gorootDir: /usr/local
-      windows:
-        imageName: windows-2019
-        gorootDir: C:\
-  pool:
-    vmImage: $(imageName)
-  
-  steps:
-  - bash: |
-      latestGo=$(curl "https://golang.org/VERSION?m=text")
-      echo "##vso[task.setvariable variable=LATEST_GO]$latestGo"
-      echo "Latest Go version: $latestGo"
-    displayName: "Get latest Go version"
-
-  - bash: |
-      sudo rm -f $(which go)
-      echo '##vso[task.prependpath]$(GOBIN)'
-      echo '##vso[task.prependpath]$(GOROOT)/bin'
-      mkdir -p '$(modulePath)'
-      shopt -s extglob
-      shopt -s dotglob
-      mv !(gopath) '$(modulePath)'
-    displayName: Remove old Go, set GOBIN/GOROOT, and move project into GOPATH
-
-  # Install Go (this varies by platform)
-  - bash: |
-      wget "https://dl.google.com/go/$(LATEST_GO).linux-amd64.tar.gz"
-      sudo tar -C $(gorootDir) -xzf "$(LATEST_GO).linux-amd64.tar.gz"
-    condition: eq( variables['Agent.OS'], 'Linux' )
-    displayName: Install Go on Linux
-
-  - bash: |
-      wget "https://dl.google.com/go/$(LATEST_GO).darwin-amd64.tar.gz"
-      sudo tar -C $(gorootDir) -xzf "$(LATEST_GO).darwin-amd64.tar.gz"
-    condition: eq( variables['Agent.OS'], 'Darwin' )
-    displayName: Install Go on macOS
-
-  # The low performance is partly due to PowerShell's attempt to update the progress bar. Disabling it speeds up the process.
-  # Reference: https://github.com/PowerShell/PowerShell/issues/2138
-  - powershell: |
-      $ProgressPreference = 'SilentlyContinue'
-      Write-Host "Downloading Go..."
-      (New-Object System.Net.WebClient).DownloadFile("https://dl.google.com/go/$(LATEST_GO).windows-amd64.zip", "$(LATEST_GO).windows-amd64.zip")
-      Write-Host "Extracting Go... (I'm slow too)"
-      7z x "$(LATEST_GO).windows-amd64.zip" -o"$(gorootDir)"
-    condition: eq( variables['Agent.OS'], 'Windows_NT' )
-    displayName: Install Go on Windows
-
-  - bash: curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.19.1
-    displayName: Install golangci-lint
-
-  - script: |
-      go get github.com/axw/gocov/gocov
-      go get github.com/AlekSi/gocov-xml
-      go get -u github.com/jstemmer/go-junit-report
-    displayName: Install test and coverage analysis tools
-
-  - bash: |
-      printf "Using go at: $(which go)\n"
-      printf "Go version: $(go version)\n"
-      printf "\n\nGo environment:\n\n"
-      go env
-      printf "\n\nSystem environment:\n\n"
-      env
-    displayName: Print Go version and environment
-
-  - script: |
-      go get -v -t -d ./...
-      mkdir test-results
-    workingDirectory: '$(modulePath)'
-    displayName: Get dependencies
-
-  # its behavior is governed by .golangci.yml
-  - script: |
-      (golangci-lint run --out-format junit-xml) > test-results/lint-result.xml
-      exit 0
-    workingDirectory: '$(modulePath)'
-    continueOnError: true
-    displayName: Run lint check
-
-  - script: |
-      (go test -v -coverprofile=cover-profile.out -race ./... 2>&1) > test-results/test-result.out
-    workingDirectory: '$(modulePath)'
-    continueOnError: true
-    displayName: Run tests
-
-  - script: |
-      mkdir coverage
-      gocov convert cover-profile.out > coverage/coverage.json
-      # Because Windows doesn't work with input redirection like *nix, but output redirection works.
-      (cat ./coverage/coverage.json | gocov-xml) > coverage/coverage.xml 
-    workingDirectory: '$(modulePath)'
-    displayName: Prepare coverage reports
-
-  - script: |
-      (cat ./test-results/test-result.out | go-junit-report) > test-results/test-result.xml
-    workingDirectory: '$(modulePath)'
-    displayName: Prepare test report
-
-  - task: PublishCodeCoverageResults@1
-    displayName: Publish test coverage report
-    inputs:
-      codeCoverageTool: Cobertura 
-      summaryFileLocation: $(modulePath)/coverage/coverage.xml
-
-  - task: PublishTestResults@2
-    displayName: Publish unit test
-    inputs:
-      testResultsFormat: 'JUnit'
-      testResultsFiles: $(modulePath)/test-results/test-result.xml
-      testRunTitle: $(agent.OS) Unit Test
-      mergeTestResults: false
-
-  - task: PublishTestResults@2
-    displayName: Publish lint results
-    inputs:
-      testResultsFormat: 'JUnit'
-      testResultsFiles: $(modulePath)/test-results/lint-result.xml
-      testRunTitle: $(agent.OS) Lint
-      mergeTestResults: false
-
-  - bash: |
-      exit 1
-    condition: eq(variables['Agent.JobStatus'], 'SucceededWithIssues')
-    displayName: Coerce correct build result
-
-- job: fuzzing
-  displayName: 'Fuzzing'
-  # Only run this job on schedules or PRs for non-forks.
-  condition: or(eq(variables['System.PullRequest.IsFork'], 'False'), eq(variables['Build.Reason'], 'Schedule') )
-  strategy:
-    matrix:
-      linux:
-        imageName: ubuntu-16.04
-        gorootDir: /usr/local
-  pool:
-    vmImage: $(imageName)
-
-  steps:
-  - bash: |
-      latestGo=$(curl "https://golang.org/VERSION?m=text")
-      echo "##vso[task.setvariable variable=LATEST_GO]$latestGo"
-      echo "Latest Go version: $latestGo"
-    displayName: "Get latest Go version"
-
-  - bash: |
-      sudo rm -f $(which go)
-      echo '##vso[task.prependpath]$(GOBIN)'
-      echo '##vso[task.prependpath]$(GOROOT)/bin'
-      mkdir -p '$(modulePath)'
-      shopt -s extglob
-      shopt -s dotglob
-      mv !(gopath) '$(modulePath)'
-    displayName: Remove old Go, set GOBIN/GOROOT, and move project into GOPATH
-
-  - bash: |
-      wget "https://dl.google.com/go/$(LATEST_GO).linux-amd64.tar.gz"
-      sudo tar -C $(gorootDir) -xzf "$(LATEST_GO).linux-amd64.tar.gz"
-    condition: eq( variables['Agent.OS'], 'Linux' )
-    displayName: Install Go on Linux
-
-  - bash: |
-      # Install Clang-7.0 because other versions seem to be missing the file libclang_rt.fuzzer-x86_64.a
-      sudo add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-7 main"
-      wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
-      sudo apt update && sudo apt install -y clang-7 lldb-7 lld-7
-
-      go get -v github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build
-      wget -q -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v2.4.74/fuzzit_Linux_x86_64
-      chmod a+x fuzzit
-      mv fuzzit $(GOBIN)
-    displayName: Download go-fuzz tools and the Fuzzit CLI, and move Fuzzit CLI to GOBIN
-    condition: and(eq(variables['System.PullRequest.IsFork'], 'False') , eq( variables['Agent.OS'], 'Linux' ))
-
-  - bash: |
-      declare -A fuzzers_funcs=(\
-        ["./admin_fuzz.go"]="FuzzAdmin" \
-        ["./caddyconfig/httpcaddyfile/adapter_fuzz.go"]="FuzzHTTPCaddyfileAdapter" \
-        ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="FuzzParseAddress" \
-        ["./caddyconfig/caddyfile/parse_fuzz.go"]="FuzzParseCaddyfile" \
-        ["./listeners_fuzz.go"]="FuzzParseNetworkAddress" \
-        ["./replacer_fuzz.go"]="FuzzReplacer" \
-      )
-
-      declare -A fuzzers_targets=(\
-        ["./admin_fuzz.go"]="admin" \
-        ["./caddyconfig/httpcaddyfile/adapter_fuzz.go"]="caddyfile-adapter" \
-        ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="parse-address" \
-        ["./caddyconfig/caddyfile/parse_fuzz.go"]="parse-caddyfile" \
-        ["./listeners_fuzz.go"]="parse-listen-addr" \
-        ["./replacer_fuzz.go"]="replacer" \
-      )
-
-      fuzz_type="regression"
-      if [[ $(Build.Reason) == "Schedule" ]]; then
-        fuzz_type="fuzzing"
-      fi
-
-      for f in $(find . -name \*_fuzz.go); do
-        FUZZER_DIRECTORY=$(dirname $f)
-        echo "go-fuzz-build func ${fuzzers_funcs[$f]} residing in $f"
-        go-fuzz-build -func "${fuzzers_funcs[$f]}" -libfuzzer -o "$FUZZER_DIRECTORY/${fuzzers_targets[$f]}.a" $FUZZER_DIRECTORY
-        echo "Generating fuzzer binary of func ${fuzzers_funcs[$f]} which resides in $f"
-        clang-7 -fsanitize=fuzzer "$FUZZER_DIRECTORY/${fuzzers_targets[$f]}.a" -o "$FUZZER_DIRECTORY/${fuzzers_targets[$f]}"
-        fuzzit create job caddyserver/${fuzzers_targets[$f]} $FUZZER_DIRECTORY/${fuzzers_targets[$f]} --api-key ${FUZZIT_API_KEY} --type "${fuzz_type}" --branch "${SYSTEM_PULLREQUEST_SOURCEBRANCH}" --revision "${BUILD_SOURCEVERSION}"
-        echo "Completed $f"
-      done
-    env:
-      FUZZIT_API_KEY: $(FUZZIT_API_KEY)
-    workingDirectory: '$(modulePath)'
-    displayName: Generate fuzzers & submit them to Fuzzit

caddy.go

@@ -15,24 +15,64 @@
 package caddy
 
 import (
+	"bytes"
 	"context"
 	"encoding/json"
 	"fmt"
+	"io"
+	"io/ioutil"
 	"log"
+	"net/http"
+	"os"
+	"path"
+	"path/filepath"
 	"runtime/debug"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
 
-	"github.com/mholt/certmagic"
+	"github.com/caddyserver/certmagic"
+	"go.uber.org/zap"
 )
 
-// Config represents a Caddy configuration.
+// Config is the top (or beginning) of the Caddy configuration structure.
+// Caddy config is expressed natively as a JSON document. If you prefer
+// not to work with JSON directly, there are [many config adapters](/docs/config-adapters)
+// available that can convert various inputs into Caddy JSON.
+//
+// Many parts of this config are extensible through the use of Caddy modules.
+// Fields which have a json.RawMessage type and which appear as dots (•••) in
+// the online docs can be fulfilled by modules in a certain module
+// namespace. The docs show which modules can be used in a given place.
+//
+// Whenever a module is used, its name must be given either inline as part of
+// the module, or as the key to the module's value. The docs will make it clear
+// which to use.
+//
+// Generally, all config settings are optional, as it is Caddy convention to
+// have good, documented default values. If a parameter is required, the docs
+// should say so.
+//
+// Go programs which are directly building a Config struct value should take
+// care to populate the JSON-encodable fields of the struct (i.e. the fields
+// with `json` struct tags) if employing the module lifecycle (e.g. Provision
+// method calls).
 type Config struct {
-	Admin      *AdminConfig               `json:"admin,omitempty"`
-	Logging    *Logging                   `json:"logging,omitempty"`
-	StorageRaw json.RawMessage            `json:"storage,omitempty"`
-	AppsRaw    map[string]json.RawMessage `json:"apps,omitempty"`
+	Admin   *AdminConfig `json:"admin,omitempty"`
+	Logging *Logging     `json:"logging,omitempty"`
+
+	// StorageRaw is a storage module that defines how/where Caddy
+	// stores assets (such as TLS certificates). The default storage
+	// module is `caddy.storage.file_system` (the local file system),
+	// and the default path
+	// [depends on the OS and environment](/docs/conventions#data-directory).
+	StorageRaw json.RawMessage `json:"storage,omitempty" caddy:"namespace=caddy.storage inline_key=module"`
+
+	// AppsRaw are the apps that Caddy will load and run. The
+	// app module name is the key, and the app's config is the
+	// associated value.
+	AppsRaw ModuleMap `json:"apps,omitempty" caddy:"namespace="`
 
 	apps    map[string]App
 	storage certmagic.Storage
@@ -46,13 +86,167 @@ type App interface {
 	Stop() error
 }
 
-// Run runs Caddy with the given config.
-func Run(newCfg *Config) error {
+// Run runs the given config, replacing any existing config.
+func Run(cfg *Config) error {
+	cfgJSON, err := json.Marshal(cfg)
+	if err != nil {
+		return err
+	}
+	return Load(cfgJSON, true)
+}
+
+// Load loads the given config JSON and runs it only
+// if it is different from the current config or
+// forceReload is true.
+func Load(cfgJSON []byte, forceReload bool) error {
+	return changeConfig(http.MethodPost, "/"+rawConfigKey, cfgJSON, forceReload)
+}
+
+// changeConfig changes the current config (rawCfg) according to the
+// method, traversed via the given path, and uses the given input as
+// the new value (if applicable; i.e. "DELETE" doesn't have an input).
+// If the resulting config is the same as the previous, no reload will
+// occur unless forceReload is true. This function is safe for
+// concurrent use.
+func changeConfig(method, path string, input []byte, forceReload bool) error {
+	switch method {
+	case http.MethodGet,
+		http.MethodHead,
+		http.MethodOptions,
+		http.MethodConnect,
+		http.MethodTrace:
+		return fmt.Errorf("method not allowed")
+	}
+
 	currentCfgMu.Lock()
 	defer currentCfgMu.Unlock()
 
+	err := unsyncedConfigAccess(method, path, input, nil)
+	if err != nil {
+		return err
+	}
+
+	// the mutation is complete, so encode the entire config as JSON
+	newCfg, err := json.Marshal(rawCfg[rawConfigKey])
+	if err != nil {
+		return APIError{
+			Code: http.StatusBadRequest,
+			Err:  fmt.Errorf("encoding new config: %v", err),
+		}
+	}
+
+	// if nothing changed, no need to do a whole reload unless the client forces it
+	if !forceReload && bytes.Equal(rawCfgJSON, newCfg) {
+		Log().Named("admin.api").Info("config is unchanged")
+		return nil
+	}
+
+	// find any IDs in this config and index them
+	idx := make(map[string]string)
+	err = indexConfigObjects(rawCfg[rawConfigKey], "/"+rawConfigKey, idx)
+	if err != nil {
+		return APIError{
+			Code: http.StatusInternalServerError,
+			Err:  fmt.Errorf("indexing config: %v", err),
+		}
+	}
+
+	// load this new config; if it fails, we need to revert to
+	// our old representation of caddy's actual config
+	err = unsyncedDecodeAndRun(newCfg)
+	if err != nil {
+		if len(rawCfgJSON) > 0 {
+			// restore old config state to keep it consistent
+			// with what caddy is still running; we need to
+			// unmarshal it again because it's likely that
+			// pointers deep in our rawCfg map were modified
+			var oldCfg interface{}
+			err2 := json.Unmarshal(rawCfgJSON, &oldCfg)
+			if err2 != nil {
+				err = fmt.Errorf("%v; additionally, restoring old config: %v", err, err2)
+			}
+			rawCfg[rawConfigKey] = oldCfg
+		}
+
+		return fmt.Errorf("loading new config: %v", err)
+	}
+
+	// success, so update our stored copy of the encoded
+	// config to keep it consistent with what caddy is now
+	// running (storing an encoded copy is not strictly
+	// necessary, but avoids an extra json.Marshal for
+	// each config change)
+	rawCfgJSON = newCfg
+	rawCfgIndex = idx
+
+	return nil
+}
+
+// readConfig traverses the current config to path
+// and writes its JSON encoding to out.
+func readConfig(path string, out io.Writer) error {
+	currentCfgMu.RLock()
+	defer currentCfgMu.RUnlock()
+	return unsyncedConfigAccess(http.MethodGet, path, nil, out)
+}
+
+// indexConfigObjects recursively searches ptr for object fields named
+// "@id" and maps that ID value to the full configPath in the index.
+// This function is NOT safe for concurrent access; obtain a write lock
+// on currentCfgMu.
+func indexConfigObjects(ptr interface{}, configPath string, index map[string]string) error {
+	switch val := ptr.(type) {
+	case map[string]interface{}:
+		for k, v := range val {
+			if k == idKey {
+				switch idVal := v.(type) {
+				case string:
+					index[idVal] = configPath
+				case float64: // all JSON numbers decode as float64
+					index[fmt.Sprintf("%v", idVal)] = configPath
+				default:
+					return fmt.Errorf("%s: %s field must be a string or number", configPath, idKey)
+				}
+				continue
+			}
+			// traverse this object property recursively
+			err := indexConfigObjects(val[k], path.Join(configPath, k), index)
+			if err != nil {
+				return err
+			}
+		}
+	case []interface{}:
+		// traverse each element of the array recursively
+		for i := range val {
+			err := indexConfigObjects(val[i], path.Join(configPath, strconv.Itoa(i)), index)
+			if err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}
+
+// unsyncedDecodeAndRun removes any meta fields (like @id tags)
+// from cfgJSON, decodes the result into a *Config, and runs
+// it as the new config, replacing any other current config.
+// It does NOT update the raw config state, as this is a
+// lower-level function; most callers will want to use Load
+// instead. A write lock on currentCfgMu is required!
+func unsyncedDecodeAndRun(cfgJSON []byte) error {
+	// remove any @id fields from the JSON, which would cause
+	// loading to break since the field wouldn't be recognized
+	strippedCfgJSON := RemoveMetaFields(cfgJSON)
+
+	var newCfg *Config
+	err := strictUnmarshalJSON(strippedCfgJSON, &newCfg)
+	if err != nil {
+		return err
+	}
+
 	// run the new config and start all its apps
-	err := run(newCfg, true)
+	err = run(newCfg, true)
 	if err != nil {
 		return err
 	}
@@ -64,6 +258,30 @@ func Run(newCfg *Config) error {
 	// Stop, Cleanup each old app
 	unsyncedStop(oldCfg)
 
+	// autosave a non-nil config, if not disabled
+	if newCfg != nil &&
+		(newCfg.Admin == nil ||
+			newCfg.Admin.Config == nil ||
+			newCfg.Admin.Config.Persist == nil ||
+			*newCfg.Admin.Config.Persist) {
+		dir := filepath.Dir(ConfigAutosavePath)
+		err := os.MkdirAll(dir, 0700)
+		if err != nil {
+			Log().Error("unable to create folder for config autosave",
+				zap.String("dir", dir),
+				zap.Error(err))
+		} else {
+			err := ioutil.WriteFile(ConfigAutosavePath, cfgJSON, 0600)
+			if err == nil {
+				Log().Info("autosaved config", zap.String("file", ConfigAutosavePath))
+			} else {
+				Log().Error("unable to autosave config",
+					zap.String("file", ConfigAutosavePath),
+					zap.Error(err))
+			}
+		}
+	}
+
 	return nil
 }
 
@@ -77,11 +295,11 @@ func Run(newCfg *Config) error {
 // the config if you are not going to start it,
 // so that each provisioned module will be
 // cleaned up.
+//
+// This is a low-level function; most callers
+// will want to use Run instead, which also
+// updates the config's raw state.
 func run(newCfg *Config, start bool) error {
-	if newCfg == nil {
-		return nil
-	}
-
 	// because we will need to roll back any state
 	// modifications if this function errors, we
 	// keep a single error value and scope all
@@ -91,6 +309,18 @@ func run(newCfg *Config, start bool) error {
 	// been set by a short assignment
 	var err error
 
+	// start the admin endpoint (and stop any prior one)
+	if start {
+		err = replaceAdmin(newCfg)
+		if err != nil {
+			return fmt.Errorf("starting caddy administration endpoint: %v", err)
+		}
+	}
+
+	if newCfg == nil {
+		return nil
+	}
+
 	// prepare the new config for use
 	newCfg.apps = make(map[string]App)
 
@@ -130,7 +360,7 @@ func run(newCfg *Config, start bool) error {
 	// set up global storage and make it CertMagic's default storage, too
 	err = func() error {
 		if newCfg.StorageRaw != nil {
-			val, err := ctx.LoadModuleInline("module", "caddy.storage", newCfg.StorageRaw)
+			val, err := ctx.LoadModule(newCfg, "StorageRaw")
 			if err != nil {
 				return fmt.Errorf("loading storage module: %v", err)
 			}
@@ -139,10 +369,10 @@ func run(newCfg *Config, start bool) error {
 				return fmt.Errorf("creating storage value: %v", err)
 			}
 			newCfg.storage = stor
-			newCfg.StorageRaw = nil // allow GC to deallocate
 		}
+
 		if newCfg.storage == nil {
-			newCfg.storage = &certmagic.FileStorage{Path: dataDir()}
+			newCfg.storage = DefaultStorage
 		}
 		certmagic.Default.Storage = newCfg.storage
 
@@ -152,14 +382,12 @@ func run(newCfg *Config, start bool) error {
 		return err
 	}
 
-	// Load, Provision, Validate each app and their submodules
+	// Load and Provision each app and their submodules
 	err = func() error {
-		for modName, rawMsg := range newCfg.AppsRaw {
-			val, err := ctx.LoadModule(modName, rawMsg)
-			if err != nil {
-				return fmt.Errorf("loading app module '%s': %v", modName, err)
+		for appName := range newCfg.AppsRaw {
+			if _, err := ctx.App(appName); err != nil {
+				return err
 			}
-			newCfg.apps[modName] = val.(App)
 		}
 		return nil
 	}()
@@ -198,22 +426,25 @@ func run(newCfg *Config, start bool) error {
 // It is the antithesis of Run(). This function
 // will log any errors that occur during the
 // stopping of individual apps and continue to
-// stop the others.
+// stop the others. Stop should only be called
+// if not replacing with a new config.
 func Stop() error {
 	currentCfgMu.Lock()
 	defer currentCfgMu.Unlock()
 	unsyncedStop(currentCfg)
 	currentCfg = nil
+	rawCfgJSON = nil
+	rawCfgIndex = nil
+	rawCfg[rawConfigKey] = nil
 	return nil
 }
 
-// unsyncedStop stops cfg from running, but if
-// applicable, you need to acquire locks yourself.
-// It is a no-op if cfg is nil. If any app
-// returns an error when stopping, it is logged
-// and the function continues with the next app.
-// This function assumes all apps in cfg were
-// successfully started.
+// unsyncedStop stops cfg from running, but has
+// no locking around cfg. It is a no-op if cfg is
+// nil. If any app returns an error when stopping,
+// it is logged and the function continues stopping
+// the next app. This function assumes all apps in
+// cfg were successfully started first.
 func unsyncedStop(cfg *Config) {
 	if cfg == nil {
 		return
@@ -231,6 +462,20 @@ func unsyncedStop(cfg *Config) {
 	cfg.cancelFunc()
 }
 
+// stopAndCleanup calls stop and cleans up anything
+// else that is expedient. This should only be used
+// when stopping and not replacing with a new config.
+func stopAndCleanup() error {
+	if err := Stop(); err != nil {
+		return err
+	}
+	certmagic.CleanUpOwnLocks()
+	if pidfile != "" {
+		return os.Remove(pidfile)
+	}
+	return nil
+}
+
 // Validate loads, provisions, and validates
 // cfg, but does not start running it.
 func Validate(cfg *Config) error {
@@ -241,17 +486,54 @@ func Validate(cfg *Config) error {
 	return err
 }
 
-// Duration is a JSON-string-unmarshable duration type.
+// Duration can be an integer or a string. An integer is
+// interpreted as nanoseconds. If a string, it is a Go
+// time.Duration value such as `300ms`, `1.5h`, or `2h45m`;
+// valid units are `ns`, `us`/`µs`, `ms`, `s`, `m`, `h`, and `d`.
 type Duration time.Duration
 
 // UnmarshalJSON satisfies json.Unmarshaler.
 func (d *Duration) UnmarshalJSON(b []byte) error {
-	dd, err := time.ParseDuration(strings.Trim(string(b), `"`))
-	if err != nil {
-		return err
+	if len(b) == 0 {
+		return io.EOF
 	}
-	*d = Duration(dd)
-	return nil
+	var dur time.Duration
+	var err error
+	if b[0] == byte('"') && b[len(b)-1] == byte('"') {
+		dur, err = ParseDuration(strings.Trim(string(b), `"`))
+	} else {
+		err = json.Unmarshal(b, &dur)
+	}
+	*d = Duration(dur)
+	return err
+}
+
+// ParseDuration parses a duration string, adding
+// support for the "d" unit meaning number of days,
+// where a day is assumed to be 24h.
+func ParseDuration(s string) (time.Duration, error) {
+	var inNumber bool
+	var numStart int
+	for i := 0; i < len(s); i++ {
+		ch := s[i]
+		if ch == 'd' {
+			daysStr := s[numStart:i]
+			days, err := strconv.ParseFloat(daysStr, 64)
+			if err != nil {
+				return 0, err
+			}
+			hours := days * 24.0
+			hoursStr := strconv.FormatFloat(hours, 'f', -1, 64)
+			s = s[:numStart] + hoursStr + "h" + s[i+1:]
+			i--
+			continue
+		}
+		if !inNumber {
+			numStart = i
+		}
+		inNumber = (ch >= '0' && ch <= '9') || ch == '.' || ch == '-' || ch == '+'
+	}
+	return time.ParseDuration(s)
 }
 
 // GoModule returns the build info of this Caddy
@@ -277,7 +559,7 @@ func goModule(mod *debug.Module) *debug.Module {
 		// TODO: track related Go issue: https://github.com/golang/go/issues/29228
 		// once that issue is fixed, we should just be able to use bi.Main... hopefully.
 		for _, dep := range bi.Deps {
-			if dep.Path == "github.com/caddyserver/caddy/v2" {
+			if dep.Path == ImportPath {
 				return dep
 			}
 		}
@@ -289,8 +571,31 @@ func goModule(mod *debug.Module) *debug.Module {
 // CtxKey is a value type for use with context.WithValue.
 type CtxKey string
 
-// currentCfg is the currently-loaded configuration.
+// This group of variables pertains to the current configuration.
 var (
-	currentCfg   *Config
+	// currentCfgMu protects everything in this var block.
 	currentCfgMu sync.RWMutex
+
+	// currentCfg is the currently-running configuration.
+	currentCfg *Config
+
+	// rawCfg is the current, generic-decoded configuration;
+	// we initialize it as a map with one field ("config")
+	// to maintain parity with the API endpoint and to avoid
+	// the special case of having to access/mutate the variable
+	// directly without traversing into it.
+	rawCfg = map[string]interface{}{
+		rawConfigKey: nil,
+	}
+
+	// rawCfgJSON is the JSON-encoded form of rawCfg. Keeping
+	// this around avoids an extra Marshal call during changes.
+	rawCfgJSON []byte
+
+	// rawCfgIndex is the map of user-assigned ID to expanded
+	// path, for converting /id/ paths to /config/ paths.
+	rawCfgIndex map[string]string
 )
+
+// ImportPath is the package import path for Caddy core.
+const ImportPath = "github.com/caddyserver/caddy/v2"

caddy_test.go

@@ -0,0 +1,74 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddy
+
+import (
+	"testing"
+	"time"
+)
+
+func TestParseDuration(t *testing.T) {
+	const day = 24 * time.Hour
+	for i, tc := range []struct {
+		input  string
+		expect time.Duration
+	}{
+		{
+			input:  "3h",
+			expect: 3 * time.Hour,
+		},
+		{
+			input:  "1d",
+			expect: day,
+		},
+		{
+			input:  "1d30m",
+			expect: day + 30*time.Minute,
+		},
+		{
+			input:  "1m2d",
+			expect: time.Minute + day*2,
+		},
+		{
+			input:  "1m2d30s",
+			expect: time.Minute + day*2 + 30*time.Second,
+		},
+		{
+			input:  "1d2d",
+			expect: 3 * day,
+		},
+		{
+			input:  "1.5d",
+			expect: time.Duration(1.5 * float64(day)),
+		},
+		{
+			input:  "4m1.25d",
+			expect: 4*time.Minute + time.Duration(1.25*float64(day)),
+		},
+		{
+			input:  "-1.25d12h",
+			expect: time.Duration(-1.25*float64(day)) - 12*time.Hour,
+		},
+	} {
+		actual, err := ParseDuration(tc.input)
+		if err != nil {
+			t.Errorf("Test %d ('%s'): Got error: %v", i, tc.input, err)
+			continue
+		}
+		if actual != tc.expect {
+			t.Errorf("Test %d ('%s'): Expected=%s Actual=%s", i, tc.input, tc.expect, actual)
+		}
+	}
+}

caddyconfig/caddyfile/adapter.go

@@ -15,7 +15,6 @@
 package caddyfile
 
 import (
-	"bytes"
 	"encoding/json"
 	"fmt"
 
@@ -42,7 +41,7 @@ func (a Adapter) Adapt(body []byte, options map[string]interface{}) ([]byte, []c
 		filename = "Caddyfile"
 	}
 
-	serverBlocks, err := Parse(filename, bytes.NewReader(body))
+	serverBlocks, err := Parse(filename, body)
 	if err != nil {
 		return nil, nil, err
 	}
@@ -69,6 +68,11 @@ func (a Adapter) Adapt(body []byte, options map[string]interface{}) ([]byte, []c
 // into JSON. Caddyfile-unmarshaled values
 // will not be used directly; they will be
 // encoded as JSON and then used from that.
+// Implementations must be able to support
+// multiple segments (instances of their
+// directive or batch of tokens); typically
+// this means wrapping all token logic in
+// a loop: `for d.Next() { ... }`.
 type Unmarshaler interface {
 	UnmarshalCaddyfile(d *Dispenser) error
 }

caddyconfig/caddyfile/dispenser.go

@@ -17,6 +17,8 @@ package caddyfile
 import (
 	"errors"
 	"fmt"
+	"io"
+	"log"
 	"strings"
 )
 
@@ -37,6 +39,16 @@ func NewDispenser(tokens []Token) *Dispenser {
 	}
 }
 
+// NewTestDispenser parses input into tokens and creates a new
+// Dispenser for test purposes only; any errors are fatal.
+func NewTestDispenser(input string) *Dispenser {
+	tokens, err := allTokens("Testfile", []byte(input))
+	if err != nil && err != io.EOF {
+		log.Fatalf("getting all tokens from input: %v", err)
+	}
+	return NewDispenser(tokens)
+}
+
 // Next loads the next token. Returns true if a token
 // was loaded; false otherwise. If false, all tokens
 // have been consumed.
@@ -152,8 +164,10 @@ func (d *Dispenser) NextBlock(initialNestingLevel int) bool {
 		if !d.Next() {
 			return false // should be EOF error
 		}
-		if d.Val() == "}" {
+		if d.Val() == "}" && !d.nextOnSameLine() {
 			d.nesting--
+		} else if d.Val() == "{" && !d.nextOnSameLine() {
+			d.nesting++
 		}
 		return d.nesting > initialNestingLevel
 	}
@@ -247,13 +261,20 @@ func (d *Dispenser) RemainingArgs() []string {
 	return args
 }
 
-// NewFromNextTokens returns a new dispenser with a copy of
+// NewFromNextSegment returns a new dispenser with a copy of
 // the tokens from the current token until the end of the
 // "directive" whether that be to the end of the line or
 // the end of a block that starts at the end of the line;
 // in other words, until the end of the segment.
-func (d *Dispenser) NewFromNextTokens() *Dispenser {
-	tkns := []Token{d.Token()}
+func (d *Dispenser) NewFromNextSegment() *Dispenser {
+	return NewDispenser(d.NextSegment())
+}
+
+// NextSegment returns a copy of the tokens from the current
+// token until the end of the line or block that starts at
+// the end of the line.
+func (d *Dispenser) NextSegment() Segment {
+	tkns := Segment{d.Token()}
 	for d.NextArg() {
 		tkns = append(tkns, d.Token())
 	}
@@ -262,9 +283,9 @@ func (d *Dispenser) NewFromNextTokens() *Dispenser {
 		if !openedBlock {
 			// because NextBlock() consumes the initial open
 			// curly brace, we rewind here to append it, since
-			// our case is special in that we want to include
-			// all the tokens including surrounding curly braces
-			// for a new dispenser to have
+			// our case is special in that we want the new
+			// dispenser to have all the tokens including
+			// surrounding curly braces
 			d.Prev()
 			tkns = append(tkns, d.Token())
 			d.Next()
@@ -273,10 +294,14 @@ func (d *Dispenser) NewFromNextTokens() *Dispenser {
 		tkns = append(tkns, d.Token())
 	}
 	if openedBlock {
-		// include closing brace accordingly
+		// include closing brace
 		tkns = append(tkns, d.Token())
+
+		// do not consume the closing curly brace; the
+		// next iteration of the enclosing loop will
+		// call Next() and consume it
 	}
-	return NewDispenser(tkns)
+	return tkns
 }
 
 // Token returns the current token.
@@ -339,7 +364,7 @@ func (d *Dispenser) Errf(format string, args ...interface{}) error {
 // array will become apparent (or worse, hide from you like they
 // did me for 3 and a half freaking hours late one night).
 func (d *Dispenser) Delete() []Token {
-	if d.cursor >= 0 && d.cursor < len(d.tokens)-1 {
+	if d.cursor >= 0 && d.cursor <= len(d.tokens)-1 {
 		d.tokens = append(d.tokens[:d.cursor], d.tokens[d.cursor+1:]...)
 		d.cursor--
 	}

caddyconfig/caddyfile/dispenser_test.go

@@ -15,8 +15,6 @@
 package caddyfile
 
 import (
-	"io"
-	"log"
 	"reflect"
 	"strings"
 	"testing"
@@ -27,7 +25,7 @@ func TestDispenser_Val_Next(t *testing.T) {
 			  dir1 arg1
 			  dir2 arg2 arg3
 			  dir3`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	if val := d.Val(); val != "" {
 		t.Fatalf("Val(): Should return empty string when no token loaded; got '%s'", val)
@@ -65,7 +63,7 @@ func TestDispenser_NextArg(t *testing.T) {
 	input := `dir1 arg1
 			  dir2 arg2 arg3
 			  dir3`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	assertNext := func(shouldLoad bool, expectedVal string, expectedCursor int) {
 		if d.Next() != shouldLoad {
@@ -112,7 +110,7 @@ func TestDispenser_NextLine(t *testing.T) {
 	input := `host:port
 			  dir1 arg1
 			  dir2 arg2 arg3`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	assertNextLine := func(shouldLoad bool, expectedVal string, expectedCursor int) {
 		if d.NextLine() != shouldLoad {
@@ -145,7 +143,7 @@ func TestDispenser_NextBlock(t *testing.T) {
 			  }
 			  foobar2 {
 			  }`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	assertNextBlock := func(shouldLoad bool, expectedCursor, expectedNesting int) {
 		if loaded := d.NextBlock(0); loaded != shouldLoad {
@@ -175,7 +173,7 @@ func TestDispenser_Args(t *testing.T) {
 			  dir2 arg4 arg5
 			  dir3 arg6 arg7
 			  dir4`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	d.Next() // dir1
 
@@ -242,7 +240,7 @@ func TestDispenser_RemainingArgs(t *testing.T) {
 			  dir2 arg4 arg5
 			  dir3 arg6 { arg7
 			  dir4`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	d.Next() // dir1
 
@@ -279,7 +277,7 @@ func TestDispenser_ArgErr_Err(t *testing.T) {
 	input := `dir1 {
 			  }
 			  dir2 arg1 arg2`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	d.cursor = 1 // {
 
@@ -306,11 +304,3 @@ func TestDispenser_ArgErr_Err(t *testing.T) {
 		t.Errorf("Expected error message with custom message in it ('foobar'); got '%v'", err)
 	}
 }
-
-func newTestDispenser(input string) *Dispenser {
-	tokens, err := allTokens("Testfile", strings.NewReader(input))
-	if err != nil && err != io.EOF {
-		log.Fatalf("getting all tokens from input: %v", err)
-	}
-	return NewDispenser(tokens)
-}

caddyconfig/caddyfile/formatter.go

@@ -0,0 +1,213 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"bytes"
+	"io"
+	"unicode"
+)
+
+// Format formats the input Caddyfile to a standard, nice-looking
+// appearance. It works by reading each rune of the input and taking
+// control over all the bracing and whitespace that is written; otherwise,
+// words, comments, placeholders, and escaped characters are all treated
+// literally and written as they appear in the input.
+func Format(input []byte) []byte {
+	input = bytes.TrimSpace(input)
+
+	out := new(bytes.Buffer)
+	rdr := bytes.NewReader(input)
+
+	var (
+		last rune // the last character that was written to the result
+
+		space           = true // whether current/previous character was whitespace (beginning of input counts as space)
+		beginningOfLine = true // whether we are at beginning of line
+
+		openBrace        bool // whether current word/token is or started with open curly brace
+		openBraceWritten bool // if openBrace, whether that brace was written or not
+		openBraceSpace   bool // whether there was a non-newline space before open brace
+
+		newLines int // count of newlines consumed
+
+		comment bool // whether we're in a comment
+		quoted  bool // whether we're in a quoted segment
+		escaped bool // whether current char is escaped
+
+		nesting int // indentation level
+	)
+
+	write := func(ch rune) {
+		out.WriteRune(ch)
+		last = ch
+	}
+
+	indent := func() {
+		for tabs := nesting; tabs > 0; tabs-- {
+			write('\t')
+		}
+	}
+
+	nextLine := func() {
+		write('\n')
+		beginningOfLine = true
+	}
+
+	for {
+		ch, _, err := rdr.ReadRune()
+		if err != nil {
+			if err == io.EOF {
+				break
+			}
+			panic(err)
+		}
+
+		if comment {
+			if ch == '\n' {
+				comment = false
+			} else {
+				write(ch)
+				continue
+			}
+		}
+
+		if !escaped && ch == '\\' {
+			if space {
+				write(' ')
+				space = false
+			}
+			write(ch)
+			escaped = true
+			continue
+		}
+
+		if escaped {
+			write(ch)
+			escaped = false
+			continue
+		}
+
+		if quoted {
+			if ch == '"' {
+				quoted = false
+			}
+			write(ch)
+			continue
+		}
+
+		if space && ch == '"' {
+			quoted = true
+		}
+
+		if unicode.IsSpace(ch) {
+			space = true
+			if ch == '\n' {
+				newLines++
+			}
+			continue
+		}
+		spacePrior := space
+		space = false
+
+		//////////////////////////////////////////////////////////
+		// I find it helpful to think of the formatting loop in two
+		// main sections; by the time we reach this point, we
+		// know we are in a "regular" part of the file: we know
+		// the character is not a space, not in a literal segment
+		// like a comment or quoted, it's not escaped, etc.
+		//////////////////////////////////////////////////////////
+
+		if ch == '#' {
+			comment = true
+		}
+
+		if openBrace && spacePrior && !openBraceWritten {
+			if nesting == 0 && last == '}' {
+				nextLine()
+				nextLine()
+			}
+
+			openBrace = false
+			if beginningOfLine {
+				indent()
+			} else if !openBraceSpace {
+				write(' ')
+			}
+			write('{')
+			openBraceWritten = true
+			nextLine()
+			newLines = 0
+			nesting++
+		}
+
+		switch {
+		case ch == '{':
+			openBrace = true
+			openBraceWritten = false
+			openBraceSpace = spacePrior && !beginningOfLine
+			if openBraceSpace {
+				write(' ')
+			}
+			continue
+
+		case ch == '}' && (spacePrior || !openBrace):
+			if last != '\n' {
+				nextLine()
+			}
+			if nesting > 0 {
+				nesting--
+			}
+			indent()
+			write('}')
+			newLines = 0
+			continue
+		}
+
+		if newLines > 2 {
+			newLines = 2
+		}
+		for i := 0; i < newLines; i++ {
+			nextLine()
+		}
+		newLines = 0
+		if beginningOfLine {
+			indent()
+		}
+		if nesting == 0 && last == '}' && beginningOfLine {
+			nextLine()
+			nextLine()
+		}
+
+		if !beginningOfLine && spacePrior {
+			write(' ')
+		}
+
+		if openBrace && !openBraceWritten {
+			write('{')
+			openBraceWritten = true
+		}
+		write(ch)
+
+		beginningOfLine = false
+	}
+
+	// the Caddyfile does not need any leading or trailing spaces, but...
+	trimmedResult := bytes.TrimSpace(out.Bytes())
+
+	// ...Caddyfiles should, however, end with a newline because
+	// newlines are significant to the syntax of the file
+	return append(trimmedResult, '\n')
+}

caddyconfig/caddyfile/formatter_test.go

@@ -0,0 +1,327 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestFormatter(t *testing.T) {
+	for i, tc := range []struct {
+		description string
+		input       string
+		expect      string
+	}{
+		{
+			description: "very simple",
+			input: `abc   def
+	g hi jkl
+mn`,
+			expect: `abc def
+g hi jkl
+mn`,
+		},
+		{
+			description: "basic indentation, line breaks, and nesting",
+			input: `  a
+b
+
+	c {
+		d
+}
+
+e { f
+}
+
+
+
+g {
+h {
+i
+}
+}
+
+j { k {
+l
+}
+}
+
+m {
+	n { o
+	}
+	p { q r
+s }
+}
+
+	{
+{ t
+		u
+
+	v
+
+w
+}
+}`,
+			expect: `a
+b
+
+c {
+	d
+}
+
+e {
+	f
+}
+
+g {
+	h {
+		i
+	}
+}
+
+j {
+	k {
+		l
+	}
+}
+
+m {
+	n {
+		o
+	}
+	p {
+		q r
+		s
+	}
+}
+
+{
+	{
+		t
+		u
+
+		v
+
+		w
+	}
+}`,
+		},
+		{
+			description: "block spacing",
+			input: `a{
+	b
+}
+
+c{ d
+}`,
+			expect: `a {
+	b
+}
+
+c {
+	d
+}`,
+		},
+		{
+			description: "advanced spacing",
+			input: `abc {
+	def
+}ghi{
+	jkl mno
+pqr}`,
+			expect: `abc {
+	def
+}
+
+ghi {
+	jkl mno
+	pqr
+}`,
+		},
+		{
+			description: "env var placeholders",
+			input: `{$A}
+
+b {
+{$C}
+}
+
+d { {$E}
+}
+
+{ {$F}
+}
+`,
+			expect: `{$A}
+
+b {
+	{$C}
+}
+
+d {
+	{$E}
+}
+
+{
+	{$F}
+}`,
+		},
+		{
+			description: "comments",
+			input: `#a "\n"
+
+ #b {
+	c
+}
+
+d {
+e#f
+# g
+}
+
+h { # i
+}`,
+			expect: `#a "\n"
+
+#b {
+c
+}
+
+d {
+	e#f
+	# g
+}
+
+h {
+	# i
+}`,
+		},
+		{
+			description: "quotes and escaping",
+			input: `"a \"b\" "#c
+	d
+
+e {
+"f"
+}
+
+g { "h"
+}
+
+i {
+	"foo
+bar"
+}
+
+j {
+"\"k\" l m"
+}`,
+			expect: `"a \"b\" "#c
+d
+
+e {
+	"f"
+}
+
+g {
+	"h"
+}
+
+i {
+	"foo
+bar"
+}
+
+j {
+	"\"k\" l m"
+}`,
+		},
+		{
+			description: "bad nesting (too many open)",
+			input: `a
+{
+	{
+}`,
+			expect: `a {
+	{
+	}
+`,
+		},
+		{
+			description: "bad nesting (too many close)",
+			input: `a
+{
+	{
+}}}`,
+			expect: `a {
+	{
+	}
+}
+}
+`,
+		},
+		{
+			description: "json",
+			input: `foo
+bar      "{\"key\":34}"
+`,
+			expect: `foo
+bar "{\"key\":34}"`,
+		},
+		{
+			description: "escaping after spaces",
+			input:       `foo \"literal\"`,
+			expect:      `foo \"literal\"`,
+		},
+		{
+			description: "simple placeholders as standalone tokens",
+			input:       `foo {bar}`,
+			expect:      `foo {bar}`,
+		},
+		{
+			description: "simple placeholders within tokens",
+			input:       `foo{bar} foo{bar}baz`,
+			expect:      `foo{bar} foo{bar}baz`,
+		},
+		{
+			description: "placeholders and malformed braces",
+			input:       `foo{bar} foo{ bar}baz`,
+			expect: `foo{bar} foo {
+	bar
+}
+
+baz`,
+		},
+		{
+			description: "hash within string is not a comment",
+			input:       `redir / /some/#/path`,
+			expect:      `redir / /some/#/path`,
+		},
+	} {
+		// the formatter should output a trailing newline,
+		// even if the tests aren't written to expect that
+		if !strings.HasSuffix(tc.expect, "\n") {
+			tc.expect += "\n"
+		}
+
+		actual := Format([]byte(tc.input))
+
+		if string(actual) != tc.expect {
+			t.Errorf("\n[TEST %d: %s]\n====== EXPECTED ======\n%s\n====== ACTUAL ======\n%s^^^^^^^^^^^^^^^^^^^^^",
+				i, tc.description, string(tc.expect), string(actual))
+		}
+	}
+}

caddyconfig/caddyfile/lexer.go

@@ -16,6 +16,7 @@ package caddyfile
 
 import (
 	"bufio"
+	"bytes"
 	"io"
 	"unicode"
 )
@@ -73,7 +74,7 @@ func (l *lexer) load(input io.Reader) error {
 // a token was loaded; false otherwise.
 func (l *lexer) next() bool {
 	var val []rune
-	var comment, quoted, escaped bool
+	var comment, quoted, btQuoted, escaped bool
 
 	makeToken := func() bool {
 		l.token.Text = string(val)
@@ -92,13 +93,13 @@ func (l *lexer) next() bool {
 			panic(err)
 		}
 
-		if !escaped && ch == '\\' {
+		if !escaped && !btQuoted && ch == '\\' {
 			escaped = true
 			continue
 		}
 
-		if quoted {
-			if escaped {
+		if quoted || btQuoted {
+			if quoted && escaped {
 				// all is literal in quoted area,
 				// so only escape quotes
 				if ch != '"' {
@@ -106,7 +107,10 @@ func (l *lexer) next() bool {
 				}
 				escaped = false
 			} else {
-				if ch == '"' {
+				if quoted && ch == '"' {
+					return makeToken()
+				}
+				if btQuoted && ch == '`' {
 					return makeToken()
 				}
 			}
@@ -138,7 +142,7 @@ func (l *lexer) next() bool {
 			continue
 		}
 
-		if ch == '#' {
+		if ch == '#' && len(val) == 0 {
 			comment = true
 		}
 		if comment {
@@ -151,6 +155,10 @@ func (l *lexer) next() bool {
 				quoted = true
 				continue
 			}
+			if ch == '`' {
+				btQuoted = true
+				continue
+			}
 		}
 
 		if escaped {
@@ -161,3 +169,21 @@ func (l *lexer) next() bool {
 		val = append(val, ch)
 	}
 }
+
+// Tokenize takes bytes as input and lexes it into
+// a list of tokens that can be parsed as a Caddyfile.
+// Also takes a filename to fill the token's File as
+// the source of the tokens, which is important to
+// determine relative paths for `import` directives.
+func Tokenize(input []byte, filename string) ([]Token, error) {
+	l := lexer{}
+	if err := l.load(bytes.NewReader(input)); err != nil {
+		return nil, err
+	}
+	var tokens []Token
+	for l.next() {
+		l.token.File = filename
+		tokens = append(tokens, l.token)
+	}
+	return tokens, nil
+}

caddyconfig/caddyfile/lexer_test.go

@@ -15,37 +15,35 @@
 package caddyfile
 
 import (
-	"log"
-	"strings"
 	"testing"
 )
 
 type lexerTestCase struct {
-	input    string
+	input    []byte
 	expected []Token
 }
 
 func TestLexer(t *testing.T) {
 	testCases := []lexerTestCase{
 		{
-			input: `host:123`,
+			input: []byte(`host:123`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 			},
 		},
 		{
-			input: `host:123
+			input: []byte(`host:123
 
-					directive`,
+					directive`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 3, Text: "directive"},
 			},
 		},
 		{
-			input: `host:123 {
+			input: []byte(`host:123 {
 						directive
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -54,7 +52,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `host:123 { directive }`,
+			input: []byte(`host:123 { directive }`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -63,12 +61,12 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `host:123 {
+			input: []byte(`host:123 {
 						#comment
 						directive
 						# comment
 						foobar # another comment
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -78,8 +76,28 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `a "quoted value" b
-					foobar`,
+			input: []byte(`host:123 {
+						# hash inside string is not a comment
+						redir / /some/#/path
+					}`),
+			expected: []Token{
+				{Line: 1, Text: "host:123"},
+				{Line: 1, Text: "{"},
+				{Line: 3, Text: "redir"},
+				{Line: 3, Text: "/"},
+				{Line: 3, Text: "/some/#/path"},
+				{Line: 4, Text: "}"},
+			},
+		},
+		{
+			input: []byte("# comment at beginning of file\n# comment at beginning of line\nhost:123"),
+			expected: []Token{
+				{Line: 3, Text: "host:123"},
+			},
+		},
+		{
+			input: []byte(`a "quoted value" b
+					foobar`),
 			expected: []Token{
 				{Line: 1, Text: "a"},
 				{Line: 1, Text: "quoted value"},
@@ -88,7 +106,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `A "quoted \"value\" inside" B`,
+			input: []byte(`A "quoted \"value\" inside" B`),
 			expected: []Token{
 				{Line: 1, Text: "A"},
 				{Line: 1, Text: `quoted "value" inside`},
@@ -96,7 +114,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "An escaped \"newline\\\ninside\" quotes",
+			input: []byte("An escaped \"newline\\\ninside\" quotes"),
 			expected: []Token{
 				{Line: 1, Text: "An"},
 				{Line: 1, Text: "escaped"},
@@ -105,7 +123,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "An escaped newline\\\noutside quotes",
+			input: []byte("An escaped newline\\\noutside quotes"),
 			expected: []Token{
 				{Line: 1, Text: "An"},
 				{Line: 1, Text: "escaped"},
@@ -115,7 +133,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "line1\\\nescaped\nline2\nline3",
+			input: []byte("line1\\\nescaped\nline2\nline3"),
 			expected: []Token{
 				{Line: 1, Text: "line1"},
 				{Line: 1, Text: "escaped"},
@@ -124,7 +142,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "line1\\\nescaped1\\\nescaped2\nline4\nline5",
+			input: []byte("line1\\\nescaped1\\\nescaped2\nline4\nline5"),
 			expected: []Token{
 				{Line: 1, Text: "line1"},
 				{Line: 1, Text: "escaped1"},
@@ -134,34 +152,34 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `"unescapable\ in quotes"`,
+			input: []byte(`"unescapable\ in quotes"`),
 			expected: []Token{
 				{Line: 1, Text: `unescapable\ in quotes`},
 			},
 		},
 		{
-			input: `"don't\escape"`,
+			input: []byte(`"don't\escape"`),
 			expected: []Token{
 				{Line: 1, Text: `don't\escape`},
 			},
 		},
 		{
-			input: `"don't\\escape"`,
+			input: []byte(`"don't\\escape"`),
 			expected: []Token{
 				{Line: 1, Text: `don't\\escape`},
 			},
 		},
 		{
-			input: `un\escapable`,
+			input: []byte(`un\escapable`),
 			expected: []Token{
 				{Line: 1, Text: `un\escapable`},
 			},
 		},
 		{
-			input: `A "quoted value with line
+			input: []byte(`A "quoted value with line
 					break inside" {
 						foobar
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "A"},
 				{Line: 1, Text: "quoted value with line\n\t\t\t\t\tbreak inside"},
@@ -171,13 +189,13 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `"C:\php\php-cgi.exe"`,
+			input: []byte(`"C:\php\php-cgi.exe"`),
 			expected: []Token{
 				{Line: 1, Text: `C:\php\php-cgi.exe`},
 			},
 		},
 		{
-			input: `empty "" string`,
+			input: []byte(`empty "" string`),
 			expected: []Token{
 				{Line: 1, Text: `empty`},
 				{Line: 1, Text: ``},
@@ -185,7 +203,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "skip those\r\nCR characters",
+			input: []byte("skip those\r\nCR characters"),
 			expected: []Token{
 				{Line: 1, Text: "skip"},
 				{Line: 1, Text: "those"},
@@ -194,30 +212,54 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "\xEF\xBB\xBF:8080", // test with leading byte order mark
+			input: []byte("\xEF\xBB\xBF:8080"), // test with leading byte order mark
 			expected: []Token{
 				{Line: 1, Text: ":8080"},
 			},
 		},
+		{
+			input: []byte("simple `backtick quoted` string"),
+			expected: []Token{
+				{Line: 1, Text: `simple`},
+				{Line: 1, Text: `backtick quoted`},
+				{Line: 1, Text: `string`},
+			},
+		},
+		{
+			input: []byte("multiline `backtick\nquoted\n` string"),
+			expected: []Token{
+				{Line: 1, Text: `multiline`},
+				{Line: 1, Text: "backtick\nquoted\n"},
+				{Line: 3, Text: `string`},
+			},
+		},
+		{
+			input: []byte("nested `\"quotes inside\" backticks` string"),
+			expected: []Token{
+				{Line: 1, Text: `nested`},
+				{Line: 1, Text: `"quotes inside" backticks`},
+				{Line: 1, Text: `string`},
+			},
+		},
+		{
+			input: []byte("reverse-nested \"`backticks` inside\" quotes"),
+			expected: []Token{
+				{Line: 1, Text: `reverse-nested`},
+				{Line: 1, Text: "`backticks` inside"},
+				{Line: 1, Text: `quotes`},
+			},
+		},
 	}
 
 	for i, testCase := range testCases {
-		actual := tokenize(testCase.input)
+		actual, err := Tokenize(testCase.input, "")
+		if err != nil {
+			t.Errorf("%v", err)
+		}
 		lexerCompare(t, i, testCase.expected, actual)
 	}
 }
 
-func tokenize(input string) (tokens []Token) {
-	l := lexer{}
-	if err := l.load(strings.NewReader(input)); err != nil {
-		log.Printf("[ERROR] load failed: %v", err)
-	}
-	for l.next() {
-		tokens = append(tokens, l.token)
-	}
-	return
-}
-
 func lexerCompare(t *testing.T, n int, expected, actual []Token) {
 	if len(expected) != len(actual) {
 		t.Errorf("Test case %d: expected %d token(s) but got %d", n, len(expected), len(actual))

caddyconfig/caddyfile/parse.go

@@ -15,11 +15,15 @@
 package caddyfile
 
 import (
-	"io"
+	"bytes"
+	"io/ioutil"
 	"log"
 	"os"
 	"path/filepath"
+	"strconv"
 	"strings"
+
+	"github.com/caddyserver/caddy/v2"
 )
 
 // Parse parses the input just enough to group tokens, in
@@ -28,7 +32,10 @@ import (
 // Directives that do not appear in validDirectives will cause
 // an error. If you do not want to check for valid directives,
 // pass in nil instead.
-func Parse(filename string, input io.Reader) ([]ServerBlock, error) {
+//
+// Environment variables in {$ENVIRONMENT_VARIABLE} notation
+// will be replaced before parsing begins.
+func Parse(filename string, input []byte) ([]ServerBlock, error) {
 	tokens, err := allTokens(filename, input)
 	if err != nil {
 		return nil, err
@@ -37,19 +44,52 @@ func Parse(filename string, input io.Reader) ([]ServerBlock, error) {
 	return p.parseAll()
 }
 
+// replaceEnvVars replaces all occurrences of environment variables.
+func replaceEnvVars(input []byte) ([]byte, error) {
+	var offset int
+	for {
+		begin := bytes.Index(input[offset:], spanOpen)
+		if begin < 0 {
+			break
+		}
+		begin += offset // make beginning relative to input, not offset
+		end := bytes.Index(input[begin+len(spanOpen):], spanClose)
+		if end < 0 {
+			break
+		}
+		end += begin + len(spanOpen) // make end relative to input, not begin
+
+		// get the name; if there is no name, skip it
+		envVarName := input[begin+len(spanOpen) : end]
+		if len(envVarName) == 0 {
+			offset = end + len(spanClose)
+			continue
+		}
+
+		// get the value of the environment variable
+		envVarValue := []byte(os.ExpandEnv(os.Getenv(string(envVarName))))
+
+		// splice in the value
+		input = append(input[:begin],
+			append(envVarValue, input[end+len(spanClose):]...)...)
+
+		// continue at the end of the replacement
+		offset = begin + len(envVarValue)
+	}
+	return input, nil
+}
+
 // allTokens lexes the entire input, but does not parse it.
 // It returns all the tokens from the input, unstructured
 // and in order.
-func allTokens(filename string, input io.Reader) ([]Token, error) {
-	l := new(lexer)
-	err := l.load(input)
+func allTokens(filename string, input []byte) ([]Token, error) {
+	input, err := replaceEnvVars(input)
 	if err != nil {
 		return nil, err
 	}
-	var tokens []Token
-	for l.next() {
-		l.token.File = filename
-		tokens = append(tokens, l.token)
+	tokens, err := Tokenize(input, filename)
+	if err != nil {
+		return nil, err
 	}
 	return tokens, nil
 }
@@ -128,7 +168,7 @@ func (p *parser) addresses() error {
 	var expectingAnother bool
 
 	for {
-		tkn := replaceEnvVars(p.Val())
+		tkn := p.Val()
 
 		// special case: import directive replaces tokens during parse-time
 		if tkn == "import" && p.isNewLine() {
@@ -245,15 +285,23 @@ func (p *parser) doImport() error {
 	if !p.NextArg() {
 		return p.ArgErr()
 	}
-	importPattern := replaceEnvVars(p.Val())
+	importPattern := p.Val()
 	if importPattern == "" {
 		return p.Err("Import requires a non-empty filepath")
 	}
-	if p.NextArg() {
-		return p.Err("Import takes only one argument (glob pattern or file)")
+
+	// grab remaining args as placeholder replacements
+	args := p.RemainingArgs()
+
+	// add args to the replacer
+	repl := caddy.NewReplacer()
+	for index, arg := range args {
+		repl.Set("args."+strconv.Itoa(index), arg)
 	}
-	// splice out the import directive and its argument (2 tokens total)
-	tokensBefore := p.tokens[:p.cursor-1]
+
+	// splice out the import directive and its arguments
+	// (2 tokens, plus the length of args)
+	tokensBefore := p.tokens[:p.cursor-1-len(args)]
 	tokensAfter := p.tokens[p.cursor+1:]
 	var importedTokens []Token
 
@@ -305,10 +353,20 @@ func (p *parser) doImport() error {
 		}
 	}
 
+	// copy the tokens so we don't overwrite p.definedSnippets
+	tokensCopy := make([]Token, len(importedTokens))
+	copy(tokensCopy, importedTokens)
+
+	// run the argument replacer on the tokens
+	for index, token := range tokensCopy {
+		token.Text = repl.ReplaceKnown(token.Text, "")
+		tokensCopy[index] = token
+	}
+
 	// splice the imported tokens in the place of the import statement
 	// and rewind cursor so Next() will land on first imported token
-	p.tokens = append(tokensBefore, append(importedTokens, tokensAfter...)...)
-	p.cursor--
+	p.tokens = append(tokensBefore, append(tokensCopy, tokensAfter...)...)
+	p.cursor -= len(args) + 1
 
 	return nil
 }
@@ -328,7 +386,12 @@ func (p *parser) doSingleImport(importFile string) ([]Token, error) {
 		return nil, p.Errf("Could not import %s: is a directory", importFile)
 	}
 
-	importedTokens, err := allTokens(importFile, file)
+	input, err := ioutil.ReadAll(file)
+	if err != nil {
+		return nil, p.Errf("Could not read imported file %s: %v", importFile, err)
+	}
+
+	importedTokens, err := allTokens(importFile, input)
 	if err != nil {
 		return nil, p.Errf("Could not read tokens while importing %s: %v", importFile, err)
 	}
@@ -353,8 +416,6 @@ func (p *parser) doSingleImport(importFile string) ([]Token, error) {
 // are loaded into the current server block for later use
 // by directive setup functions.
 func (p *parser) directive() error {
-	// evaluate any env vars in directive token
-	p.tokens[p.cursor].Text = replaceEnvVars(p.tokens[p.cursor].Text)
 
 	// a segment is a list of tokens associated with this directive
 	var segment Segment
@@ -379,7 +440,7 @@ func (p *parser) directive() error {
 			p.cursor-- // cursor is advanced when we continue, so roll back one more
 			continue
 		}
-		p.tokens[p.cursor].Text = replaceEnvVars(p.tokens[p.cursor].Text)
+
 		segment = append(segment, p.Token())
 	}
 
@@ -414,36 +475,6 @@ func (p *parser) closeCurlyBrace() error {
 	return nil
 }
 
-// replaceEnvVars replaces environment variables that appear in the token
-// and understands both the $UNIX and %WINDOWS% syntaxes.
-func replaceEnvVars(s string) string {
-	s = replaceEnvReferences(s, "{%", "%}")
-	s = replaceEnvReferences(s, "{$", "}")
-	return s
-}
-
-// replaceEnvReferences performs the actual replacement of env variables
-// in s, given the placeholder start and placeholder end strings.
-func replaceEnvReferences(s, refStart, refEnd string) string {
-	index := strings.Index(s, refStart)
-	for index != -1 {
-		endIndex := strings.Index(s[index:], refEnd)
-		if endIndex == -1 {
-			break
-		}
-
-		endIndex += index
-		if endIndex > index+len(refStart) {
-			ref := s[index : endIndex+len(refEnd)]
-			s = strings.Replace(s, ref, os.Getenv(ref[len(refStart):len(ref)-len(refEnd)]), -1)
-		} else {
-			return s
-		}
-		index = strings.Index(s, refStart)
-	}
-	return s
-}
-
 func (p *parser) isSnippet() (bool, string) {
 	keys := p.block.Keys
 	// A snippet block is a single key with parens. Nothing else qualifies.
@@ -514,3 +545,7 @@ func (s Segment) Directive() string {
 	}
 	return ""
 }
+
+// spanOpen and spanClose are used to bound spans that
+// contain the name of an environment variable.
+var spanOpen, spanClose = []byte{'{', '$'}, []byte{'}'}

caddyconfig/caddyfile/parse_test.go

@@ -15,15 +15,15 @@
 package caddyfile
 
 import (
+	"bytes"
 	"io/ioutil"
 	"os"
 	"path/filepath"
-	"strings"
 	"testing"
 )
 
 func TestAllTokens(t *testing.T) {
-	input := strings.NewReader("a b c\nd e")
+	input := []byte("a b c\nd e")
 	expected := []string{"a", "b", "c", "d", "e"}
 	tokens, err := allTokens("TestAllTokens", input)
 
@@ -182,14 +182,17 @@ func TestParseOneAndImport(t *testing.T) {
 			"host1",
 		}, []int{1, 2}},
 
-		{`import testdata/import_test1.txt testdata/import_test2.txt`, true, []string{}, []int{}},
-
 		{`import testdata/not_found.txt`, true, []string{}, []int{}},
 
 		{`""`, false, []string{}, []int{}},
 
 		{``, false, []string{}, []int{}},
 
+		// import with args
+		{`import testdata/import_args0.txt a`, false, []string{"a"}, []int{}},
+		{`import testdata/import_args1.txt a b`, false, []string{"a", "b"}, []int{}},
+		{`import testdata/import_args*.txt a b`, false, []string{"a"}, []int{2}},
+
 		// test cases found by fuzzing!
 		{`import }{$"`, true, []string{}, []int{}},
 		{`import /*/*.txt`, true, []string{}, []int{}},
@@ -210,6 +213,7 @@ func TestParseOneAndImport(t *testing.T) {
 			t.Errorf("Test %d: Expected no error, but got: %v", i, err)
 		}
 
+		// t.Logf("%+v\n", result)
 		if len(result.Keys) != len(test.keys) {
 			t.Errorf("Test %d: Expected %d keys, got %d",
 				i, len(test.keys), len(result.Keys))
@@ -256,7 +260,7 @@ func TestRecursiveImport(t *testing.T) {
 			return false
 		}
 		if len(got.Segments[0]) != 1 || len(got.Segments[1]) != 2 {
-			t.Errorf("got unexpect tokens: %v", got.Segments)
+			t.Errorf("got unexpected tokens: %v", got.Segments)
 			return false
 		}
 		return true
@@ -351,7 +355,7 @@ func TestDirectiveImport(t *testing.T) {
 			return false
 		}
 		if len(got.Segments[0]) != 1 || len(got.Segments[1]) != 8 {
-			t.Errorf("got unexpect tokens: %v", got.Segments)
+			t.Errorf("got unexpected tokens: %v", got.Segments)
 			return false
 		}
 		return true
@@ -473,89 +477,84 @@ func TestParseAll(t *testing.T) {
 }
 
 func TestEnvironmentReplacement(t *testing.T) {
-	os.Setenv("PORT", "8080")
-	os.Setenv("ADDRESS", "servername.com")
 	os.Setenv("FOOBAR", "foobar")
-	os.Setenv("PARTIAL_DIR", "r1")
 
-	// basic test; unix-style env vars
-	p := testParser(`{$ADDRESS}`)
-	blocks, _ := p.parseAll()
-	if actual, expected := blocks[0].Keys[0], "servername.com"; expected != actual {
-		t.Errorf("Expected key to be '%s' but was '%s'", expected, actual)
-	}
-
-	// basic test; unix-style env vars
-	p = testParser(`di{$PARTIAL_DIR}`)
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Keys[0], "dir1"; expected != actual {
-		t.Errorf("Expected key to be '%s' but was '%s'", expected, actual)
-	}
-
-	// multiple vars per token
-	p = testParser(`{$ADDRESS}:{$PORT}`)
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Keys[0], "servername.com:8080"; expected != actual {
-		t.Errorf("Expected key to be '%s' but was '%s'", expected, actual)
-	}
-
-	// windows-style var and unix style in same token
-	p = testParser(`{%ADDRESS%}:{$PORT}`)
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Keys[0], "servername.com:8080"; expected != actual {
-		t.Errorf("Expected key to be '%s' but was '%s'", expected, actual)
-	}
-
-	// reverse order
-	p = testParser(`{$ADDRESS}:{%PORT%}`)
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Keys[0], "servername.com:8080"; expected != actual {
-		t.Errorf("Expected key to be '%s' but was '%s'", expected, actual)
-	}
-
-	// env var in server block body as argument
-	p = testParser(":{%PORT%}\ndir1 {$FOOBAR}")
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Keys[0], ":8080"; expected != actual {
-		t.Errorf("Expected key to be '%s' but was '%s'", expected, actual)
-	}
-	if actual, expected := blocks[0].Segments[0][1].Text, "foobar"; expected != actual {
-		t.Errorf("Expected argument to be '%s' but was '%s'", expected, actual)
-	}
-
-	// combined windows env vars in argument
-	p = testParser(":{%PORT%}\ndir1 {%ADDRESS%}/{%FOOBAR%}")
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Segments[0][1].Text, "servername.com/foobar"; expected != actual {
-		t.Errorf("Expected argument to be '%s' but was '%s'", expected, actual)
-	}
-
-	// malformed env var (windows)
-	p = testParser(":1234\ndir1 {%ADDRESS}")
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Segments[0][1].Text, "{%ADDRESS}"; expected != actual {
-		t.Errorf("Expected host to be '%s' but was '%s'", expected, actual)
-	}
-
-	// malformed (non-existent) env var (unix)
-	p = testParser(`:{$PORT$}`)
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Keys[0], ":"; expected != actual {
-		t.Errorf("Expected key to be '%s' but was '%s'", expected, actual)
-	}
-
-	// in quoted field
-	p = testParser(":1234\ndir1 \"Test {$FOOBAR} test\"")
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Segments[0][1].Text, "Test foobar test"; expected != actual {
-		t.Errorf("Expected argument to be '%s' but was '%s'", expected, actual)
-	}
-
-	// after end token
-	p = testParser(":1234\nanswer \"{{ .Name }} {$FOOBAR}\"")
-	blocks, _ = p.parseAll()
-	if actual, expected := blocks[0].Segments[0][1].Text, "{{ .Name }} foobar"; expected != actual {
-		t.Errorf("Expected argument to be '%s' but was '%s'", expected, actual)
+	for i, test := range []struct {
+		input  string
+		expect string
+	}{
+		{
+			input:  "",
+			expect: "",
+		},
+		{
+			input:  "foo",
+			expect: "foo",
+		},
+		{
+			input:  "{$NOT_SET}",
+			expect: "",
+		},
+		{
+			input:  "foo{$NOT_SET}bar",
+			expect: "foobar",
+		},
+		{
+			input:  "{$FOOBAR}",
+			expect: "foobar",
+		},
+		{
+			input:  "foo {$FOOBAR} bar",
+			expect: "foo foobar bar",
+		},
+		{
+			input:  "foo{$FOOBAR}bar",
+			expect: "foofoobarbar",
+		},
+		{
+			input:  "foo\n{$FOOBAR}\nbar",
+			expect: "foo\nfoobar\nbar",
+		},
+		{
+			input:  "{$FOOBAR} {$FOOBAR}",
+			expect: "foobar foobar",
+		},
+		{
+			input:  "{$FOOBAR}{$FOOBAR}",
+			expect: "foobarfoobar",
+		},
+		{
+			input:  "{$FOOBAR",
+			expect: "{$FOOBAR",
+		},
+		{
+			input:  "{$LONGER_NAME $FOOBAR}",
+			expect: "",
+		},
+		{
+			input:  "{$}",
+			expect: "{$}",
+		},
+		{
+			input:  "{$$}",
+			expect: "",
+		},
+		{
+			input:  "{$",
+			expect: "{$",
+		},
+		{
+			input:  "}{$",
+			expect: "}{$",
+		},
+	} {
+		actual, err := replaceEnvVars([]byte(test.input))
+		if err != nil {
+			t.Fatal(err)
+		}
+		if !bytes.Equal(actual, []byte(test.expect)) {
+			t.Errorf("Test %d: Expected: '%s' but got '%s'", i, test.expect, actual)
+		}
 	}
 }
 
@@ -573,10 +572,6 @@ func TestSnippets(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	if len(blocks) != 1 {
 		t.Fatalf("Expect exactly one server block. Got %d.", len(blocks))
 	}
@@ -621,10 +616,6 @@ func TestImportedFilesIgnoreNonDirectiveImportTokens(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	auth := blocks[0].Segments[0]
 	line := auth[0].Text + " " + auth[1].Text + " " + auth[2].Text + " " + auth[3].Text
 	if line != "basicauth / import password" {
@@ -656,10 +647,6 @@ func TestSnippetAcrossMultipleFiles(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	if len(blocks) != 1 {
 		t.Fatalf("Expect exactly one server block. Got %d.", len(blocks))
 	}
@@ -675,5 +662,5 @@ func TestSnippetAcrossMultipleFiles(t *testing.T) {
 }
 
 func testParser(input string) parser {
-	return parser{Dispenser: newTestDispenser(input)}
+	return parser{Dispenser: NewTestDispenser(input)}
 }

caddyconfig/caddyfile/testdata/import_args0.txt

@@ -0,0 +1 @@
+{args.0}

caddyconfig/caddyfile/testdata/import_args1.txt

@@ -0,0 +1 @@
+{args.0} {args.1}

caddyconfig/configadapters.go

@@ -17,6 +17,8 @@ package caddyconfig
 import (
 	"encoding/json"
 	"fmt"
+
+	"github.com/caddyserver/caddy/v2"
 )
 
 // Adapter is a type which can adapt a configuration to Caddy JSON.
@@ -49,14 +51,13 @@ func JSON(val interface{}, warnings *[]Warning) json.RawMessage {
 	return b
 }
 
-// JSONModuleObject is like JSON, except it marshals val into a JSON object
-// and then adds a key to that object named fieldName with the value fieldVal.
-// This is useful for JSON-encoding module values where the module name has to
-// be described within the object by a certain key; for example,
-// "responder": "file_server" for a file server HTTP responder. The val must
-// encode into a map[string]interface{} (i.e. it must be a struct or map),
-// and any errors are converted into warnings, so this can be conveniently
-// used when filling a struct. For correct code, there should be no errors.
+// JSONModuleObject is like JSON(), except it marshals val into a JSON object
+// with an added key named fieldName with the value fieldVal. This is useful
+// for encoding module values where the module name has to be described within
+// the object by a certain key; for example, `"handler": "file_server"` for a
+// file server HTTP handler (fieldName="handler" and fieldVal="file_server").
+// The val parameter must encode into a map[string]interface{} (i.e. it must be
+// a struct or map). Any errors are converted into warnings.
 func JSONModuleObject(val interface{}, fieldName, fieldVal string, warnings *[]Warning) json.RawMessage {
 	// encode to a JSON object first
 	enc, err := json.Marshal(val)
@@ -99,13 +100,14 @@ func JSONIndent(val interface{}) ([]byte, error) {
 }
 
 // RegisterAdapter registers a config adapter with the given name.
-// This should usually be done at init-time.
-func RegisterAdapter(name string, adapter Adapter) error {
+// This should usually be done at init-time. It panics if the
+// adapter cannot be registered successfully.
+func RegisterAdapter(name string, adapter Adapter) {
 	if _, ok := configAdapters[name]; ok {
-		return fmt.Errorf("%s: already registered", name)
+		panic(fmt.Errorf("%s: already registered", name))
 	}
 	configAdapters[name] = adapter
-	return nil
+	caddy.RegisterModule(adapterModule{name, adapter})
 }
 
 // GetAdapter returns the adapter with the given name,
@@ -114,4 +116,21 @@ func GetAdapter(name string) Adapter {
 	return configAdapters[name]
 }
 
+// adapterModule is a wrapper type that can turn any config
+// adapter into a Caddy module, which has the benefit of being
+// counted with other modules, even though they do not
+// technically extend the Caddy configuration structure.
+// See caddyserver/caddy#3132.
+type adapterModule struct {
+	name string
+	Adapter
+}
+
+func (am adapterModule) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  caddy.ModuleID("caddy.adapters." + am.name),
+		New: func() caddy.Module { return am },
+	}
+}
+
 var configAdapters = make(map[string]Adapter)

caddyconfig/httpcaddyfile/adapter_fuzz.go

@@ -1,49 +0,0 @@
-// Copyright 2015 Matthew Holt and The Caddy Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-// +build gofuzz
-// +build gofuzz_libfuzzer
-
-package httpcaddyfile
-
-import (
-	"bytes"
-
-	"github.com/caddyserver/caddy/v2"
-	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
-)
-
-func FuzzHTTPCaddyfileAdapter(data []byte) int {
-	adapter := caddyfile.Adapter{
-		ServerType: ServerType{},
-	}
-	b, warns, err := adapter.Adapt(data, nil)
-	// Adapt func calls the Setup() func of the ServerType,
-	// thus it's going across multiple layers, each can
-	// return warnings or errors. Marking the presence of
-	// errors or warnings as interesting in this case
-	// could push the fuzzer towards a path where we only
-	// catch errors. Let's push the fuzzer to where it passes
-	// but breaks.
-	if (err != nil) || (warns != nil && len(warns) > 0) {
-		return 0
-	}
-
-	// adapted Caddyfile should be parseable by the configuration loader in admin.go
-	err = caddy.Load(bytes.NewReader(b))
-	if err != nil {
-		return 0
-	}
-	return 1
-}

caddyconfig/httpcaddyfile/addresses.go

@@ -17,14 +17,15 @@ package httpcaddyfile
 import (
 	"fmt"
 	"net"
-	"net/url"
 	"reflect"
 	"strconv"
 	"strings"
+	"unicode"
 
+	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
-	"github.com/mholt/certmagic"
+	"github.com/caddyserver/certmagic"
 )
 
 // mapAddressToServerBlocks returns a map of listener address to list of server
@@ -46,7 +47,7 @@ import (
 // key of its server block (specifying the host part), and each key may have
 // a different port. And we definitely need to be sure that a site which is
 // bound to be served on a specific interface is not served on others just
-// beceause that is more convenient: it would be a potential security risk
+// because that is more convenient: it would be a potential security risk
 // if the difference between interfaces means private vs. public.
 //
 // So what this function does for the example above is iterate each server
@@ -73,7 +74,8 @@ import (
 // repetition may be undesirable, so call consolidateAddrMappings() to map
 // multiple addresses to the same lists of server blocks (a many:many mapping).
 // (Doing this is essentially a map-reduce technique.)
-func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBlock) (map[string][]serverBlock, error) {
+func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBlock,
+	options map[string]interface{}) (map[string][]serverBlock, error) {
 	sbmap := make(map[string][]serverBlock)
 
 	for i, sblock := range originalServerBlocks {
@@ -88,7 +90,7 @@ func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBloc
 			// arguments to the 'bind' directive (although they will all have
 			// the same port, since the port is defined by the key or is implicit
 			// through automatic HTTPS)
-			addrs, err := st.listenerAddrsForServerBlockKey(sblock, key)
+			addrs, err := st.listenerAddrsForServerBlockKey(sblock, key, options)
 			if err != nil {
 				return nil, fmt.Errorf("server block %d, key %d (%s): determining listener address: %v", i, j, key, err)
 			}
@@ -105,12 +107,22 @@ func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBloc
 		// server block are only the ones which use the address; but
 		// the contents (tokens) are of course the same
 		for addr, keys := range addrToKeys {
+			// parse keys so that we only have to do it once
+			parsedKeys := make([]Address, 0, len(keys))
+			for _, key := range keys {
+				addr, err := ParseAddress(key)
+				if err != nil {
+					return nil, fmt.Errorf("parsing key '%s': %v", key, err)
+				}
+				parsedKeys = append(parsedKeys, addr.Normalize())
+			}
 			sbmap[addr] = append(sbmap[addr], serverBlock{
 				block: caddyfile.ServerBlock{
 					Keys:     keys,
 					Segments: sblock.block.Segments,
 				},
 				pile: sblock.pile,
+				keys: parsedKeys,
 			})
 		}
 	}
@@ -127,7 +139,7 @@ func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBloc
 // association from multiple addresses to multiple server blocks; i.e. each element of
 // the returned slice) becomes a server definition in the output JSON.
 func (st *ServerType) consolidateAddrMappings(addrToServerBlocks map[string][]serverBlock) []sbAddrAssociation {
-	var sbaddrs []sbAddrAssociation
+	sbaddrs := make([]sbAddrAssociation, 0, len(addrToServerBlocks))
 	for addr, sblocks := range addrToServerBlocks {
 		// we start with knowing that at least this address
 		// maps to these server blocks
@@ -154,24 +166,41 @@ func (st *ServerType) consolidateAddrMappings(addrToServerBlocks map[string][]se
 	return sbaddrs
 }
 
-func (st *ServerType) listenerAddrsForServerBlockKey(sblock serverBlock, key string) ([]string, error) {
+func (st *ServerType) listenerAddrsForServerBlockKey(sblock serverBlock, key string,
+	options map[string]interface{}) ([]string, error) {
 	addr, err := ParseAddress(key)
 	if err != nil {
 		return nil, fmt.Errorf("parsing key: %v", err)
 	}
 	addr = addr.Normalize()
 
-	lnPort := DefaultPort
+	// figure out the HTTP and HTTPS ports; either
+	// use defaults, or override with user config
+	httpPort, httpsPort := strconv.Itoa(caddyhttp.DefaultHTTPPort), strconv.Itoa(caddyhttp.DefaultHTTPSPort)
+	if hport, ok := options["http_port"]; ok {
+		httpPort = strconv.Itoa(hport.(int))
+	}
+	if hsport, ok := options["https_port"]; ok {
+		httpsPort = strconv.Itoa(hsport.(int))
+	}
+
+	// default port is the HTTPS port
+	lnPort := httpsPort
 	if addr.Port != "" {
 		// port explicitly defined
 		lnPort = addr.Port
-	} else if certmagic.HostQualifies(addr.Host) {
-		// automatic HTTPS
-		lnPort = strconv.Itoa(certmagic.HTTPSPort)
+	} else if addr.Scheme == "http" {
+		// port inferred from scheme
+		lnPort = httpPort
+	}
+
+	// error if scheme and port combination violate convention
+	if (addr.Scheme == "http" && lnPort == httpsPort) || (addr.Scheme == "https" && lnPort == httpPort) {
+		return nil, fmt.Errorf("[%s] scheme and port violate convention", key)
 	}
 
 	// the bind directive specifies hosts, but is optional
-	var lnHosts []string
+	lnHosts := make([]string, 0, len(sblock.pile))
 	for _, cfgVal := range sblock.pile["bind"] {
 		lnHosts = append(lnHosts, cfgVal.Value.([]string)...)
 	}
@@ -182,15 +211,19 @@ func (st *ServerType) listenerAddrsForServerBlockKey(sblock serverBlock, key str
 	// use a map to prevent duplication
 	listeners := make(map[string]struct{})
 	for _, host := range lnHosts {
-		listeners[net.JoinHostPort(host, lnPort)] = struct{}{}
+		addr, err := caddy.ParseNetworkAddress(host)
+		if err == nil && addr.IsUnixNetwork() {
+			listeners[host] = struct{}{}
+		} else {
+			listeners[net.JoinHostPort(host, lnPort)] = struct{}{}
+		}
 	}
 
 	// now turn map into list
-	var listenersList []string
+	listenersList := make([]string, 0, len(listeners))
 	for lnStr := range listeners {
 		listenersList = append(listenersList, lnStr)
 	}
-	// sort.Strings(listenersList) // TODO: is sorting necessary?
 
 	return listenersList, nil
 }
@@ -209,48 +242,55 @@ type Address struct {
 // ParseAddress parses an address string into a structured format with separate
 // scheme, host, port, and path portions, as well as the original input string.
 func ParseAddress(str string) (Address, error) {
-	httpPort, httpsPort := strconv.Itoa(certmagic.HTTPPort), strconv.Itoa(certmagic.HTTPSPort)
+	const maxLen = 4096
+	if len(str) > maxLen {
+		str = str[:maxLen]
+	}
+	remaining := strings.TrimSpace(str)
+	a := Address{Original: remaining}
 
-	input := str
-
-	// Split input into components (prepend with // to force host portion by default)
-	if !strings.Contains(str, "//") && !strings.HasPrefix(str, "/") {
-		str = "//" + str
+	// extract scheme
+	splitScheme := strings.SplitN(remaining, "://", 2)
+	switch len(splitScheme) {
+	case 0:
+		return a, nil
+	case 1:
+		remaining = splitScheme[0]
+	case 2:
+		a.Scheme = splitScheme[0]
+		remaining = splitScheme[1]
 	}
 
-	u, err := url.Parse(str)
-	if err != nil {
-		return Address{}, err
-	}
-
-	// separate host and port
-	host, port, err := net.SplitHostPort(u.Host)
-	if err != nil {
-		host, port, err = net.SplitHostPort(u.Host + ":")
+	// extract host and port
+	hostSplit := strings.SplitN(remaining, "/", 2)
+	if len(hostSplit) > 0 {
+		host, port, err := net.SplitHostPort(hostSplit[0])
 		if err != nil {
-			host = u.Host
+			host, port, err = net.SplitHostPort(hostSplit[0] + ":")
+			if err != nil {
+				host = hostSplit[0]
+			}
+		}
+		a.Host = host
+		a.Port = port
+	}
+	if len(hostSplit) == 2 {
+		// all that remains is the path
+		a.Path = "/" + hostSplit[1]
+	}
+
+	// make sure port is valid
+	if a.Port != "" {
+		if portNum, err := strconv.Atoi(a.Port); err != nil {
+			return Address{}, fmt.Errorf("invalid port '%s': %v", a.Port, err)
+		} else if portNum < 0 || portNum > 65535 {
+			return Address{}, fmt.Errorf("port %d is out of range", portNum)
 		}
 	}
 
-	// see if we can set port based off scheme
-	if port == "" {
-		if u.Scheme == "http" {
-			port = httpPort
-		} else if u.Scheme == "https" {
-			port = httpsPort
-		}
-	}
-
-	// error if scheme and port combination violate convention
-	if (u.Scheme == "http" && port == httpsPort) || (u.Scheme == "https" && port == httpPort) {
-		return Address{}, fmt.Errorf("[%s] scheme and port violate convention", input)
-	}
-
-	return Address{Original: input, Scheme: u.Scheme, Host: host, Port: port, Path: u.Path}, err
+	return a, nil
 }
 
-// TODO: which of the methods on Address are even used?
-
 // String returns a human-readable form of a. It will
 // be a cleaned-up and filled-out URL string.
 func (a Address) String() string {
@@ -285,20 +325,17 @@ func (a Address) String() string {
 // Normalize returns a normalized version of a.
 func (a Address) Normalize() Address {
 	path := a.Path
-	if !caseSensitivePath {
-		path = strings.ToLower(path)
-	}
 
 	// ensure host is normalized if it's an IP address
-	host := a.Host
+	host := strings.TrimSpace(a.Host)
 	if ip := net.ParseIP(host); ip != nil {
 		host = ip.String()
 	}
 
 	return Address{
 		Original: a.Original,
-		Scheme:   strings.ToLower(a.Scheme),
-		Host:     strings.ToLower(host),
+		Scheme:   lowerExceptPlaceholders(a.Scheme),
+		Host:     lowerExceptPlaceholders(host),
 		Port:     a.Port,
 		Path:     path,
 	}
@@ -326,9 +363,30 @@ func (a Address) Key() string {
 	return res
 }
 
-const (
-	// DefaultPort is the default port to use.
-	DefaultPort = "2015"
-
-	caseSensitivePath = false // TODO: Used?
-)
+// lowerExceptPlaceholders lowercases s except within
+// placeholders (substrings in non-escaped '{ }' spans).
+// See https://github.com/caddyserver/caddy/issues/3264
+func lowerExceptPlaceholders(s string) string {
+	var sb strings.Builder
+	var escaped, inPlaceholder bool
+	for _, ch := range s {
+		if ch == '\\' && !escaped {
+			escaped = true
+			sb.WriteRune(ch)
+			continue
+		}
+		if ch == '{' && !escaped {
+			inPlaceholder = true
+		}
+		if ch == '}' && inPlaceholder && !escaped {
+			inPlaceholder = false
+		}
+		if inPlaceholder {
+			sb.WriteRune(ch)
+		} else {
+			sb.WriteRune(unicode.ToLower(ch))
+		}
+		escaped = false
+	}
+	return sb.String()
+}

caddyconfig/httpcaddyfile/addresses_fuzz.go

@@ -13,7 +13,6 @@
 // limitations under the License.
 
 // +build gofuzz
-// +build gofuzz_libfuzzer
 
 package httpcaddyfile
 

caddyconfig/httpcaddyfile/addresses_test.go

@@ -1,7 +1,6 @@
 package httpcaddyfile
 
 import (
-	"strings"
 	"testing"
 )
 
@@ -11,6 +10,7 @@ func TestParseAddress(t *testing.T) {
 		scheme, host, port, path string
 		shouldErr                bool
 	}{
+		{``, "", "", "", "", false},
 		{`localhost`, "", "localhost", "", "", false},
 		{`localhost:1234`, "", "localhost", "1234", "", false},
 		{`localhost:`, "", "localhost", "", "", false},
@@ -27,25 +27,29 @@ func TestParseAddress(t *testing.T) {
 		{`http://localhost:https`, "", "", "", "", true}, // conflict
 		{`http://localhost:http`, "", "", "", "", true},  // repeated scheme
 		{`host:https/path`, "", "", "", "", true},
-		{`http://localhost:443`, "", "", "", "", true}, // not conventional
-		{`https://localhost:80`, "", "", "", "", true}, // not conventional
-		{`http://localhost`, "http", "localhost", "80", "", false},
-		{`https://localhost`, "https", "localhost", "443", "", false},
-		{`http://127.0.0.1`, "http", "127.0.0.1", "80", "", false},
-		{`https://127.0.0.1`, "https", "127.0.0.1", "443", "", false},
-		{`http://[::1]`, "http", "::1", "80", "", false},
+		{`http://localhost:443`, "http", "localhost", "443", "", false}, // NOTE: not conventional
+		{`https://localhost:80`, "https", "localhost", "80", "", false}, // NOTE: not conventional
+		{`http://localhost`, "http", "localhost", "", "", false},
+		{`https://localhost`, "https", "localhost", "", "", false},
+		{`http://{env.APP_DOMAIN}`, "http", "{env.APP_DOMAIN}", "", "", false},
+		{`{env.APP_DOMAIN}:80`, "", "{env.APP_DOMAIN}", "80", "", false},
+		{`{env.APP_DOMAIN}/path`, "", "{env.APP_DOMAIN}", "", "/path", false},
+		{`example.com/{env.APP_PATH}`, "", "example.com", "", "/{env.APP_PATH}", false},
+		{`http://127.0.0.1`, "http", "127.0.0.1", "", "", false},
+		{`https://127.0.0.1`, "https", "127.0.0.1", "", "", false},
+		{`http://[::1]`, "http", "::1", "", "", false},
 		{`http://localhost:1234`, "http", "localhost", "1234", "", false},
 		{`https://127.0.0.1:1234`, "https", "127.0.0.1", "1234", "", false},
 		{`http://[::1]:1234`, "http", "::1", "1234", "", false},
 		{``, "", "", "", "", false},
-		{`::1`, "", "::1", "", "", true},
-		{`localhost::`, "", "localhost::", "", "", true},
-		{`#$%@`, "", "", "", "", true},
+		{`::1`, "", "::1", "", "", false},
+		{`localhost::`, "", "localhost::", "", "", false},
+		{`#$%@`, "", "#$%@", "", "", false}, // don't want to presume what the hostname could be
 		{`host/path`, "", "host", "", "/path", false},
-		{`http://host/`, "http", "host", "80", "/", false},
-		{`//asdf`, "", "asdf", "", "", false},
+		{`http://host/`, "http", "host", "", "/", false},
+		{`//asdf`, "", "", "", "//asdf", false},
 		{`:1234/asdf`, "", "", "1234", "/asdf", false},
-		{`http://host/path`, "http", "host", "80", "/path", false},
+		{`http://host/path`, "http", "host", "", "/path", false},
 		{`https://host:443/path/foo`, "https", "host", "443", "/path/foo", false},
 		{`host:80/path`, "", "host", "80", "/path", false},
 		{`/path`, "", "", "", "/path", false},
@@ -56,7 +60,7 @@ func TestParseAddress(t *testing.T) {
 			t.Errorf("Test %d (%s): Expected no error, but had error: %v", i, test.input, err)
 		}
 		if err == nil && test.shouldErr {
-			t.Errorf("Test %d (%s): Expected error, but had none", i, test.input)
+			t.Errorf("Test %d (%s): Expected error, but had none (%#v)", i, test.input, actual)
 		}
 
 		if !test.shouldErr && actual.Original != test.input {
@@ -104,6 +108,10 @@ func TestKeyNormalization(t *testing.T) {
 		input  string
 		expect string
 	}{
+		{
+			input:  "example.com",
+			expect: "example.com",
+		},
 		{
 			input:  "http://host:1234/path",
 			expect: "http://host:1234/path",
@@ -120,6 +128,22 @@ func TestKeyNormalization(t *testing.T) {
 			input:  "A:2015/Path",
 			expect: "a:2015/Path",
 		},
+		{
+			input:  "sub.{env.MY_DOMAIN}",
+			expect: "sub.{env.MY_DOMAIN}",
+		},
+		{
+			input:  "sub.ExAmPle",
+			expect: "sub.example",
+		},
+		{
+			input:  "sub.\\{env.MY_DOMAIN\\}",
+			expect: "sub.\\{env.my_domain\\}",
+		},
+		{
+			input:  "sub.{env.MY_DOMAIN}.com",
+			expect: "sub.{env.MY_DOMAIN}.com",
+		},
 		{
 			input:  ":80",
 			expect: ":80",
@@ -151,15 +175,8 @@ func TestKeyNormalization(t *testing.T) {
 			t.Errorf("Test %d: Parsing address '%s': %v", i, tc.input, err)
 			continue
 		}
-		expect := tc.expect
-		if !caseSensitivePath {
-			// every other part of the address should be lowercased when normalized,
-			// so simply lower-case the whole thing to do case-insensitive comparison
-			// of the path as well
-			expect = strings.ToLower(expect)
-		}
-		if actual := addr.Normalize().Key(); actual != expect {
-			t.Errorf("Test %d: Normalized key for address '%s' was '%s' but expected '%s'", i, tc.input, actual, expect)
+		if actual := addr.Normalize().Key(); actual != tc.expect {
+			t.Errorf("Test %d: Input '%s': Expected '%s' but got '%s'", i, tc.input, tc.expect, actual)
 		}
 
 	}

caddyconfig/httpcaddyfile/builtins.go

@@ -15,25 +15,41 @@
 package httpcaddyfile
 
 import (
-	"encoding/json"
+	"encoding/base64"
+	"encoding/pem"
 	"fmt"
 	"html"
+	"io/ioutil"
 	"net/http"
 	"reflect"
+	"strings"
 
+	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 	"github.com/caddyserver/caddy/v2/modules/caddytls"
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
+	"go.uber.org/zap/zapcore"
 )
 
 func init() {
 	RegisterDirective("bind", parseBind)
-	RegisterDirective("root", parseRoot)
 	RegisterDirective("tls", parseTLS)
+	RegisterHandlerDirective("root", parseRoot)
 	RegisterHandlerDirective("redir", parseRedir)
 	RegisterHandlerDirective("respond", parseRespond)
+	RegisterHandlerDirective("route", parseRoute)
+	RegisterHandlerDirective("handle", parseHandle)
+	RegisterDirective("handle_errors", parseHandleErrors)
+	RegisterDirective("log", parseLog)
 }
 
+// parseBind parses the bind directive. Syntax:
+//
+//     bind <addresses...>
+//
 func parseBind(h Helper) ([]ConfigValue, error) {
 	var lnHosts []string
 	for h.Next() {
@@ -42,57 +58,38 @@ func parseBind(h Helper) ([]ConfigValue, error) {
 	return h.NewBindAddresses(lnHosts), nil
 }
 
-func parseRoot(h Helper) ([]ConfigValue, error) {
-	if !h.Next() {
-		return nil, h.ArgErr()
-	}
-
-	matcherSet, ok, err := h.MatcherToken()
-	if err != nil {
-		return nil, err
-	}
-	if !ok {
-		// no matcher token; oops
-		h.Dispenser.Prev()
-	}
-
-	if !h.NextArg() {
-		return nil, h.ArgErr()
-	}
-	root := h.Val()
-	if h.NextArg() {
-		return nil, h.ArgErr()
-	}
-
-	varsHandler := caddyhttp.VarsMiddleware{"root": root}
-	route := caddyhttp.Route{
-		HandlersRaw: []json.RawMessage{
-			caddyconfig.JSONModuleObject(varsHandler, "handler", "vars", nil),
-		},
-	}
-	if matcherSet != nil {
-		route.MatcherSetsRaw = []map[string]json.RawMessage{matcherSet}
-	}
-
-	return h.NewVarsRoute(route), nil
-}
-
+// parseTLS parses the tls directive. Syntax:
+//
+//     tls [<email>|internal]|[<cert_file> <key_file>] {
+//         protocols <min> [<max>]
+//         ciphers   <cipher_suites...>
+//         curves    <curves...>
+//         client_auth {
+//             mode                   [request|require|verify_if_given|require_and_verify]
+//             trusted_ca_cert        <base64_der>
+//             trusted_ca_cert_file   <filename>
+//             trusted_leaf_cert      <base64_der>
+//             trusted_leaf_cert_file <filename>
+//         }
+//         alpn      <values...>
+//         load      <paths...>
+//         ca        <acme_ca_endpoint>
+//         ca_root   <pem_file>
+//         dns       <provider_name>
+//         on_demand
+//         eab    <key_id> <mac_key>
+//         issuer <module_name> ...
+//     }
+//
 func parseTLS(h Helper) ([]ConfigValue, error) {
-	var configVals []ConfigValue
-
 	cp := new(caddytls.ConnectionPolicy)
 	var fileLoader caddytls.FileLoader
 	var folderLoader caddytls.FolderLoader
-	var mgr caddytls.ACMEManagerMaker
-	var off bool
-
-	// fill in global defaults, if configured
-	if email := h.Option("email"); email != nil {
-		mgr.Email = email.(string)
-	}
-	if acmeCA := h.Option("acme_ca"); acmeCA != nil {
-		mgr.CA = acmeCA.(string)
-	}
+	var certSelector caddytls.CustomCertSelectionPolicy
+	var acmeIssuer *caddytls.ACMEIssuer
+	var internalIssuer *caddytls.InternalIssuer
+	var issuer certmagic.Issuer
+	var onDemand bool
 
 	for h.Next() {
 		// file certificate loader
@@ -100,27 +97,71 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 		switch len(firstLine) {
 		case 0:
 		case 1:
-			if firstLine[0] == "off" {
-				off = true
+			if firstLine[0] == "internal" {
+				internalIssuer = new(caddytls.InternalIssuer)
+			} else if !strings.Contains(firstLine[0], "@") {
+				return nil, h.Err("single argument must either be 'internal' or an email address")
 			} else {
-				mgr.Email = firstLine[0]
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
+				acmeIssuer.Email = firstLine[0]
 			}
+
 		case 2:
-			fileLoader = append(fileLoader, caddytls.CertKeyFilePair{
-				Certificate: firstLine[0],
-				Key:         firstLine[1],
-				// TODO: add tags, for enterprise module's certificate selection
-			})
+			certFilename := firstLine[0]
+			keyFilename := firstLine[1]
+
+			// tag this certificate so if multiple certs match, specifically
+			// this one that the user has provided will be used, see #2588:
+			// https://github.com/caddyserver/caddy/issues/2588 ... but we
+			// must be careful about how we do this; being careless will
+			// lead to failed handshakes
+			//
+			// we need to remember which cert files we've seen, since we
+			// must load each cert only once; otherwise, they each get a
+			// different tag... since a cert loaded twice has the same
+			// bytes, it will overwrite the first one in the cache, and
+			// only the last cert (and its tag) will survive, so a any conn
+			// policy that is looking for any tag but the last one to be
+			// loaded won't find it, and TLS handshakes will fail (see end)
+			// of issue #3004)
+			//
+			// tlsCertTags maps certificate filenames to their tag.
+			// This is used to remember which tag is used for each
+			// certificate files, since we need to avoid loading
+			// the same certificate files more than once, overwriting
+			// previous tags
+			tlsCertTags, ok := h.State["tlsCertTags"].(map[string]string)
+			if !ok {
+				tlsCertTags = make(map[string]string)
+				h.State["tlsCertTags"] = tlsCertTags
+			}
+
+			tag, ok := tlsCertTags[certFilename]
+			if !ok {
+				// haven't seen this cert file yet, let's give it a tag
+				// and add a loader for it
+				tag = fmt.Sprintf("cert%d", len(tlsCertTags))
+				fileLoader = append(fileLoader, caddytls.CertKeyFilePair{
+					Certificate: certFilename,
+					Key:         keyFilename,
+					Tags:        []string{tag},
+				})
+				// remember this for next time we see this cert file
+				tlsCertTags[certFilename] = tag
+			}
+			certSelector.AnyTag = append(certSelector.AnyTag, tag)
+
 		default:
 			return nil, h.ArgErr()
 		}
 
 		var hasBlock bool
-		for h.NextBlock(0) {
+		for nesting := h.Nesting(); h.NextBlock(nesting); {
 			hasBlock = true
 
 			switch h.Val() {
-			// connection policy
 			case "protocols":
 				args := h.RemainingArgs()
 				if len(args) == 0 {
@@ -138,13 +179,15 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 					}
 					cp.ProtocolMax = args[1]
 				}
+
 			case "ciphers":
 				for h.NextArg() {
-					if _, ok := caddytls.SupportedCipherSuites[h.Val()]; !ok {
+					if !caddytls.CipherSuiteNameSupported(h.Val()) {
 						return nil, h.Errf("Wrong cipher suite name or cipher suite not supported: '%s'", h.Val())
 					}
 					cp.CipherSuites = append(cp.CipherSuites, h.Val())
 				}
+
 			case "curves":
 				for h.NextArg() {
 					if _, ok := caddytls.SupportedCurves[h.Val()]; !ok {
@@ -152,6 +195,58 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 					}
 					cp.Curves = append(cp.Curves, h.Val())
 				}
+
+			case "client_auth":
+				cp.ClientAuthentication = &caddytls.ClientAuthentication{}
+				for nesting := h.Nesting(); h.NextBlock(nesting); {
+					subdir := h.Val()
+					switch subdir {
+					case "mode":
+						if !h.Args(&cp.ClientAuthentication.Mode) {
+							return nil, h.ArgErr()
+						}
+						if h.NextArg() {
+							return nil, h.ArgErr()
+						}
+
+					case "trusted_ca_cert",
+						"trusted_leaf_cert":
+						if !h.NextArg() {
+							return nil, h.ArgErr()
+						}
+						if subdir == "trusted_ca_cert" {
+							cp.ClientAuthentication.TrustedCACerts = append(cp.ClientAuthentication.TrustedCACerts, h.Val())
+						} else {
+							cp.ClientAuthentication.TrustedLeafCerts = append(cp.ClientAuthentication.TrustedLeafCerts, h.Val())
+						}
+
+					case "trusted_ca_cert_file",
+						"trusted_leaf_cert_file":
+						if !h.NextArg() {
+							return nil, h.ArgErr()
+						}
+						filename := h.Val()
+						certDataPEM, err := ioutil.ReadFile(filename)
+						if err != nil {
+							return nil, err
+						}
+						block, _ := pem.Decode(certDataPEM)
+						if block == nil || block.Type != "CERTIFICATE" {
+							return nil, h.Errf("no CERTIFICATE pem block found in %s", h.Val())
+						}
+						if subdir == "trusted_ca_cert_file" {
+							cp.ClientAuthentication.TrustedCACerts = append(cp.ClientAuthentication.TrustedCACerts,
+								base64.StdEncoding.EncodeToString(block.Bytes))
+						} else {
+							cp.ClientAuthentication.TrustedLeafCerts = append(cp.ClientAuthentication.TrustedLeafCerts,
+								base64.StdEncoding.EncodeToString(block.Bytes))
+						}
+
+					default:
+						return nil, h.Errf("unknown subdirective for client_auth: %s", subdir)
+					}
+				}
+
 			case "alpn":
 				args := h.RemainingArgs()
 				if len(args) == 0 {
@@ -159,17 +254,94 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 				}
 				cp.ALPN = args
 
-			// certificate folder loader
 			case "load":
 				folderLoader = append(folderLoader, h.RemainingArgs()...)
 
-			// automation policy
 			case "ca":
 				arg := h.RemainingArgs()
 				if len(arg) != 1 {
 					return nil, h.ArgErr()
 				}
-				mgr.CA = arg[0]
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
+				acmeIssuer.CA = arg[0]
+
+			case "eab":
+				arg := h.RemainingArgs()
+				if len(arg) != 2 {
+					return nil, h.ArgErr()
+				}
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
+				acmeIssuer.ExternalAccount = &acme.EAB{
+					KeyID:  arg[0],
+					MACKey: arg[1],
+				}
+
+			case "issuer":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				modName := h.Val()
+				mod, err := caddy.GetModule("tls.issuance." + modName)
+				if err != nil {
+					return nil, h.Errf("getting issuer module '%s': %v", modName, err)
+				}
+				unm, ok := mod.New().(caddyfile.Unmarshaler)
+				if !ok {
+					return nil, h.Errf("issuer module '%s' is not a Caddyfile unmarshaler", mod.ID)
+				}
+				err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+				if err != nil {
+					return nil, err
+				}
+				issuer, ok = unm.(certmagic.Issuer)
+				if !ok {
+					return nil, h.Errf("module %s is not a certmagic.Issuer", mod.ID)
+				}
+
+			case "dns":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				provName := h.Val()
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
+				if acmeIssuer.Challenges == nil {
+					acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
+					acmeIssuer.Challenges.DNS = new(caddytls.DNSChallengeConfig)
+				}
+				dnsProvModule, err := caddy.GetModule("dns.providers." + provName)
+				if err != nil {
+					return nil, h.Errf("getting DNS provider module named '%s': %v", provName, err)
+				}
+				dnsProvModuleInstance := dnsProvModule.New()
+				if unm, ok := dnsProvModuleInstance.(caddyfile.Unmarshaler); ok {
+					err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+					if err != nil {
+						return nil, err
+					}
+				}
+				acmeIssuer.Challenges.DNS.ProviderRaw = caddyconfig.JSONModuleObject(dnsProvModuleInstance, "name", provName, h.warnings)
+
+			case "ca_root":
+				arg := h.RemainingArgs()
+				if len(arg) != 1 {
+					return nil, h.ArgErr()
+				}
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
+				acmeIssuer.TrustedRootsPEMFiles = append(acmeIssuer.TrustedRootsPEMFiles, arg[0])
+
+			case "on_demand":
+				if h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				onDemand = true
 
 			default:
 				return nil, h.Errf("unknown subdirective: %s", h.Val())
@@ -182,42 +354,110 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 		}
 	}
 
-	// connection policy
-	configVals = append(configVals, ConfigValue{
-		Class: "tls.connection_policy",
-		Value: cp,
-	})
+	// begin building the final config values
+	var configVals []ConfigValue
 
 	// certificate loaders
 	if len(fileLoader) > 0 {
 		configVals = append(configVals, ConfigValue{
-			Class: "tls.certificate_loader",
+			Class: "tls.cert_loader",
 			Value: fileLoader,
 		})
 	}
 	if len(folderLoader) > 0 {
 		configVals = append(configVals, ConfigValue{
-			Class: "tls.certificate_loader",
+			Class: "tls.cert_loader",
 			Value: folderLoader,
 		})
 	}
 
-	// automation policy
-	if off {
+	// issuer
+	if acmeIssuer != nil && internalIssuer != nil {
+		// the logic to support this would be complex
+		return nil, h.Err("cannot use both ACME and internal issuers in same server block")
+	}
+	if issuer != nil && (acmeIssuer != nil || internalIssuer != nil) {
+		// similarly, the logic to support this would be complex
+		return nil, h.Err("when defining an issuer, all its config must be in its block, rather than from separate tls subdirectives")
+	}
+	switch {
+	case issuer != nil:
 		configVals = append(configVals, ConfigValue{
-			Class: "tls.off",
-			Value: true,
+			Class: "tls.cert_issuer",
+			Value: issuer,
 		})
-	} else if !reflect.DeepEqual(mgr, caddytls.ACMEManagerMaker{}) {
+
+	case internalIssuer != nil:
 		configVals = append(configVals, ConfigValue{
-			Class: "tls.automation_manager",
-			Value: mgr,
+			Class: "tls.cert_issuer",
+			Value: internalIssuer,
+		})
+
+	case acmeIssuer != nil:
+		// fill in global defaults, if configured
+		if email := h.Option("email"); email != nil && acmeIssuer.Email == "" {
+			acmeIssuer.Email = email.(string)
+		}
+		if acmeCA := h.Option("acme_ca"); acmeCA != nil && acmeIssuer.CA == "" {
+			acmeIssuer.CA = acmeCA.(string)
+		}
+		if caPemFile := h.Option("acme_ca_root"); caPemFile != nil {
+			acmeIssuer.TrustedRootsPEMFiles = append(acmeIssuer.TrustedRootsPEMFiles, caPemFile.(string))
+		}
+		configVals = append(configVals, ConfigValue{
+			Class: "tls.cert_issuer",
+			Value: disambiguateACMEIssuer(acmeIssuer),
 		})
 	}
 
+	// on-demand TLS
+	if onDemand {
+		configVals = append(configVals, ConfigValue{
+			Class: "tls.on_demand",
+			Value: true,
+		})
+	}
+
+	// custom certificate selection
+	if len(certSelector.AnyTag) > 0 {
+		cp.CertSelection = &certSelector
+	}
+
+	// connection policy -- always add one, to ensure that TLS
+	// is enabled, because this directive was used (this is
+	// needed, for instance, when a site block has a key of
+	// just ":5000" - i.e. no hostname, and only on-demand TLS
+	// is enabled)
+	configVals = append(configVals, ConfigValue{
+		Class: "tls.connection_policy",
+		Value: cp,
+	})
+
 	return configVals, nil
 }
 
+// parseRoot parses the root directive. Syntax:
+//
+//     root [<matcher>] <path>
+//
+func parseRoot(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	var root string
+	for h.Next() {
+		if !h.NextArg() {
+			return nil, h.ArgErr()
+		}
+		root = h.Val()
+		if h.NextArg() {
+			return nil, h.ArgErr()
+		}
+	}
+	return caddyhttp.VarsMiddleware{"root": root}, nil
+}
+
+// parseRedir parses the redir directive. Syntax:
+//
+//     redir [<matcher>] <to> [<code>]
+//
 func parseRedir(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	if !h.Next() {
 		return nil, h.ArgErr()
@@ -236,10 +476,10 @@ func parseRedir(h Helper) (caddyhttp.MiddlewareHandler, error) {
 		code = "301"
 	}
 	if code == "temporary" || code == "" {
-		code = "307"
+		code = "302"
 	}
 	var body string
-	if code == "meta" {
+	if code == "html" {
 		// Script tag comes first since that will better imitate a redirect in the browser's
 		// history, but the meta tag is a fallback for most non-JS clients.
 		const metaRedir = `<!DOCTYPE html>
@@ -263,6 +503,7 @@ func parseRedir(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	}, nil
 }
 
+// parseRespond parses the respond directive.
 func parseRespond(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	sr := new(caddyhttp.StaticResponse)
 	err := sr.UnmarshalCaddyfile(h.Dispenser)
@@ -271,3 +512,162 @@ func parseRespond(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	}
 	return sr, nil
 }
+
+// parseRoute parses the route directive.
+func parseRoute(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	sr := new(caddyhttp.Subroute)
+
+	allResults, err := parseSegmentAsConfig(h)
+	if err != nil {
+		return nil, err
+	}
+
+	for _, result := range allResults {
+		switch handler := result.Value.(type) {
+		case caddyhttp.Route:
+			sr.Routes = append(sr.Routes, handler)
+		case caddyhttp.Subroute:
+			// directives which return a literal subroute instead of a route
+			// means they intend to keep those handlers together without
+			// them being reordered; we're doing that anyway since we're in
+			// the route directive, so just append its handlers
+			sr.Routes = append(sr.Routes, handler.Routes...)
+		default:
+			return nil, h.Errf("%s directive returned something other than an HTTP route or subroute: %#v (only handler directives can be used in routes)", result.directive, result.Value)
+		}
+	}
+
+	return sr, nil
+}
+
+func parseHandle(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	return ParseSegmentAsSubroute(h)
+}
+
+func parseHandleErrors(h Helper) ([]ConfigValue, error) {
+	subroute, err := ParseSegmentAsSubroute(h)
+	if err != nil {
+		return nil, err
+	}
+	return []ConfigValue{
+		{
+			Class: "error_route",
+			Value: subroute,
+		},
+	}, nil
+}
+
+// parseLog parses the log directive. Syntax:
+//
+//     log {
+//         output <writer_module> ...
+//         format <encoder_module> ...
+//         level  <level>
+//     }
+//
+func parseLog(h Helper) ([]ConfigValue, error) {
+	var configValues []ConfigValue
+	for h.Next() {
+		// log does not currently support any arguments
+		if h.NextArg() {
+			return nil, h.ArgErr()
+		}
+
+		cl := new(caddy.CustomLog)
+
+		for h.NextBlock(0) {
+			switch h.Val() {
+			case "output":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				moduleName := h.Val()
+
+				// can't use the usual caddyfile.Unmarshaler flow with the
+				// standard writers because they are in the caddy package
+				// (because they are the default) and implementing that
+				// interface there would unfortunately create circular import
+				var wo caddy.WriterOpener
+				switch moduleName {
+				case "stdout":
+					wo = caddy.StdoutWriter{}
+				case "stderr":
+					wo = caddy.StderrWriter{}
+				case "discard":
+					wo = caddy.DiscardWriter{}
+				default:
+					mod, err := caddy.GetModule("caddy.logging.writers." + moduleName)
+					if err != nil {
+						return nil, h.Errf("getting log writer module named '%s': %v", moduleName, err)
+					}
+					unm, ok := mod.New().(caddyfile.Unmarshaler)
+					if !ok {
+						return nil, h.Errf("log writer module '%s' is not a Caddyfile unmarshaler", mod)
+					}
+					err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+					if err != nil {
+						return nil, err
+					}
+					wo, ok = unm.(caddy.WriterOpener)
+					if !ok {
+						return nil, h.Errf("module %s is not a WriterOpener", mod)
+					}
+				}
+				cl.WriterRaw = caddyconfig.JSONModuleObject(wo, "output", moduleName, h.warnings)
+
+			case "format":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				moduleName := h.Val()
+				mod, err := caddy.GetModule("caddy.logging.encoders." + moduleName)
+				if err != nil {
+					return nil, h.Errf("getting log encoder module named '%s': %v", moduleName, err)
+				}
+				unm, ok := mod.New().(caddyfile.Unmarshaler)
+				if !ok {
+					return nil, h.Errf("log encoder module '%s' is not a Caddyfile unmarshaler", mod)
+				}
+				err = unm.UnmarshalCaddyfile(h.NewFromNextSegment())
+				if err != nil {
+					return nil, err
+				}
+				enc, ok := unm.(zapcore.Encoder)
+				if !ok {
+					return nil, h.Errf("module %s is not a zapcore.Encoder", mod)
+				}
+				cl.EncoderRaw = caddyconfig.JSONModuleObject(enc, "format", moduleName, h.warnings)
+
+			case "level":
+				if !h.NextArg() {
+					return nil, h.ArgErr()
+				}
+				cl.Level = h.Val()
+				if h.NextArg() {
+					return nil, h.ArgErr()
+				}
+
+			default:
+				return nil, h.Errf("unrecognized subdirective: %s", h.Val())
+			}
+		}
+
+		var val namedCustomLog
+		if !reflect.DeepEqual(cl, new(caddy.CustomLog)) {
+			logCounter, ok := h.State["logCounter"].(int)
+			if !ok {
+				logCounter = 0
+			}
+			val.name = fmt.Sprintf("log%d", logCounter)
+			cl.Include = []string{"http.log.access." + val.name}
+			val.log = cl
+			logCounter++
+			h.State["logCounter"] = logCounter
+		}
+		configValues = append(configValues, ConfigValue{
+			Class: "custom_log",
+			Value: val,
+		})
+	}
+	return configValues, nil
+}

caddyconfig/httpcaddyfile/builtins_test.go

@@ -0,0 +1,62 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	_ "github.com/caddyserver/caddy/v2/modules/logging"
+)
+
+func TestLogDirectiveSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectWarn  bool
+		expectError bool
+	}{
+		{
+			input: `:8080 {
+				log
+			}
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log {
+					output file foo.log
+				}
+			}
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log /foo {
+					output file foo.log
+				}
+			}
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if len(warnings) > 0 != tc.expectWarn {
+			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
+			continue
+		}
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+	}
+}

caddyconfig/httpcaddyfile/directives.go

@@ -16,6 +16,10 @@ package httpcaddyfile
 
 import (
 	"encoding/json"
+	"net"
+	"sort"
+	"strconv"
+	"strings"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
@@ -23,21 +27,58 @@ import (
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
 )
 
-// defaultDirectiveOrder specifies the order
+// directiveOrder specifies the order
 // to apply directives in HTTP routes.
-var defaultDirectiveOrder = []string{
+//
+// The root directive goes first in case rewrites or
+// redirects depend on existence of files, i.e. the
+// file matcher, which must know the root first.
+//
+// The header directive goes second so that headers
+// can be manipulated before doing redirects.
+var directiveOrder = []string{
+	"map",
+	"root",
+
+	"header",
+
+	"redir",
 	"rewrite",
+
+	// URI manipulation
+	"uri",
 	"try_files",
+
+	// middleware handlers; some wrap responses
 	"basicauth",
-	"headers",
 	"request_header",
 	"encode",
 	"templates",
-	"redir",
+
+	// special routing & dispatching directives
+	"handle",
+	"handle_path",
+	"route",
+	"push",
+
+	// handlers that typically respond to requests
 	"respond",
+	"metrics",
 	"reverse_proxy",
 	"php_fastcgi",
 	"file_server",
+	"acme_server",
+}
+
+// directiveIsOrdered returns true if dir is
+// a known, ordered (sorted) directive.
+func directiveIsOrdered(dir string) bool {
+	for _, d := range directiveOrder {
+		if d == dir {
+			return true
+		}
+	}
+	return false
 }
 
 // RegisterDirective registers a unique directive dir with an
@@ -53,6 +94,8 @@ func RegisterDirective(dir string, setupFunc UnmarshalFunc) {
 
 // RegisterHandlerDirective is like RegisterDirective, but for
 // directives which specifically output only an HTTP handler.
+// Directives registered with this function will always have
+// an optional matcher token as the first argument.
 func RegisterHandlerDirective(dir string, setupFunc UnmarshalHandlerFunc) {
 	RegisterDirective(dir, func(h Helper) ([]ConfigValue, error) {
 		if !h.Next() {
@@ -64,7 +107,12 @@ func RegisterHandlerDirective(dir string, setupFunc UnmarshalHandlerFunc) {
 			return nil, err
 		}
 		if ok {
-			h.Dispenser.Delete() // strip matcher token
+			// strip matcher token; we don't need to
+			// use the return value here because a
+			// new dispenser should have been made
+			// solely for this directive's tokens,
+			// with no other uses of same slice
+			h.Dispenser.Delete()
 		}
 
 		h.Dispenser.Reset() // pretend this lookahead never happened
@@ -77,14 +125,28 @@ func RegisterHandlerDirective(dir string, setupFunc UnmarshalHandlerFunc) {
 	})
 }
 
+// RegisterGlobalOption registers a unique global option opt with
+// an associated unmarshaling (setup) function. When the global
+// option opt is encountered in a Caddyfile, setupFunc will be
+// called to unmarshal its tokens.
+func RegisterGlobalOption(opt string, setupFunc UnmarshalGlobalFunc) {
+	if _, ok := registeredGlobalOptions[opt]; ok {
+		panic("global option " + opt + " already registered")
+	}
+	registeredGlobalOptions[opt] = setupFunc
+}
+
 // Helper is a type which helps setup a value from
 // Caddyfile tokens.
 type Helper struct {
 	*caddyfile.Dispenser
-	options     map[string]interface{}
-	warnings    *[]caddyconfig.Warning
-	matcherDefs map[string]map[string]json.RawMessage
-	parentBlock caddyfile.ServerBlock
+	// State stores intermediate variables during caddyfile adaptation.
+	State        map[string]interface{}
+	options      map[string]interface{}
+	warnings     *[]caddyconfig.Warning
+	matcherDefs  map[string]caddy.ModuleMap
+	parentBlock  caddyfile.ServerBlock
+	groupCounter counter
 }
 
 // Option gets the option keyed by name.
@@ -112,29 +174,51 @@ func (h Helper) Caddyfiles() []string {
 }
 
 // JSON converts val into JSON. Any errors are added to warnings.
-func (h Helper) JSON(val interface{}, warnings *[]caddyconfig.Warning) json.RawMessage {
+func (h Helper) JSON(val interface{}) json.RawMessage {
 	return caddyconfig.JSON(val, h.warnings)
 }
 
-// MatcherToken assumes the current token is (possibly) a matcher, and
-// if so, returns the matcher set along with a true value. If the current
+// MatcherToken assumes the next argument token is (possibly) a matcher,
+// and if so, returns the matcher set along with a true value. If the next
 // token is not a matcher, nil and false is returned. Note that a true
 // value may be returned with a nil matcher set if it is a catch-all.
-func (h Helper) MatcherToken() (map[string]json.RawMessage, bool, error) {
+func (h Helper) MatcherToken() (caddy.ModuleMap, bool, error) {
 	if !h.NextArg() {
 		return nil, false, nil
 	}
 	return matcherSetFromMatcherToken(h.Dispenser.Token(), h.matcherDefs, h.warnings)
 }
 
-// NewRoute returns config values relevant to creating a new HTTP route.
-func (h Helper) NewRoute(matcherSet map[string]json.RawMessage,
-	handler caddyhttp.MiddlewareHandler) []ConfigValue {
-	mod, err := caddy.GetModule(caddy.GetModuleName(handler))
+// ExtractMatcherSet is like MatcherToken, except this is a higher-level
+// method that returns the matcher set described by the matcher token,
+// or nil if there is none, and deletes the matcher token from the
+// dispenser and resets it as if this look-ahead never happened. Useful
+// when wrapping a route (one or more handlers) in a user-defined matcher.
+func (h Helper) ExtractMatcherSet() (caddy.ModuleMap, error) {
+	matcherSet, hasMatcher, err := h.MatcherToken()
 	if err != nil {
-		// TODO: append to warnings
+		return nil, err
 	}
-	var matcherSetsRaw []map[string]json.RawMessage
+	if hasMatcher {
+		h.Dispenser.Delete() // strip matcher token
+	}
+	h.Dispenser.Reset() // pretend this lookahead never happened
+	return matcherSet, nil
+}
+
+// NewRoute returns config values relevant to creating a new HTTP route.
+func (h Helper) NewRoute(matcherSet caddy.ModuleMap,
+	handler caddyhttp.MiddlewareHandler) []ConfigValue {
+	mod, err := caddy.GetModule(caddy.GetModuleID(handler))
+	if err != nil {
+		*h.warnings = append(*h.warnings, caddyconfig.Warning{
+			File:    h.File(),
+			Line:    h.Line(),
+			Message: err.Error(),
+		})
+		return nil
+	}
+	var matcherSetsRaw []caddy.ModuleMap
 	if matcherSet != nil {
 		matcherSetsRaw = append(matcherSetsRaw, matcherSet)
 	}
@@ -143,22 +227,121 @@ func (h Helper) NewRoute(matcherSet map[string]json.RawMessage,
 			Class: "route",
 			Value: caddyhttp.Route{
 				MatcherSetsRaw: matcherSetsRaw,
-				HandlersRaw:    []json.RawMessage{caddyconfig.JSONModuleObject(handler, "handler", mod.ID(), h.warnings)},
+				HandlersRaw:    []json.RawMessage{caddyconfig.JSONModuleObject(handler, "handler", mod.ID.Name(), h.warnings)},
 			},
 		},
 	}
 }
 
+// GroupRoutes adds the routes (caddyhttp.Route type) in vals to the
+// same group, if there is more than one route in vals.
+func (h Helper) GroupRoutes(vals []ConfigValue) {
+	// ensure there's at least two routes; group of one is pointless
+	var count int
+	for _, v := range vals {
+		if _, ok := v.Value.(caddyhttp.Route); ok {
+			count++
+			if count > 1 {
+				break
+			}
+		}
+	}
+	if count < 2 {
+		return
+	}
+
+	// now that we know the group will have some effect, do it
+	groupName := h.groupCounter.nextGroup()
+	for i := range vals {
+		if route, ok := vals[i].Value.(caddyhttp.Route); ok {
+			route.Group = groupName
+			vals[i].Value = route
+		}
+	}
+}
+
 // NewBindAddresses returns config values relevant to adding
 // listener bind addresses to the config.
 func (h Helper) NewBindAddresses(addrs []string) []ConfigValue {
 	return []ConfigValue{{Class: "bind", Value: addrs}}
 }
 
-// NewVarsRoute returns config values relevant to adding a
-// "vars" wrapper route to the config.
-func (h Helper) NewVarsRoute(route caddyhttp.Route) []ConfigValue {
-	return []ConfigValue{{Class: "var", Value: route}}
+// ParseSegmentAsSubroute parses the segment such that its subdirectives
+// are themselves treated as directives, from which a subroute is built
+// and returned.
+func ParseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	allResults, err := parseSegmentAsConfig(h)
+	if err != nil {
+		return nil, err
+	}
+
+	return buildSubroute(allResults, h.groupCounter)
+}
+
+// parseSegmentAsConfig parses the segment such that its subdirectives
+// are themselves treated as directives, including named matcher definitions,
+// and the raw Config structs are returned.
+func parseSegmentAsConfig(h Helper) ([]ConfigValue, error) {
+	var allResults []ConfigValue
+
+	for h.Next() {
+		// don't allow non-matcher args on the first line
+		if h.NextArg() {
+			return nil, h.ArgErr()
+		}
+
+		// slice the linear list of tokens into top-level segments
+		var segments []caddyfile.Segment
+		for nesting := h.Nesting(); h.NextBlock(nesting); {
+			segments = append(segments, h.NextSegment())
+		}
+
+		// copy existing matcher definitions so we can augment
+		// new ones that are defined only in this scope
+		matcherDefs := make(map[string]caddy.ModuleMap, len(h.matcherDefs))
+		for key, val := range h.matcherDefs {
+			matcherDefs[key] = val
+		}
+
+		// find and extract any embedded matcher definitions in this scope
+		for i := 0; i < len(segments); i++ {
+			seg := segments[i]
+			if strings.HasPrefix(seg.Directive(), matcherPrefix) {
+				// parse, then add the matcher to matcherDefs
+				err := parseMatcherDefinitions(caddyfile.NewDispenser(seg), matcherDefs)
+				if err != nil {
+					return nil, err
+				}
+				// remove the matcher segment (consumed), then step back the loop
+				segments = append(segments[:i], segments[i+1:]...)
+				i--
+			}
+		}
+
+		// with matchers ready to go, evaluate each directive's segment
+		for _, seg := range segments {
+			dir := seg.Directive()
+			dirFunc, ok := registeredDirectives[dir]
+			if !ok {
+				return nil, h.Errf("unrecognized directive: %s", dir)
+			}
+
+			subHelper := h
+			subHelper.Dispenser = caddyfile.NewDispenser(seg)
+			subHelper.matcherDefs = matcherDefs
+
+			results, err := dirFunc(subHelper)
+			if err != nil {
+				return nil, h.Errf("parsing caddyfile tokens for '%s': %v", dir, err)
+			}
+			for _, result := range results {
+				result.directive = dir
+				allResults = append(allResults, result)
+			}
+		}
+	}
+
+	return allResults, nil
 }
 
 // ConfigValue represents a value to be added to the final
@@ -181,12 +364,123 @@ type ConfigValue struct {
 	directive string
 }
 
-// serverBlock pairs a Caddyfile server block
-// with a "pile" of config values, keyed by class
-// name.
+func sortRoutes(routes []ConfigValue) {
+	dirPositions := make(map[string]int)
+	for i, dir := range directiveOrder {
+		dirPositions[dir] = i
+	}
+
+	sort.SliceStable(routes, func(i, j int) bool {
+		// if the directives are different, just use the established directive order
+		iDir, jDir := routes[i].directive, routes[j].directive
+		if iDir != jDir {
+			return dirPositions[iDir] < dirPositions[jDir]
+		}
+
+		// directives are the same; sub-sort by path matcher length if there's
+		// only one matcher set and one path (this is a very common case and
+		// usually -- but not always -- helpful/expected, oh well; user can
+		// always take manual control of order using handler or route blocks)
+		iRoute, ok := routes[i].Value.(caddyhttp.Route)
+		if !ok {
+			return false
+		}
+		jRoute, ok := routes[j].Value.(caddyhttp.Route)
+		if !ok {
+			return false
+		}
+
+		// decode the path matchers, if there is just one of them
+		var iPM, jPM caddyhttp.MatchPath
+		if len(iRoute.MatcherSetsRaw) == 1 {
+			_ = json.Unmarshal(iRoute.MatcherSetsRaw[0]["path"], &iPM)
+		}
+		if len(jRoute.MatcherSetsRaw) == 1 {
+			_ = json.Unmarshal(jRoute.MatcherSetsRaw[0]["path"], &jPM)
+		}
+
+		// sort by longer path (more specific) first; missing path
+		// matchers or multi-matchers are treated as zero-length paths
+		var iPathLen, jPathLen int
+		if len(iPM) > 0 {
+			iPathLen = len(iPM[0])
+		}
+		if len(jPM) > 0 {
+			jPathLen = len(jPM[0])
+		}
+
+		// if both directives have no path matcher, use whichever one
+		// has any kind of matcher defined first.
+		if iPathLen == 0 && jPathLen == 0 {
+			return len(iRoute.MatcherSetsRaw) > 0 && len(jRoute.MatcherSetsRaw) == 0
+		}
+
+		// sort with the most-specific (longest) path first
+		return iPathLen > jPathLen
+	})
+}
+
+// serverBlock pairs a Caddyfile server block with
+// a "pile" of config values, keyed by class name,
+// as well as its parsed keys for convenience.
 type serverBlock struct {
 	block caddyfile.ServerBlock
 	pile  map[string][]ConfigValue // config values obtained from directives
+	keys  []Address
+}
+
+// hostsFromKeys returns a list of all the non-empty hostnames found in
+// the keys of the server block sb. If logger mode is false, a key with
+// an empty hostname portion will return an empty slice, since that
+// server block is interpreted to effectively match all hosts. An empty
+// string is never added to the slice.
+//
+// If loggerMode is true, then the non-standard ports of keys will be
+// joined to the hostnames. This is to effectively match the Host
+// header of requests that come in for that key.
+//
+// The resulting slice is not sorted but will never have duplicates.
+func (sb serverBlock) hostsFromKeys(loggerMode bool) []string {
+	// ensure each entry in our list is unique
+	hostMap := make(map[string]struct{})
+	for _, addr := range sb.keys {
+		if addr.Host == "" {
+			if !loggerMode {
+				// server block contains a key like ":443", i.e. the host portion
+				// is empty / catch-all, which means to match all hosts
+				return []string{}
+			}
+			// never append an empty string
+			continue
+		}
+		if loggerMode &&
+			addr.Port != "" &&
+			addr.Port != strconv.Itoa(caddyhttp.DefaultHTTPPort) &&
+			addr.Port != strconv.Itoa(caddyhttp.DefaultHTTPSPort) {
+			hostMap[net.JoinHostPort(addr.Host, addr.Port)] = struct{}{}
+		} else {
+			hostMap[addr.Host] = struct{}{}
+		}
+	}
+
+	// convert map to slice
+	sblockHosts := make([]string, 0, len(hostMap))
+	for host := range hostMap {
+		sblockHosts = append(sblockHosts, host)
+	}
+
+	return sblockHosts
+}
+
+// hasHostCatchAllKey returns true if sb has a key that
+// omits a host portion, i.e. it "catches all" hosts.
+func (sb serverBlock) hasHostCatchAllKey() bool {
+	for _, addr := range sb.keys {
+		if addr.Host == "" {
+			return true
+		}
+	}
+	return false
 }
 
 type (
@@ -205,6 +499,13 @@ type (
 	// for you. These are passed to a call to
 	// RegisterHandlerDirective.
 	UnmarshalHandlerFunc func(h Helper) (caddyhttp.MiddlewareHandler, error)
+
+	// UnmarshalGlobalFunc is a function which can unmarshal Caddyfile
+	// tokens into a global option config value using a Helper type.
+	// These are passed in a call to RegisterGlobalOption.
+	UnmarshalGlobalFunc func(d *caddyfile.Dispenser) (interface{}, error)
 )
 
 var registeredDirectives = make(map[string]UnmarshalFunc)
+
+var registeredGlobalOptions = make(map[string]UnmarshalGlobalFunc)

caddyconfig/httpcaddyfile/directives_test.go

@@ -0,0 +1,94 @@
+package httpcaddyfile
+
+import (
+	"reflect"
+	"sort"
+	"testing"
+)
+
+func TestHostsFromKeys(t *testing.T) {
+	for i, tc := range []struct {
+		keys             []Address
+		expectNormalMode []string
+		expectLoggerMode []string
+	}{
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+			},
+			[]string{"foo"},
+			[]string{"foo"},
+		},
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+				{Original: "bar", Host: "bar"},
+			},
+			[]string{"bar", "foo"},
+			[]string{"bar", "foo"},
+		},
+		{
+			[]Address{
+				{Original: ":2015", Port: "2015"},
+			},
+			[]string{}, []string{},
+		},
+		{
+			[]Address{
+				{Original: ":443", Port: "443"},
+			},
+			[]string{}, []string{},
+		},
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+				{Original: ":2015", Port: "2015"},
+			},
+			[]string{}, []string{"foo"},
+		},
+		{
+			[]Address{
+				{Original: "example.com:2015", Host: "example.com", Port: "2015"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com:2015"},
+		},
+		{
+			[]Address{
+				{Original: "example.com:80", Host: "example.com", Port: "80"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com"},
+		},
+		{
+			[]Address{
+				{Original: "https://:2015/foo", Scheme: "https", Port: "2015", Path: "/foo"},
+			},
+			[]string{},
+			[]string{},
+		},
+		{
+			[]Address{
+				{Original: "https://example.com:2015/foo", Scheme: "https", Host: "example.com", Port: "2015", Path: "/foo"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com:2015"},
+		},
+	} {
+		sb := serverBlock{keys: tc.keys}
+
+		// test in normal mode
+		actual := sb.hostsFromKeys(false)
+		sort.Strings(actual)
+		if !reflect.DeepEqual(tc.expectNormalMode, actual) {
+			t.Errorf("Test %d (loggerMode=false): Expected: %v Actual: %v", i, tc.expectNormalMode, actual)
+		}
+
+		// test in logger mode
+		actual = sb.hostsFromKeys(true)
+		sort.Strings(actual)
+		if !reflect.DeepEqual(tc.expectLoggerMode, actual) {
+			t.Errorf("Test %d (loggerMode=true): Expected: %v Actual: %v", i, tc.expectLoggerMode, actual)
+		}
+	}
+}

caddyconfig/httpcaddyfile/httptype_test.go

@@ -0,0 +1,237 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+)
+
+func TestMatcherSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectWarn  bool
+		expectError bool
+	}{
+		{
+			input: `http://localhost
+			@debug {
+				query showdebug=1
+			}
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `http://localhost
+			@debug {
+				query bad format
+			}
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
+		{
+			input: `http://localhost
+			@debug {
+				not {
+					path /somepath*
+				}
+			}
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `http://localhost
+			@debug {
+				not path /somepath*
+			}
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `http://localhost
+			@debug not path /somepath*
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `@matcher {
+				path /matcher-not-allowed/outside-of-site-block/*
+			}
+			http://localhost
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if len(warnings) > 0 != tc.expectWarn {
+			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
+			continue
+		}
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+	}
+}
+
+func TestSpecificity(t *testing.T) {
+	for i, tc := range []struct {
+		input  string
+		expect int
+	}{
+		{"", 0},
+		{"*", 0},
+		{"*.*", 1},
+		{"{placeholder}", 0},
+		{"/{placeholder}", 1},
+		{"foo", 3},
+		{"example.com", 11},
+		{"a.example.com", 13},
+		{"*.example.com", 12},
+		{"/foo", 4},
+		{"/foo*", 4},
+		{"{placeholder}.example.com", 12},
+		{"{placeholder.example.com", 24},
+		{"}.", 2},
+		{"}{", 2},
+		{"{}", 0},
+		{"{{{}}", 1},
+	} {
+		actual := specificity(tc.input)
+		if actual != tc.expect {
+			t.Errorf("Test %d (%s): Expected %d but got %d", i, tc.input, tc.expect, actual)
+		}
+	}
+}
+
+func TestGlobalOptions(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectWarn  bool
+		expectError bool
+	}{
+		{
+			input: `
+				{
+					email test@example.com
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin off
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 127.0.0.1:2020
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin {
+						disabled false
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
+		{
+			input: `
+				{
+					admin {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 127.0.0.1:2020 {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 192.168.1.1:2020 127.0.0.1:2020 {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
+		{
+			input: `
+				{
+					admin off {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectWarn:  false,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if len(warnings) > 0 != tc.expectWarn {
+			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
+			continue
+		}
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+	}
+}

caddyconfig/httpcaddyfile/options.go

@@ -15,14 +15,41 @@
 package httpcaddyfile
 
 import (
-	"fmt"
 	"strconv"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	"github.com/caddyserver/caddy/v2/modules/caddytls"
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
 )
 
-func parseOptHTTPPort(d *caddyfile.Dispenser) (int, error) {
+func init() {
+	RegisterGlobalOption("debug", parseOptTrue)
+	RegisterGlobalOption("http_port", parseOptHTTPPort)
+	RegisterGlobalOption("https_port", parseOptHTTPSPort)
+	RegisterGlobalOption("default_sni", parseOptSingleString)
+	RegisterGlobalOption("order", parseOptOrder)
+	RegisterGlobalOption("experimental_http3", parseOptTrue)
+	RegisterGlobalOption("storage", parseOptStorage)
+	RegisterGlobalOption("acme_ca", parseOptSingleString)
+	RegisterGlobalOption("acme_ca_root", parseOptSingleString)
+	RegisterGlobalOption("acme_dns", parseOptSingleString)
+	RegisterGlobalOption("acme_eab", parseOptACMEEAB)
+	RegisterGlobalOption("cert_issuer", parseOptCertIssuer)
+	RegisterGlobalOption("email", parseOptSingleString)
+	RegisterGlobalOption("admin", parseOptAdmin)
+	RegisterGlobalOption("on_demand_tls", parseOptOnDemand)
+	RegisterGlobalOption("local_certs", parseOptTrue)
+	RegisterGlobalOption("key_type", parseOptSingleString)
+	RegisterGlobalOption("auto_https", parseOptAutoHTTPS)
+}
+
+func parseOptTrue(d *caddyfile.Dispenser) (interface{}, error) {
+	return true, nil
+}
+
+func parseOptHTTPPort(d *caddyfile.Dispenser) (interface{}, error) {
 	var httpPort int
 	for d.Next() {
 		var httpPortStr string
@@ -38,7 +65,7 @@ func parseOptHTTPPort(d *caddyfile.Dispenser) (int, error) {
 	return httpPort, nil
 }
 
-func parseOptHTTPSPort(d *caddyfile.Dispenser) (int, error) {
+func parseOptHTTPSPort(d *caddyfile.Dispenser) (interface{}, error) {
 	var httpsPort int
 	for d.Next() {
 		var httpsPortStr string
@@ -54,62 +81,165 @@ func parseOptHTTPSPort(d *caddyfile.Dispenser) (int, error) {
 	return httpsPort, nil
 }
 
-func parseOptExperimentalHTTP3(d *caddyfile.Dispenser) (bool, error) {
-	return true, nil
-}
+func parseOptOrder(d *caddyfile.Dispenser) (interface{}, error) {
+	newOrder := directiveOrder
 
-func parseOptHandlerOrder(d *caddyfile.Dispenser) ([]string, error) {
-	if !d.Next() {
-		return nil, d.ArgErr()
-	}
-	order := d.RemainingArgs()
-	if len(order) == 1 && order[0] == "appearance" {
-		return []string{"appearance"}, nil
-	}
-	if len(order) > 0 && d.NextBlock(0) {
-		return nil, d.Err("cannot open block if there are arguments")
-	}
-	for d.NextBlock(0) {
-		order = append(order, d.Val())
+	for d.Next() {
+		// get directive name
+		if !d.Next() {
+			return nil, d.ArgErr()
+		}
+		dirName := d.Val()
+		if _, ok := registeredDirectives[dirName]; !ok {
+			return nil, d.Errf("%s is not a registered directive", dirName)
+		}
+
+		// get positional token
+		if !d.Next() {
+			return nil, d.ArgErr()
+		}
+		pos := d.Val()
+
+		// if directive exists, first remove it
+		for i, d := range newOrder {
+			if d == dirName {
+				newOrder = append(newOrder[:i], newOrder[i+1:]...)
+				break
+			}
+		}
+
+		// act on the positional
+		switch pos {
+		case "first":
+			newOrder = append([]string{dirName}, newOrder...)
+			if d.NextArg() {
+				return nil, d.ArgErr()
+			}
+			directiveOrder = newOrder
+			return newOrder, nil
+		case "last":
+			newOrder = append(newOrder, dirName)
+			if d.NextArg() {
+				return nil, d.ArgErr()
+			}
+			directiveOrder = newOrder
+			return newOrder, nil
+		case "before":
+		case "after":
+		default:
+			return nil, d.Errf("unknown positional '%s'", pos)
+		}
+
+		// get name of other directive
+		if !d.NextArg() {
+			return nil, d.ArgErr()
+		}
+		otherDir := d.Val()
 		if d.NextArg() {
 			return nil, d.ArgErr()
 		}
+
+		// insert directive into proper position
+		for i, d := range newOrder {
+			if d == otherDir {
+				if pos == "before" {
+					newOrder = append(newOrder[:i], append([]string{dirName}, newOrder[i:]...)...)
+				} else if pos == "after" {
+					newOrder = append(newOrder[:i+1], append([]string{dirName}, newOrder[i+1:]...)...)
+				}
+				break
+			}
+		}
 	}
-	if len(order) == 0 {
-		return nil, d.ArgErr()
-	}
-	return order, nil
+
+	directiveOrder = newOrder
+
+	return newOrder, nil
 }
 
-func parseOptStorage(d *caddyfile.Dispenser) (caddy.StorageConverter, error) {
-	if !d.Next() {
+func parseOptStorage(d *caddyfile.Dispenser) (interface{}, error) {
+	if !d.Next() { // consume option name
 		return nil, d.ArgErr()
 	}
-	args := d.RemainingArgs()
-	if len(args) != 1 {
+	if !d.Next() { // get storage module name
 		return nil, d.ArgErr()
 	}
-	modName := args[0]
+	modName := d.Val()
 	mod, err := caddy.GetModule("caddy.storage." + modName)
 	if err != nil {
-		return nil, fmt.Errorf("getting storage module '%s': %v", modName, err)
+		return nil, d.Errf("getting storage module '%s': %v", modName, err)
 	}
 	unm, ok := mod.New().(caddyfile.Unmarshaler)
 	if !ok {
-		return nil, fmt.Errorf("storage module '%s' is not a Caddyfile unmarshaler", mod.Name)
+		return nil, d.Errf("storage module '%s' is not a Caddyfile unmarshaler", mod.ID)
 	}
-	err = unm.UnmarshalCaddyfile(d.NewFromNextTokens())
+	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
 	if err != nil {
 		return nil, err
 	}
 	storage, ok := unm.(caddy.StorageConverter)
 	if !ok {
-		return nil, fmt.Errorf("module %s is not a StorageConverter", mod.Name)
+		return nil, d.Errf("module %s is not a StorageConverter", mod.ID)
 	}
 	return storage, nil
 }
 
-func parseOptACMECA(d *caddyfile.Dispenser) (string, error) {
+func parseOptACMEEAB(d *caddyfile.Dispenser) (interface{}, error) {
+	eab := new(acme.EAB)
+	for d.Next() {
+		if d.NextArg() {
+			return nil, d.ArgErr()
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "key_id":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				eab.KeyID = d.Val()
+
+			case "mac_key":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				eab.MACKey = d.Val()
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
+		}
+	}
+	return eab, nil
+}
+
+func parseOptCertIssuer(d *caddyfile.Dispenser) (interface{}, error) {
+	if !d.Next() { // consume option name
+		return nil, d.ArgErr()
+	}
+	if !d.Next() { // get issuer module name
+		return nil, d.ArgErr()
+	}
+	modName := d.Val()
+	mod, err := caddy.GetModule("tls.issuance." + modName)
+	if err != nil {
+		return nil, d.Errf("getting issuer module '%s': %v", modName, err)
+	}
+	unm, ok := mod.New().(caddyfile.Unmarshaler)
+	if !ok {
+		return nil, d.Errf("issuer module '%s' is not a Caddyfile unmarshaler", mod.ID)
+	}
+	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
+	if err != nil {
+		return nil, err
+	}
+	iss, ok := unm.(certmagic.Issuer)
+	if !ok {
+		return nil, d.Errf("module %s is not a certmagic.Issuer", mod.ID)
+	}
+	return iss, nil
+}
+
+func parseOptSingleString(d *caddyfile.Dispenser) (interface{}, error) {
 	d.Next() // consume parameter name
 	if !d.Next() {
 		return "", d.ArgErr()
@@ -121,7 +251,103 @@ func parseOptACMECA(d *caddyfile.Dispenser) (string, error) {
 	return val, nil
 }
 
-func parseOptEmail(d *caddyfile.Dispenser) (string, error) {
+func parseOptAdmin(d *caddyfile.Dispenser) (interface{}, error) {
+	adminCfg := new(caddy.AdminConfig)
+	for d.Next() {
+		if d.NextArg() {
+			listenAddress := d.Val()
+			if listenAddress == "off" {
+				adminCfg.Disabled = true
+				if d.Next() { // Do not accept any remaining options including block
+					return nil, d.Err("No more option is allowed after turning off admin config")
+				}
+			} else {
+				adminCfg.Listen = listenAddress
+				if d.NextArg() { // At most 1 arg is allowed
+					return nil, d.ArgErr()
+				}
+			}
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "enforce_origin":
+				adminCfg.EnforceOrigin = true
+
+			case "origins":
+				adminCfg.Origins = d.RemainingArgs()
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
+		}
+	}
+	if adminCfg.Listen == "" && !adminCfg.Disabled {
+		adminCfg.Listen = caddy.DefaultAdminListen
+	}
+	return adminCfg, nil
+}
+
+func parseOptOnDemand(d *caddyfile.Dispenser) (interface{}, error) {
+	var ond *caddytls.OnDemandConfig
+	for d.Next() {
+		if d.NextArg() {
+			return nil, d.ArgErr()
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "ask":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				if ond == nil {
+					ond = new(caddytls.OnDemandConfig)
+				}
+				ond.Ask = d.Val()
+
+			case "interval":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				dur, err := caddy.ParseDuration(d.Val())
+				if err != nil {
+					return nil, err
+				}
+				if ond == nil {
+					ond = new(caddytls.OnDemandConfig)
+				}
+				if ond.RateLimit == nil {
+					ond.RateLimit = new(caddytls.RateLimit)
+				}
+				ond.RateLimit.Interval = caddy.Duration(dur)
+
+			case "burst":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				burst, err := strconv.Atoi(d.Val())
+				if err != nil {
+					return nil, err
+				}
+				if ond == nil {
+					ond = new(caddytls.OnDemandConfig)
+				}
+				if ond.RateLimit == nil {
+					ond.RateLimit = new(caddytls.RateLimit)
+				}
+				ond.RateLimit.Burst = burst
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
+		}
+	}
+	if ond == nil {
+		return nil, d.Err("expected at least one config parameter for on_demand_tls")
+	}
+	return ond, nil
+}
+
+func parseOptAutoHTTPS(d *caddyfile.Dispenser) (interface{}, error) {
 	d.Next() // consume parameter name
 	if !d.Next() {
 		return "", d.ArgErr()
@@ -130,5 +356,8 @@ func parseOptEmail(d *caddyfile.Dispenser) (string, error) {
 	if d.Next() {
 		return "", d.ArgErr()
 	}
+	if val != "off" && val != "disable_redirects" {
+		return "", d.Errf("auto_https must be either 'off' or 'disable_redirects'")
+	}
 	return val, nil
 }

caddyconfig/httpcaddyfile/tlsapp.go

@@ -0,0 +1,491 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package httpcaddyfile
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"reflect"
+	"sort"
+	"strconv"
+	"strings"
+
+	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
+	"github.com/caddyserver/caddy/v2/modules/caddytls"
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
+)
+
+func (st ServerType) buildTLSApp(
+	pairings []sbAddrAssociation,
+	options map[string]interface{},
+	warnings []caddyconfig.Warning,
+) (*caddytls.TLS, []caddyconfig.Warning, error) {
+
+	tlsApp := &caddytls.TLS{CertificatesRaw: make(caddy.ModuleMap)}
+	var certLoaders []caddytls.CertificateLoader
+
+	httpsPort := strconv.Itoa(caddyhttp.DefaultHTTPSPort)
+	if hsp, ok := options["https_port"].(int); ok {
+		httpsPort = strconv.Itoa(hsp)
+	}
+
+	// count how many server blocks have a TLS-enabled key with
+	// no host, and find all hosts that share a server block with
+	// a hostless key, so that they don't get forgotten/omitted
+	// by auto-HTTPS (since they won't appear in route matchers)
+	var serverBlocksWithTLSHostlessKey int
+	hostsSharedWithHostlessKey := make(map[string]struct{})
+	for _, pair := range pairings {
+		for _, sb := range pair.serverBlocks {
+			for _, addr := range sb.keys {
+				if addr.Host == "" {
+					// this address has no hostname, but if it's explicitly set
+					// to HTTPS, then we need to count it as being TLS-enabled
+					if addr.Scheme == "https" || addr.Port == httpsPort {
+						serverBlocksWithTLSHostlessKey++
+					}
+					// this server block has a hostless key, now
+					// go through and add all the hosts to the set
+					for _, otherAddr := range sb.keys {
+						if otherAddr.Original == addr.Original {
+							continue
+						}
+						if otherAddr.Host != "" {
+							hostsSharedWithHostlessKey[otherAddr.Host] = struct{}{}
+						}
+					}
+					break
+				}
+			}
+		}
+	}
+
+	catchAllAP, err := newBaseAutomationPolicy(options, warnings, false)
+	if err != nil {
+		return nil, warnings, err
+	}
+
+	for _, p := range pairings {
+		for _, sblock := range p.serverBlocks {
+			// get values that populate an automation policy for this block
+			var ap *caddytls.AutomationPolicy
+
+			sblockHosts := sblock.hostsFromKeys(false)
+			if len(sblockHosts) == 0 {
+				ap = catchAllAP
+			}
+
+			// on-demand tls
+			if _, ok := sblock.pile["tls.on_demand"]; ok {
+				if ap == nil {
+					var err error
+					ap, err = newBaseAutomationPolicy(options, warnings, true)
+					if err != nil {
+						return nil, warnings, err
+					}
+				}
+				ap.OnDemand = true
+			}
+
+			// certificate issuers
+			if issuerVals, ok := sblock.pile["tls.cert_issuer"]; ok {
+				for _, issuerVal := range issuerVals {
+					issuer := issuerVal.Value.(certmagic.Issuer)
+					if ap == nil {
+						var err error
+						ap, err = newBaseAutomationPolicy(options, warnings, true)
+						if err != nil {
+							return nil, warnings, err
+						}
+					}
+					if ap == catchAllAP && !reflect.DeepEqual(ap.Issuer, issuer) {
+						return nil, warnings, fmt.Errorf("automation policy from site block is also default/catch-all policy because of key without hostname, and the two are in conflict: %#v != %#v", ap.Issuer, issuer)
+					}
+					ap.Issuer = issuer
+				}
+			}
+
+			// custom bind host
+			for _, cfgVal := range sblock.pile["bind"] {
+				// either an existing issuer is already configured (and thus, ap is not
+				// nil), or we need to configure an issuer, so we need ap to be non-nil
+				if ap == nil {
+					ap, err = newBaseAutomationPolicy(options, warnings, true)
+					if err != nil {
+						return nil, warnings, err
+					}
+				}
+
+				// if an issuer was already configured and it is NOT an ACME
+				// issuer, skip, since we intend to adjust only ACME issuers
+				var acmeIssuer *caddytls.ACMEIssuer
+				if ap.Issuer != nil {
+					// ensure we include any issuer that embeds/wraps an underlying ACME issuer
+					type acmeCapable interface{ GetACMEIssuer() *caddytls.ACMEIssuer }
+					if acmeWrapper, ok := ap.Issuer.(acmeCapable); ok {
+						acmeIssuer = acmeWrapper.GetACMEIssuer()
+					} else {
+						break
+					}
+				}
+
+				// proceed to configure the ACME issuer's bind host, without
+				// overwriting any existing settings
+				if acmeIssuer == nil {
+					acmeIssuer = new(caddytls.ACMEIssuer)
+				}
+				if acmeIssuer.Challenges == nil {
+					acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
+				}
+				if acmeIssuer.Challenges.BindHost == "" {
+					// only binding to one host is supported
+					var bindHost string
+					if bindHosts, ok := cfgVal.Value.([]string); ok && len(bindHosts) > 0 {
+						bindHost = bindHosts[0]
+					}
+					acmeIssuer.Challenges.BindHost = bindHost
+				}
+				ap.Issuer = acmeIssuer // we'll encode it later
+			}
+
+			if ap != nil {
+				if ap.Issuer != nil {
+					// encode issuer now that it's all set up
+					issuerName := ap.Issuer.(caddy.Module).CaddyModule().ID.Name()
+					ap.IssuerRaw = caddyconfig.JSONModuleObject(ap.Issuer, "module", issuerName, &warnings)
+				}
+
+				// first make sure this block is allowed to create an automation policy;
+				// doing so is forbidden if it has a key with no host (i.e. ":443")
+				// and if there is a different server block that also has a key with no
+				// host -- since a key with no host matches any host, we need its
+				// associated automation policy to have an empty Subjects list, i.e. no
+				// host filter, which is indistinguishable between the two server blocks
+				// because automation is not done in the context of a particular server...
+				// this is an example of a poor mapping from Caddyfile to JSON but that's
+				// the least-leaky abstraction I could figure out
+				if len(sblockHosts) == 0 {
+					if serverBlocksWithTLSHostlessKey > 1 {
+						// this server block and at least one other has a key with no host,
+						// making the two indistinguishable; it is misleading to define such
+						// a policy within one server block since it actually will apply to
+						// others as well
+						return nil, warnings, fmt.Errorf("cannot make a TLS automation policy from a server block that has a host-less address when there are other TLS-enabled server block addresses lacking a host")
+					}
+					if catchAllAP == nil {
+						// this server block has a key with no hosts, but there is not yet
+						// a catch-all automation policy (probably because no global options
+						// were set), so this one becomes it
+						catchAllAP = ap
+					}
+				}
+
+				// associate our new automation policy with this server block's hosts,
+				// unless, of course, the server block has a key with no hosts, in which
+				// case its automation policy becomes or blends with the default/global
+				// automation policy because, of necessity, it applies to all hostnames
+				// (i.e. it has no Subjects filter) -- in that case, we'll append it last
+				if ap != catchAllAP {
+					ap.Subjects = sblockHosts
+
+					// if a combination of public and internal names were given
+					// for this same server block and no issuer was specified, we
+					// need to separate them out in the automation policies so
+					// that the internal names can use the internal issuer and
+					// the other names can use the default/public/ACME issuer
+					var ap2 *caddytls.AutomationPolicy
+					if ap.Issuer == nil {
+						var internal, external []string
+						for _, s := range ap.Subjects {
+							if certmagic.SubjectQualifiesForPublicCert(s) {
+								external = append(external, s)
+							} else {
+								internal = append(internal, s)
+							}
+						}
+						if len(external) > 0 && len(internal) > 0 {
+							ap.Subjects = external
+							apCopy := *ap
+							ap2 = &apCopy
+							ap2.Subjects = internal
+							ap2.IssuerRaw = caddyconfig.JSONModuleObject(caddytls.InternalIssuer{}, "module", "internal", &warnings)
+						}
+					}
+					if tlsApp.Automation == nil {
+						tlsApp.Automation = new(caddytls.AutomationConfig)
+					}
+					tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap)
+					if ap2 != nil {
+						tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap2)
+					}
+				}
+			}
+
+			// certificate loaders
+			if clVals, ok := sblock.pile["tls.cert_loader"]; ok {
+				for _, clVal := range clVals {
+					certLoaders = append(certLoaders, clVal.Value.(caddytls.CertificateLoader))
+				}
+			}
+		}
+	}
+
+	// group certificate loaders by module name, then add to config
+	if len(certLoaders) > 0 {
+		loadersByName := make(map[string]caddytls.CertificateLoader)
+		for _, cl := range certLoaders {
+			name := caddy.GetModuleName(cl)
+			// ugh... technically, we may have multiple FileLoader and FolderLoader
+			// modules (because the tls directive returns one per occurrence), but
+			// the config structure expects only one instance of each kind of loader
+			// module, so we have to combine them... instead of enumerating each
+			// possible cert loader module in a type switch, we can use reflection,
+			// which works on any cert loaders that are slice types
+			if reflect.TypeOf(cl).Kind() == reflect.Slice {
+				combined := reflect.ValueOf(loadersByName[name])
+				if !combined.IsValid() {
+					combined = reflect.New(reflect.TypeOf(cl)).Elem()
+				}
+				clVal := reflect.ValueOf(cl)
+				for i := 0; i < clVal.Len(); i++ {
+					combined = reflect.Append(combined, clVal.Index(i))
+				}
+				loadersByName[name] = combined.Interface().(caddytls.CertificateLoader)
+			}
+		}
+		for certLoaderName, loaders := range loadersByName {
+			tlsApp.CertificatesRaw[certLoaderName] = caddyconfig.JSON(loaders, &warnings)
+		}
+	}
+
+	// set any of the on-demand options, for if/when on-demand TLS is enabled
+	if onDemand, ok := options["on_demand_tls"].(*caddytls.OnDemandConfig); ok {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.OnDemand = onDemand
+	}
+
+	// if any hostnames appear on the same server block as a key with
+	// no host, they will not be used with route matchers because the
+	// hostless key matches all hosts, therefore, it wouldn't be
+	// considered for auto-HTTPS, so we need to make sure those hosts
+	// are manually considered for managed certificates; we also need
+	// to make sure that any of these names which are internal-only
+	// get internal certificates by default rather than ACME
+	var al caddytls.AutomateLoader
+	internalAP := &caddytls.AutomationPolicy{
+		IssuerRaw: json.RawMessage(`{"module":"internal"}`),
+	}
+	for h := range hostsSharedWithHostlessKey {
+		al = append(al, h)
+		if !certmagic.SubjectQualifiesForPublicCert(h) {
+			internalAP.Subjects = append(internalAP.Subjects, h)
+		}
+	}
+	if len(al) > 0 {
+		tlsApp.CertificatesRaw["automate"] = caddyconfig.JSON(al, &warnings)
+	}
+	if len(internalAP.Subjects) > 0 {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, internalAP)
+	}
+
+	// if there is a global/catch-all automation policy, ensure it goes last
+	if catchAllAP != nil {
+		// first, encode its issuer, if there is one
+		if catchAllAP.Issuer != nil {
+			issuerName := catchAllAP.Issuer.(caddy.Module).CaddyModule().ID.Name()
+			catchAllAP.IssuerRaw = caddyconfig.JSONModuleObject(catchAllAP.Issuer, "module", issuerName, &warnings)
+		}
+
+		// then append it to the end of the policies list
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, catchAllAP)
+	}
+
+	// do a little verification & cleanup
+	if tlsApp.Automation != nil {
+		// consolidate automation policies that are the exact same
+		tlsApp.Automation.Policies = consolidateAutomationPolicies(tlsApp.Automation.Policies)
+
+		// ensure automation policies don't overlap subjects (this should be
+		// an error at provision-time as well, but catch it in the adapt phase
+		// for convenience)
+		automationHostSet := make(map[string]struct{})
+		for _, ap := range tlsApp.Automation.Policies {
+			for _, s := range ap.Subjects {
+				if _, ok := automationHostSet[s]; ok {
+					return nil, warnings, fmt.Errorf("hostname appears in more than one automation policy, making certificate management ambiguous: %s", s)
+				}
+				automationHostSet[s] = struct{}{}
+			}
+		}
+	}
+
+	return tlsApp, warnings, nil
+}
+
+// newBaseAutomationPolicy returns a new TLS automation policy that gets
+// its values from the global options map. It should be used as the base
+// for any other automation policies. A nil policy (and no error) will be
+// returned if there are no default/global options. However, if always is
+// true, a non-nil value will always be returned (unless there is an error).
+func newBaseAutomationPolicy(options map[string]interface{}, warnings []caddyconfig.Warning, always bool) (*caddytls.AutomationPolicy, error) {
+	issuer, hasIssuer := options["cert_issuer"]
+
+	acmeCA, hasACMECA := options["acme_ca"]
+	acmeCARoot, hasACMECARoot := options["acme_ca_root"]
+	acmeDNS, hasACMEDNS := options["acme_dns"]
+	acmeEAB, hasACMEEAB := options["acme_eab"]
+
+	email, hasEmail := options["email"]
+	localCerts, hasLocalCerts := options["local_certs"]
+	keyType, hasKeyType := options["key_type"]
+
+	hasGlobalAutomationOpts := hasIssuer || hasACMECA || hasACMECARoot || hasACMEDNS || hasACMEEAB || hasEmail || hasLocalCerts || hasKeyType
+
+	// if there are no global options related to automation policies
+	// set, then we can just return right away
+	if !hasGlobalAutomationOpts {
+		if always {
+			return new(caddytls.AutomationPolicy), nil
+		}
+		return nil, nil
+	}
+
+	ap := new(caddytls.AutomationPolicy)
+	if keyType != nil {
+		ap.KeyType = keyType.(string)
+	}
+
+	if hasIssuer {
+		if hasACMECA || hasACMEDNS || hasACMEEAB || hasEmail || hasLocalCerts {
+			return nil, fmt.Errorf("global options are ambiguous: cert_issuer is confusing when combined with acme_*, email, or local_certs options")
+		}
+		ap.Issuer = issuer.(certmagic.Issuer)
+	} else if localCerts != nil {
+		// internal issuer enabled trumps any ACME configurations; useful in testing
+		ap.Issuer = new(caddytls.InternalIssuer) // we'll encode it later
+	} else {
+		if acmeCA == nil {
+			acmeCA = ""
+		}
+		if email == nil {
+			email = ""
+		}
+		mgr := &caddytls.ACMEIssuer{
+			CA:    acmeCA.(string),
+			Email: email.(string),
+		}
+		if acmeDNS != nil {
+			provName := acmeDNS.(string)
+			dnsProvModule, err := caddy.GetModule("dns.providers." + provName)
+			if err != nil {
+				return nil, fmt.Errorf("getting DNS provider module named '%s': %v", provName, err)
+			}
+			mgr.Challenges = &caddytls.ChallengesConfig{
+				DNS: &caddytls.DNSChallengeConfig{
+					ProviderRaw: caddyconfig.JSONModuleObject(dnsProvModule.New(), "name", provName, &warnings),
+				},
+			}
+		}
+		if acmeCARoot != nil {
+			mgr.TrustedRootsPEMFiles = []string{acmeCARoot.(string)}
+		}
+		if acmeEAB != nil {
+			mgr.ExternalAccount = acmeEAB.(*acme.EAB)
+		}
+		ap.Issuer = disambiguateACMEIssuer(mgr) // we'll encode it later
+	}
+
+	return ap, nil
+}
+
+// disambiguateACMEIssuer returns an issuer based on the properties of acmeIssuer.
+// If acmeIssuer implicitly configures a certain kind of ACMEIssuer (for example,
+// ZeroSSL), the proper wrapper over acmeIssuer will be returned instead.
+func disambiguateACMEIssuer(acmeIssuer *caddytls.ACMEIssuer) certmagic.Issuer {
+	// as a special case, we integrate with ZeroSSL's ACME endpoint if it looks like an
+	// implicit ZeroSSL configuration (this requires a  wrapper type over ACMEIssuer
+	// because of the EAB generation; if EAB is provided, we can use plain ACMEIssuer)
+	if strings.Contains(acmeIssuer.CA, "acme.zerossl.com") && acmeIssuer.ExternalAccount == nil {
+		return &caddytls.ZeroSSLIssuer{ACMEIssuer: acmeIssuer}
+	}
+	return acmeIssuer
+}
+
+// consolidateAutomationPolicies combines automation policies that are the same,
+// for a cleaner overall output.
+func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls.AutomationPolicy {
+	for i := 0; i < len(aps); i++ {
+		for j := 0; j < len(aps); j++ {
+			if j == i {
+				continue
+			}
+
+			// if they're exactly equal in every way, just keep one of them
+			if reflect.DeepEqual(aps[i], aps[j]) {
+				aps = append(aps[:j], aps[j+1:]...)
+				i--
+				break
+			}
+
+			// if the policy is the same, we can keep just one, but we have
+			// to be careful which one we keep; if only one has any hostnames
+			// defined, then we need to keep the one without any hostnames,
+			// otherwise the one without any subjects (a catch-all) would be
+			// eaten up by the one with subjects; and if both have subjects, we
+			// need to combine their lists
+			if bytes.Equal(aps[i].IssuerRaw, aps[j].IssuerRaw) &&
+				bytes.Equal(aps[i].StorageRaw, aps[j].StorageRaw) &&
+				aps[i].MustStaple == aps[j].MustStaple &&
+				aps[i].KeyType == aps[j].KeyType &&
+				aps[i].OnDemand == aps[j].OnDemand &&
+				aps[i].RenewalWindowRatio == aps[j].RenewalWindowRatio {
+				if len(aps[i].Subjects) == 0 && len(aps[j].Subjects) > 0 {
+					aps = append(aps[:j], aps[j+1:]...)
+				} else if len(aps[i].Subjects) > 0 && len(aps[j].Subjects) == 0 {
+					aps = append(aps[:i], aps[i+1:]...)
+				} else {
+					// avoid repeated subjects
+					for _, subj := range aps[j].Subjects {
+						if !sliceContains(aps[i].Subjects, subj) {
+							aps[i].Subjects = append(aps[i].Subjects, subj)
+						}
+					}
+					aps = append(aps[:j], aps[j+1:]...)
+				}
+				i--
+				break
+			}
+		}
+	}
+
+	// ensure any catch-all policies go last
+	sort.SliceStable(aps, func(i, j int) bool {
+		return len(aps[i].Subjects) > len(aps[j].Subjects)
+	})
+
+	return aps
+}

caddyconfig/json5/json5.go

@@ -1,43 +0,0 @@
-// Copyright 2015 Matthew Holt and The Caddy Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package json5adapter
-
-import (
-	"encoding/json"
-
-	"github.com/caddyserver/caddy/v2/caddyconfig"
-	"github.com/ilibs/json5"
-)
-
-func init() {
-	caddyconfig.RegisterAdapter("json5", Adapter{})
-}
-
-// Adapter adapts JSON5 to Caddy JSON.
-type Adapter struct{}
-
-// Adapt converts the JSON5 config in body to Caddy JSON.
-func (a Adapter) Adapt(body []byte, options map[string]interface{}) (result []byte, warnings []caddyconfig.Warning, err error) {
-	var decoded interface{}
-	err = json5.Unmarshal(body, &decoded)
-	if err != nil {
-		return
-	}
-	result, err = json.Marshal(decoded)
-	return
-}
-
-// Interface guard
-var _ caddyconfig.Adapter = (*Adapter)(nil)

caddyconfig/jsonc/jsonc.go

@@ -1,49 +0,0 @@
-// Copyright 2015 Matthew Holt and The Caddy Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package jsoncadapter
-
-import (
-	"encoding/json"
-
-	"github.com/caddyserver/caddy/v2/caddyconfig"
-	"github.com/muhammadmuzzammil1998/jsonc"
-)
-
-func init() {
-	caddyconfig.RegisterAdapter("jsonc", Adapter{})
-}
-
-// Adapter adapts JSON-C to Caddy JSON.
-type Adapter struct{}
-
-// Adapt converts the JSON-C config in body to Caddy JSON.
-func (a Adapter) Adapt(body []byte, options map[string]interface{}) (result []byte, warnings []caddyconfig.Warning, err error) {
-	result = jsonc.ToJSON(body)
-
-	// any errors in the JSON will be
-	// reported during config load, but
-	// we can at least warn here that
-	// it is not valid JSON
-	if !json.Valid(result) {
-		warnings = append(warnings, caddyconfig.Warning{
-			Message: "Resulting JSON is invalid.",
-		})
-	}
-
-	return
-}
-
-// Interface guard
-var _ caddyconfig.Adapter = (*Adapter)(nil)

caddyconfig/load.go

@@ -0,0 +1,153 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyconfig
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"mime"
+	"net/http"
+	"strings"
+	"sync"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func init() {
+	caddy.RegisterModule(adminLoad{})
+}
+
+// adminLoad is a module that provides the /load endpoint
+// for the Caddy admin API. The only reason it's not baked
+// into the caddy package directly is because of the import
+// of the caddyconfig package for its GetAdapter function.
+// If the caddy package depends on the caddyconfig package,
+// then the caddyconfig package will not be able to import
+// the caddy package, and it can more easily cause backward
+// edges in the dependency tree (i.e. import cycle).
+// Fortunately, the admin API has first-class support for
+// adding endpoints from modules.
+type adminLoad struct{}
+
+// CaddyModule returns the Caddy module information.
+func (adminLoad) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "admin.api.load",
+		New: func() caddy.Module { return new(adminLoad) },
+	}
+}
+
+// Routes returns a route for the /load endpoint.
+func (al adminLoad) Routes() []caddy.AdminRoute {
+	return []caddy.AdminRoute{
+		{
+			Pattern: "/load",
+			Handler: caddy.AdminHandlerFunc(al.handleLoad),
+		},
+	}
+}
+
+// handleLoad replaces the entire current configuration with
+// a new one provided in the response body. It supports config
+// adapters through the use of the Content-Type header. A
+// config that is identical to the currently-running config
+// will be a no-op unless Cache-Control: must-revalidate is set.
+func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
+	if r.Method != http.MethodPost {
+		return caddy.APIError{
+			Code: http.StatusMethodNotAllowed,
+			Err:  fmt.Errorf("method not allowed"),
+		}
+	}
+
+	buf := bufPool.Get().(*bytes.Buffer)
+	buf.Reset()
+	defer bufPool.Put(buf)
+
+	_, err := io.Copy(buf, r.Body)
+	if err != nil {
+		return caddy.APIError{
+			Code: http.StatusBadRequest,
+			Err:  fmt.Errorf("reading request body: %v", err),
+		}
+	}
+	body := buf.Bytes()
+
+	// if the config is formatted other than Caddy's native
+	// JSON, we need to adapt it before loading it
+	if ctHeader := r.Header.Get("Content-Type"); ctHeader != "" {
+		ct, _, err := mime.ParseMediaType(ctHeader)
+		if err != nil {
+			return caddy.APIError{
+				Code: http.StatusBadRequest,
+				Err:  fmt.Errorf("invalid Content-Type: %v", err),
+			}
+		}
+		if !strings.HasSuffix(ct, "/json") {
+			slashIdx := strings.Index(ct, "/")
+			if slashIdx < 0 {
+				return caddy.APIError{
+					Code: http.StatusBadRequest,
+					Err:  fmt.Errorf("malformed Content-Type"),
+				}
+			}
+			adapterName := ct[slashIdx+1:]
+			cfgAdapter := GetAdapter(adapterName)
+			if cfgAdapter == nil {
+				return caddy.APIError{
+					Code: http.StatusBadRequest,
+					Err:  fmt.Errorf("unrecognized config adapter '%s'", adapterName),
+				}
+			}
+			result, warnings, err := cfgAdapter.Adapt(body, nil)
+			if err != nil {
+				return caddy.APIError{
+					Code: http.StatusBadRequest,
+					Err:  fmt.Errorf("adapting config using %s adapter: %v", adapterName, err),
+				}
+			}
+			if len(warnings) > 0 {
+				respBody, err := json.Marshal(warnings)
+				if err != nil {
+					caddy.Log().Named("admin.api.load").Error(err.Error())
+				}
+				_, _ = w.Write(respBody)
+			}
+			body = result
+		}
+	}
+
+	forceReload := r.Header.Get("Cache-Control") == "must-revalidate"
+
+	err = caddy.Load(body, forceReload)
+	if err != nil {
+		return caddy.APIError{
+			Code: http.StatusBadRequest,
+			Err:  fmt.Errorf("loading config: %v", err),
+		}
+	}
+
+	caddy.Log().Named("admin.api").Info("load complete")
+
+	return nil
+}
+
+var bufPool = sync.Pool{
+	New: func() interface{} {
+		return new(bytes.Buffer)
+	},
+}

caddytest/a.caddy.localhost.crt

@@ -0,0 +1,23 @@
+-----BEGIN CERTIFICATE-----
+MIID5zCCAs8CFG4+w/pqR5AZQ+aVB330uRRRKMF0MA0GCSqGSIb3DQEBCwUAMIGv
+MQswCQYDVQQGEwJVUzELMAkGA1UECAwCTlkxGzAZBgNVBAoMEkxvY2FsIERldmVs
+b3BlbWVudDEbMBkGA1UEBwwSTG9jYWwgRGV2ZWxvcGVtZW50MRowGAYDVQQDDBFh
+LmNhZGR5LmxvY2FsaG9zdDEbMBkGA1UECwwSTG9jYWwgRGV2ZWxvcGVtZW50MSAw
+HgYJKoZIhvcNAQkBFhFhZG1pbkBjYWRkeS5sb2NhbDAeFw0yMDAzMTMxODUwMTda
+Fw0zMDAzMTExODUwMTdaMIGvMQswCQYDVQQGEwJVUzELMAkGA1UECAwCTlkxGzAZ
+BgNVBAoMEkxvY2FsIERldmVsb3BlbWVudDEbMBkGA1UEBwwSTG9jYWwgRGV2ZWxv
+cGVtZW50MRowGAYDVQQDDBFhLmNhZGR5LmxvY2FsaG9zdDEbMBkGA1UECwwSTG9j
+YWwgRGV2ZWxvcGVtZW50MSAwHgYJKoZIhvcNAQkBFhFhZG1pbkBjYWRkeS5sb2Nh
+bDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMd9pC9wF7j0459FndPs
+Deud/rq41jEZFsVOVtjQgjS1A5ct6NfeMmSlq8i1F7uaTMPZjbOHzY6y6hzLc9+y
+/VWNgyUC543HjXnNTnp9Xug6tBBxOxvRMw5mv2nAyzjBGDePPgN84xKhOXG2Wj3u
+fOZ+VPVISefRNvjKfN87WLJ0B0HI9wplG5ASVdPQsWDY1cndrZgt2sxQ/3fjIno4
+VvrgRWC9Penizgps/a0ZcFZMD/6HJoX/mSZVa1LjopwbMTXvyHCpXkth21E+rBt6
+I9DMHerdioVQcX25CqPmAwePxPZSNGEQo/Qu32kzcmscmYxTtYBhDa+yLuHgGggI
+j7ECAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAP/94KPtkpYtkWADnhtzDmgQ6Q1pH
+SubTUZdCwQtm6/LrvpT+uFNsOj4L3Mv3TVUnIQDmKd5VvR42W2MRBiTN2LQptgEn
+C7g9BB+UA9kjL3DPk1pJMjzxLHohh0uNLi7eh4mAj8eNvjz9Z4qMWPQoVS0y7/ZK
+cCBRKh2GkIqKm34ih6pX7xmMpPEQsFoTVPRHYJfYD1SZ8Iui+EN+7WqLuJWPsPXw
+JM1HuZKn7pZmJU2MZZBsrupHGUvNMbBg2mFJcxt4D1VvU+p+a67PSjpFQ6dJG2re
+pZoF+N1vMGAFkxe6UqhcC/bXDX+ILVQHJ+RNhzDO6DcWf8dRrC2LaJk3WA==
+-----END CERTIFICATE-----

caddytest/a.caddy.localhost.key

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpAIBAAKCAQEAx32kL3AXuPTjn0Wd0+wN653+urjWMRkWxU5W2NCCNLUDly3o
+194yZKWryLUXu5pMw9mNs4fNjrLqHMtz37L9VY2DJQLnjceNec1Oen1e6Dq0EHE7
+G9EzDma/acDLOMEYN48+A3zjEqE5cbZaPe585n5U9UhJ59E2+Mp83ztYsnQHQcj3
+CmUbkBJV09CxYNjVyd2tmC3azFD/d+MiejhW+uBFYL096eLOCmz9rRlwVkwP/ocm
+hf+ZJlVrUuOinBsxNe/IcKleS2HbUT6sG3oj0Mwd6t2KhVBxfbkKo+YDB4/E9lI0
+YRCj9C7faTNyaxyZjFO1gGENr7Iu4eAaCAiPsQIDAQABAoIBAQDD/YFIBeWYlifn
+e9risQDAIrp3sk7lb9O6Rwv1+Wxi4hBEABvJsYhq74VFK/3EF4UhyWR5JIvkjYyK
+e6w887oGyoA05ZSe65XoO7fFidSrbbkoikZbPv3dQT7/ZCWEfdkQBNAVVyY0UGeC
+e3hPbjYRsb5AOSQ694X9idqC6uhqcOrBDjITFrctUoP4S6l9A6a+mLSUIwiICcuh
+mrNl+j0lzy7DMXRp/Z5Hyo5kuUlrC0dCLa1UHqtrrK7MR55AVEOihSNp1w+OC+vw
+f0VjE4JUtO7RQEQUmD1tDfLXwNfMFeWaobB2W0WMvRg0IqoitiqPxsPHRm56OxfM
+SRo/Q7QBAoGBAP8DapzBMuaIcJ7cE8Yl07ZGndWWf8buIKIItGF8rkEO3BXhrIke
+EmpOi+ELtpbMOG0APhORZyQ58f4ZOVrqZfneNKtDiEZV4mJZaYUESm1pU+2Y6+y5
+g4bpQSVKN0ow0xR+MH7qDYtSlsmBU7qAOz775L7BmMA1Bnu72aN/H1JBAoGBAMhD
+OzqCSakHOjUbEd22rPwqWmcIyVyo04gaSmcVVT2dHbqR4/t0gX5a9D9U2qwyO6xi
+/R+PXyMd32xIeVR2D/7SQ0x6dK68HXICLV8ofHZ5UQcHbxy5og4v/YxSZVTkN374
+cEsUeyB0s/UPOHLktFU5hpIlON72/Rp7b+pNIwFxAoGAczpq+Qu/YTWzlcSh1r4O
+7OT5uqI3eH7vFehTAV3iKxl4zxZa7NY+wfRd9kFhrr/2myIp6pOgBFl+hC+HoBIc
+JAyIxf5M3GNAWOpH6MfojYmzV7/qktu8l8BcJGplk0t+hVsDtMUze4nFAqZCXBpH
+Kw2M7bjyuZ78H/rgu6TcVUECgYEAo1M5ldE2U/VCApeuLX1TfWDpU8i1uK0zv3d5
+oLKkT1i5KzTak3SEO9HgC1qf8PoS8tfUio26UICHe99rnHehOfivzEq+qNdgyF+A
+M3BoeZMdgzcL5oh640k+Zte4LtDlddcWdhUhCepD7iPYrNNbQ3pkBwL2a9lRuOxc
+7OC2IPECgYBH8f3OrwXjDltIG1dDvuDPNljxLZbFEFbQyVzMePYNftgZknAyGEdh
+NW/LuWeTzstnmz/s6RE3jN5ZrrMa4sW77VA9+yU9QW2dkHqFyukQ4sfuNg6kDDNZ
++lqZYMCLw0M5P9fIbmnIYwey7tXkHfmzoCpnYHGQDN6hL0Bh0zGwmg==
+-----END RSA PRIVATE KEY-----

caddytest/caddy.ca.cer

@@ -0,0 +1,20 @@
+-----BEGIN CERTIFICATE-----
+MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQEL
+BQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkw
+ODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcN
+AQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU
+7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl0
+3WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45t
+wOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNx
+tdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTU
+ApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAd
+BgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS
+2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5u
+NY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkq
+hkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfK
+D66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEO
+fG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnk
+oNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZ
+ks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdle
+Ih6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ==
+-----END CERTIFICATE-----

caddytest/caddy.localhost.crt

@@ -0,0 +1,23 @@
+-----BEGIN CERTIFICATE-----
+MIID5zCCAs8CFFmAAFKV79uhzxc5qXbUw3oBNsYXMA0GCSqGSIb3DQEBCwUAMIGv
+MQswCQYDVQQGEwJVUzELMAkGA1UECAwCTlkxGzAZBgNVBAoMEkxvY2FsIERldmVs
+b3BlbWVudDEbMBkGA1UEBwwSTG9jYWwgRGV2ZWxvcGVtZW50MRowGAYDVQQDDBEq
+LmNhZGR5LmxvY2FsaG9zdDEbMBkGA1UECwwSTG9jYWwgRGV2ZWxvcGVtZW50MSAw
+HgYJKoZIhvcNAQkBFhFhZG1pbkBjYWRkeS5sb2NhbDAeFw0yMDAzMDIwODAxMTZa
+Fw0zMDAyMjgwODAxMTZaMIGvMQswCQYDVQQGEwJVUzELMAkGA1UECAwCTlkxGzAZ
+BgNVBAoMEkxvY2FsIERldmVsb3BlbWVudDEbMBkGA1UEBwwSTG9jYWwgRGV2ZWxv
+cGVtZW50MRowGAYDVQQDDBEqLmNhZGR5LmxvY2FsaG9zdDEbMBkGA1UECwwSTG9j
+YWwgRGV2ZWxvcGVtZW50MSAwHgYJKoZIhvcNAQkBFhFhZG1pbkBjYWRkeS5sb2Nh
+bDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJngfeirQkWaU8ihgIC5
+SKpRQX/3koRjljDK/oCbhLs+wg592kIwVv06l7+mn7NSaNBloabjuA1GqyLRsNLL
+ptrv0HvXa5qLx28+icsb2Ny3dJnQaj9w9PwjxQ1qZqEJfWRH1D8Vz9AmB+QSV/Gu
+8e8alGFewlYZVfH1kbxoTT6QorF37TeA3bh1fgKFtzsGYKswcaZNdDBBHzLunCKZ
+HU6U6L45hm+yLADj3mmDLafUeiVOt6MRLLoSD1eLRVSXGrNo+brJ87zkZntI9+W1
+JxOBoXtZCwka7k2DlAtLihsrmBZA2ZC9yVeu/SQy3qb3iCNnTFTCyAnWeTCr6Tcq
+6w8CAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAOWfXqpAmD4C3wGiMeZAeaaS4hDAR
++JmN+avPDA6F6Bq7DB4NJuIwVUlaDL2s07w5VJJtW52aZVKoBlgHR5yG/XUli6J7
+YUJRmdQJvHUSu26cmKvyoOaTrEYbmvtGICWtZc8uTlMf9wQZbJA4KyxTgEQJDXsZ
+B2XFe+wVdhAgEpobYDROi+l/p8TL5z3U24LpwVTcJy5sEZVv7Wfs886IyxU8ORt8
+VZNcDiH6V53OIGeiufIhia/mPe6jbLntfGZfIFxtCcow4IA/lTy1ned7K5fmvNNb
+ZilxOQUk+wVK8genjdrZVAnAxsYLHJIb5yf9O7rr6fWciVMF3a0k5uNK1w==
+-----END CERTIFICATE-----

caddytest/caddy.localhost.key

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEogIBAAKCAQEAmeB96KtCRZpTyKGAgLlIqlFBf/eShGOWMMr+gJuEuz7CDn3a
+QjBW/TqXv6afs1Jo0GWhpuO4DUarItGw0sum2u/Qe9drmovHbz6JyxvY3Ld0mdBq
+P3D0/CPFDWpmoQl9ZEfUPxXP0CYH5BJX8a7x7xqUYV7CVhlV8fWRvGhNPpCisXft
+N4DduHV+AoW3OwZgqzBxpk10MEEfMu6cIpkdTpTovjmGb7IsAOPeaYMtp9R6JU63
+oxEsuhIPV4tFVJcas2j5usnzvORme0j35bUnE4Ghe1kLCRruTYOUC0uKGyuYFkDZ
+kL3JV679JDLepveII2dMVMLICdZ5MKvpNyrrDwIDAQABAoIBAFcPK01zb6hfm12c
++k5aBiHOnUdgc/YRPg1XHEz5MEycQkDetZjTLrRQ7UBSbnKPgpu9lIsOtbhVLkgh
+6XAqJroiCou2oruqr+hhsqZGmBiwdvj7cNF6ADGTr05az7v22YneFdinZ481pStF
+sZocx+bm2+KHMV5zMSwXKyA0xtdJLxs2yklniDBxSZRppgppq1pDPprP5DkgKPfe
+3ekUmbQd5bHmivhW8ItbJLuf82XSsMBZ9ZhKiKIlWlbKAgiSV3SqnUQb5fi7l8hG
+yYZxbuCUIGFwKmEpUBBt/nyxrOlMiNtDh9JhrPmijTV3slq70pCLwLL/Ai2aeear
+EVA5VhkCgYEAyAmxfPqc2P7BsDAp67/sA7OEPso9qM4WyuWiVdlX2gb9TLNLYbPX
+Kk/UmpAIVzpoTAGY5Zp3wkvdD/ou8uUQsE8ioNn4S1a4G9XURH1wVhcEbUiAKI1S
+QVBH9B/Pj3eIp5OTKwob0Wj7DNdxoH7ed/Eok0EaTWzOA8pCWADKv/MCgYEAxOzY
+YsX7Nl+eyZr2+9unKyeAK/D1DCT/o99UUAHx72/xaBVP/06cfzpvKBNcF9iYc+fq
+R1yIUIrDRoSmYKBq+Kb3+nOg1nrqih/NBTokbTiI4Q+/30OQt0Al1e7y9iNKqV8H
+jYZItzluGNrWKedZbATwBwbVCY2jnNl6RMDnS3UCgYBxj3cwQUHLuoyQjjcuO80r
+qLzZvIxWiXDNDKIk5HcIMlGYOmz/8U2kGp/SgxQJGQJeq8V2C0QTjGfaCyieAcaA
+oNxCvptDgd6RBsoze5bLeNOtiqwe2WOp6n5+q5R0mOJ+Z7vzghCayGNFPgWmnH+F
+TeW/+wSIkc0+v5L8TK7NWwKBgBrlWlyLO9deUfqpHqihhICBYaEexOlGuF+yZfqT
+eW7BdFBJ8OYm33sFCR+JHV/oZlIWT8o1Wizd9vPPtEWoQ1P4wg/D8Si6GwSIeWEI
+YudD/HX4x7T/rmlI6qIAg9CYW18sqoRq3c2gm2fro6qPfYgiWIItLbWjUcBfd7Ki
+QjTtAoGARKdRv3jMWL84rlEx1nBRgL3pe9Dt+Uxzde2xT3ZeF+5Hp9NfU01qE6M6
+1I6H64smqpetlsXmCEVKwBemP3pJa6avLKgIYiQvHAD/v4rs9mqgy1RTqtYyGNhR
+1A/6dKkbiZ6wzePLLPasXVZxSKEviXf5gJooqumQVSVhCswyCZ0=
+-----END RSA PRIVATE KEY-----

caddytest/caddytest.go

@@ -0,0 +1,504 @@
+package caddytest
+
+import (
+	"bytes"
+	"context"
+	"crypto/tls"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"log"
+	"net"
+	"net/http"
+	"net/http/cookiejar"
+	"os"
+	"path"
+	"regexp"
+	"runtime"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/aryann/difflib"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	caddycmd "github.com/caddyserver/caddy/v2/cmd"
+
+	// plug in Caddy modules here
+	_ "github.com/caddyserver/caddy/v2/modules/standard"
+)
+
+// Defaults store any configuration required to make the tests run
+type Defaults struct {
+	// Port we expect caddy to listening on
+	AdminPort int
+	// Certificates we expect to be loaded before attempting to run the tests
+	Certifcates []string
+	// TestRequestTimeout is the time to wait for a http request to
+	TestRequestTimeout time.Duration
+	// LoadRequestTimeout is the time to wait for the config to be loaded against the caddy server
+	LoadRequestTimeout time.Duration
+}
+
+// Default testing values
+var Default = Defaults{
+	AdminPort:          2019,
+	Certifcates:        []string{"/caddy.localhost.crt", "/caddy.localhost.key"},
+	TestRequestTimeout: 5 * time.Second,
+	LoadRequestTimeout: 5 * time.Second,
+}
+
+var (
+	matchKey  = regexp.MustCompile(`(/[\w\d\.]+\.key)`)
+	matchCert = regexp.MustCompile(`(/[\w\d\.]+\.crt)`)
+)
+
+// Tester represents an instance of a test client.
+type Tester struct {
+	Client       *http.Client
+	configLoaded bool
+	t            *testing.T
+}
+
+// NewTester will create a new testing client with an attached cookie jar
+func NewTester(t *testing.T) *Tester {
+
+	jar, err := cookiejar.New(nil)
+	if err != nil {
+		t.Fatalf("failed to create cookiejar: %s", err)
+	}
+
+	return &Tester{
+		Client: &http.Client{
+			Transport: CreateTestingTransport(),
+			Jar:       jar,
+			Timeout:   Default.TestRequestTimeout,
+		},
+		configLoaded: false,
+		t:            t,
+	}
+}
+
+type configLoadError struct {
+	Response string
+}
+
+func (e configLoadError) Error() string { return e.Response }
+
+func timeElapsed(start time.Time, name string) {
+	elapsed := time.Since(start)
+	log.Printf("%s took %s", name, elapsed)
+}
+
+// InitServer this will configure the server with a configurion of a specific
+// type. The configType must be either "json" or the adapter type.
+func (tc *Tester) InitServer(rawConfig string, configType string) {
+
+	if err := tc.initServer(rawConfig, configType); err != nil {
+		tc.t.Logf("failed to load config: %s", err)
+		tc.t.Fail()
+	}
+}
+
+// InitServer this will configure the server with a configurion of a specific
+// type. The configType must be either "json" or the adapter type.
+func (tc *Tester) initServer(rawConfig string, configType string) error {
+
+	if testing.Short() {
+		tc.t.SkipNow()
+		return nil
+	}
+
+	err := validateTestPrerequisites()
+	if err != nil {
+		tc.t.Skipf("skipping tests as failed integration prerequisites. %s", err)
+		return nil
+	}
+
+	tc.t.Cleanup(func() {
+		if tc.t.Failed() && tc.configLoaded {
+
+			res, err := http.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
+			if err != nil {
+				tc.t.Log("unable to read the current config")
+				return
+			}
+			defer res.Body.Close()
+			body, err := ioutil.ReadAll(res.Body)
+
+			var out bytes.Buffer
+			json.Indent(&out, body, "", "  ")
+			tc.t.Logf("----------- failed with config -----------\n%s", out.String())
+		}
+	})
+
+	rawConfig = prependCaddyFilePath(rawConfig)
+	client := &http.Client{
+		Timeout: Default.LoadRequestTimeout,
+	}
+	start := time.Now()
+	req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/load", Default.AdminPort), strings.NewReader(rawConfig))
+	if err != nil {
+		tc.t.Errorf("failed to create request. %s", err)
+		return err
+	}
+
+	if configType == "json" {
+		req.Header.Add("Content-Type", "application/json")
+	} else {
+		req.Header.Add("Content-Type", "text/"+configType)
+	}
+
+	res, err := client.Do(req)
+	if err != nil {
+		tc.t.Errorf("unable to contact caddy server. %s", err)
+		return err
+	}
+	timeElapsed(start, "caddytest: config load time")
+
+	defer res.Body.Close()
+	body, err := ioutil.ReadAll(res.Body)
+	if err != nil {
+		tc.t.Errorf("unable to read response. %s", err)
+		return err
+	}
+
+	if res.StatusCode != 200 {
+		return configLoadError{Response: string(body)}
+	}
+
+	tc.configLoaded = true
+	return nil
+}
+
+var hasValidated bool
+var arePrerequisitesValid bool
+
+func validateTestPrerequisites() error {
+
+	if hasValidated {
+		if !arePrerequisitesValid {
+			return errors.New("caddy integration prerequisites failed. see first error")
+		}
+		return nil
+	}
+
+	hasValidated = true
+	arePrerequisitesValid = false
+
+	// check certificates are found
+	for _, certName := range Default.Certifcates {
+		if _, err := os.Stat(getIntegrationDir() + certName); os.IsNotExist(err) {
+			return fmt.Errorf("caddy integration test certificates (%s) not found", certName)
+		}
+	}
+
+	if isCaddyAdminRunning() != nil {
+		// start inprocess caddy server
+		os.Args = []string{"caddy", "run"}
+		go func() {
+			caddycmd.Main()
+		}()
+
+		// wait for caddy to start
+		retries := 4
+		for ; retries > 0 && isCaddyAdminRunning() != nil; retries-- {
+			time.Sleep(10 * time.Millisecond)
+		}
+	}
+
+	// assert that caddy is running
+	if err := isCaddyAdminRunning(); err != nil {
+		return err
+	}
+
+	arePrerequisitesValid = true
+	return nil
+}
+
+func isCaddyAdminRunning() error {
+	// assert that caddy is running
+	client := &http.Client{
+		Timeout: Default.LoadRequestTimeout,
+	}
+	_, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
+	if err != nil {
+		return errors.New("caddy integration test caddy server not running. Expected to be listening on localhost:2019")
+	}
+
+	return nil
+}
+
+func getIntegrationDir() string {
+
+	_, filename, _, ok := runtime.Caller(1)
+	if !ok {
+		panic("unable to determine the current file path")
+	}
+
+	return path.Dir(filename)
+}
+
+// use the convention to replace /[certificatename].[crt|key] with the full path
+// this helps reduce the noise in test configurations and also allow this
+// to run in any path
+func prependCaddyFilePath(rawConfig string) string {
+	r := matchKey.ReplaceAllString(rawConfig, getIntegrationDir()+"$1")
+	r = matchCert.ReplaceAllString(r, getIntegrationDir()+"$1")
+	return r
+}
+
+// CreateTestingTransport creates a testing transport that forces call dialing connections to happen locally
+func CreateTestingTransport() *http.Transport {
+
+	dialer := net.Dialer{
+		Timeout:   5 * time.Second,
+		KeepAlive: 5 * time.Second,
+		DualStack: true,
+	}
+
+	dialContext := func(ctx context.Context, network, addr string) (net.Conn, error) {
+		parts := strings.Split(addr, ":")
+		destAddr := fmt.Sprintf("127.0.0.1:%s", parts[1])
+		log.Printf("caddytest: redirecting the dialer from %s to %s", addr, destAddr)
+		return dialer.DialContext(ctx, network, destAddr)
+	}
+
+	return &http.Transport{
+		Proxy:                 http.ProxyFromEnvironment,
+		DialContext:           dialContext,
+		ForceAttemptHTTP2:     true,
+		MaxIdleConns:          100,
+		IdleConnTimeout:       90 * time.Second,
+		TLSHandshakeTimeout:   5 * time.Second,
+		ExpectContinueTimeout: 1 * time.Second,
+		TLSClientConfig:       &tls.Config{InsecureSkipVerify: true},
+	}
+}
+
+// AssertLoadError will load a config and expect an error
+func AssertLoadError(t *testing.T, rawConfig string, configType string, expectedError string) {
+
+	tc := NewTester(t)
+
+	err := tc.initServer(rawConfig, configType)
+	if !strings.Contains(err.Error(), expectedError) {
+		t.Errorf("expected error \"%s\" but got \"%s\"", expectedError, err.Error())
+	}
+}
+
+// AssertRedirect makes a request and asserts the redirection happens
+func (tc *Tester) AssertRedirect(requestURI string, expectedToLocation string, expectedStatusCode int) *http.Response {
+
+	redirectPolicyFunc := func(req *http.Request, via []*http.Request) error {
+		return http.ErrUseLastResponse
+	}
+
+	// using the existing client, we override the check redirect policy for this test
+	old := tc.Client.CheckRedirect
+	tc.Client.CheckRedirect = redirectPolicyFunc
+	defer func() { tc.Client.CheckRedirect = old }()
+
+	resp, err := tc.Client.Get(requestURI)
+	if err != nil {
+		tc.t.Errorf("failed to call server %s", err)
+		return nil
+	}
+
+	if expectedStatusCode != resp.StatusCode {
+		tc.t.Errorf("requesting \"%s\" expected status code: %d but got %d", requestURI, expectedStatusCode, resp.StatusCode)
+	}
+
+	loc, err := resp.Location()
+	if err != nil {
+		tc.t.Errorf("requesting \"%s\" expected location: \"%s\" but got error: %s", requestURI, expectedToLocation, err)
+	}
+
+	if expectedToLocation != loc.String() {
+		tc.t.Errorf("requesting \"%s\" expected location: \"%s\" but got \"%s\"", requestURI, expectedToLocation, loc.String())
+	}
+
+	return resp
+}
+
+// CompareAdapt adapts a config and then compares it against an expected result
+func CompareAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) bool {
+
+	cfgAdapter := caddyconfig.GetAdapter(adapterName)
+	if cfgAdapter == nil {
+		t.Logf("unrecognized config adapter '%s'", adapterName)
+		return false
+	}
+
+	options := make(map[string]interface{})
+	options["pretty"] = "true"
+
+	result, warnings, err := cfgAdapter.Adapt([]byte(rawConfig), options)
+	if err != nil {
+		t.Logf("adapting config using %s adapter: %v", adapterName, err)
+		return false
+	}
+
+	if len(warnings) > 0 {
+		for _, w := range warnings {
+			t.Logf("warning: directive: %s : %s", w.Directive, w.Message)
+		}
+	}
+
+	diff := difflib.Diff(
+		strings.Split(expectedResponse, "\n"),
+		strings.Split(string(result), "\n"))
+
+	// scan for failure
+	failed := false
+	for _, d := range diff {
+		if d.Delta != difflib.Common {
+			failed = true
+			break
+		}
+	}
+
+	if failed {
+		for _, d := range diff {
+			switch d.Delta {
+			case difflib.Common:
+				fmt.Printf("  %s\n", d.Payload)
+			case difflib.LeftOnly:
+				fmt.Printf(" - %s\n", d.Payload)
+			case difflib.RightOnly:
+				fmt.Printf(" + %s\n", d.Payload)
+			}
+		}
+		return false
+	}
+	return true
+}
+
+// AssertAdapt adapts a config and then tests it against an expected result
+func AssertAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) {
+	ok := CompareAdapt(t, rawConfig, adapterName, expectedResponse)
+	if !ok {
+		t.Fail()
+	}
+}
+
+// Generic request functions
+
+func applyHeaders(t *testing.T, req *http.Request, requestHeaders []string) {
+	requestContentType := ""
+	for _, requestHeader := range requestHeaders {
+		arr := strings.SplitAfterN(requestHeader, ":", 2)
+		k := strings.TrimRight(arr[0], ":")
+		v := strings.TrimSpace(arr[1])
+		if k == "Content-Type" {
+			requestContentType = v
+		}
+		t.Logf("Request header: %s => %s", k, v)
+		req.Header.Set(k, v)
+	}
+
+	if requestContentType == "" {
+		t.Logf("Content-Type header not provided")
+	}
+}
+
+// AssertResponseCode will execute the request and verify the status code, returns a response for additional assertions
+func (tc *Tester) AssertResponseCode(req *http.Request, expectedStatusCode int) *http.Response {
+
+	resp, err := tc.Client.Do(req)
+	if err != nil {
+		tc.t.Fatalf("failed to call server %s", err)
+	}
+
+	if expectedStatusCode != resp.StatusCode {
+		tc.t.Errorf("requesting \"%s\" expected status code: %d but got %d", req.RequestURI, expectedStatusCode, resp.StatusCode)
+	}
+
+	return resp
+}
+
+// AssertResponse request a URI and assert the status code and the body contains a string
+func (tc *Tester) AssertResponse(req *http.Request, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	resp := tc.AssertResponseCode(req, expectedStatusCode)
+
+	defer resp.Body.Close()
+	bytes, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		tc.t.Fatalf("unable to read the response body %s", err)
+	}
+
+	body := string(bytes)
+
+	if !strings.Contains(body, expectedBody) {
+		tc.t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", req.RequestURI, expectedBody, body)
+	}
+
+	return resp, body
+}
+
+// Verb specific test functions
+
+// AssertGetResponse GET a URI and expect a statusCode and body text
+func (tc *Tester) AssertGetResponse(requestURI string, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("GET", requestURI, nil)
+	if err != nil {
+		tc.t.Fatalf("unable to create request %s", err)
+	}
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertDeleteResponse request a URI and expect a statusCode and body text
+func (tc *Tester) AssertDeleteResponse(requestURI string, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("DELETE", requestURI, nil)
+	if err != nil {
+		tc.t.Fatalf("unable to create request %s", err)
+	}
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPostResponseBody POST to a URI and assert the response code and body
+func (tc *Tester) AssertPostResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("POST", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPutResponseBody PUT to a URI and assert the response code and body
+func (tc *Tester) AssertPutResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("PUT", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPatchResponseBody PATCH to a URI and assert the response code and body
+func (tc *Tester) AssertPatchResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+
+	req, err := http.NewRequest("PATCH", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}

caddytest/caddytest_test.go

@@ -0,0 +1,33 @@
+package caddytest
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestReplaceCertificatePaths(t *testing.T) {
+	rawConfig := `a.caddy.localhost:9443 {
+		tls /caddy.localhost.crt /caddy.localhost.key {
+		}
+
+		redir / https://b.caddy.localhost:9443/version 301
+    
+		respond /version 200 {
+		  body "hello from a.caddy.localhost"
+		}	
+	  }`
+
+	r := prependCaddyFilePath(rawConfig)
+
+	if !strings.Contains(r, getIntegrationDir()+"/caddy.localhost.crt") {
+		t.Error("expected the /caddy.localhost.crt to be expanded to include the full path")
+	}
+
+	if !strings.Contains(r, getIntegrationDir()+"/caddy.localhost.key") {
+		t.Error("expected the /caddy.localhost.crt to be expanded to include the full path")
+	}
+
+	if !strings.Contains(r, "https://b.caddy.localhost:9443/version") {
+		t.Error("expected redirect uri to be unchanged")
+	}
+}

caddytest/integration/autohttps_test.go

@@ -0,0 +1,22 @@
+package integration
+
+import (
+	"net/http"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestAutoHTTPtoHTTPSRedirects(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	localhost:9443
+	respond "Yahaha! You found me!"
+  `, "caddyfile")
+
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
+}

caddytest/integration/caddyfile_adapt/auto_https_disable_redirects.txt

@@ -0,0 +1,34 @@
+{
+	auto_https disable_redirects
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"disable_redirects": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/auto_https_off.txt

@@ -0,0 +1,37 @@
+{
+	auto_https off
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"tls_connection_policies": [
+						{}
+					],
+					"automatic_https": {
+						"disable": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options.txt

@@ -0,0 +1,73 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_ca_root /path/to/ca.crt
+
+	email test@example.com
+	admin off
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	local_certs
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"disabled": true
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuer": {
+							"module": "internal"
+						},
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_acme.txt

@@ -0,0 +1,85 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_eab {
+		key_id  4K2scIVbBpNd-78scadB2g
+		mac_key abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh
+	}
+	acme_ca_root /path/to/ca.crt
+	email test@example.com
+	admin off
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"disabled": true
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuer": {
+							"ca": "https://example.com",
+							"email": "test@example.com",
+							"external_account": {
+								"key_id": "4K2scIVbBpNd-78scadB2g",
+								"mac_key": "abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh"
+							},
+							"module": "acme",
+							"trusted_roots_pem_files": [
+								"/path/to/ca.crt"
+							]
+						},
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_admin.txt

@@ -0,0 +1,81 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_ca_root /path/to/ca.crt
+
+	email test@example.com
+	admin {
+		origins localhost:2019 [::1]:2019 127.0.0.1:2019 192.168.10.128
+	}
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	local_certs
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"listen": "localhost:2019",
+		"origins": [
+			"localhost:2019",
+			"[::1]:2019",
+			"127.0.0.1:2019",
+			"192.168.10.128"
+		]
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuer": {
+							"module": "internal"
+						},
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					},
+					"ask": "https://example.com"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/handle_path.txt

@@ -0,0 +1,52 @@
+:80
+handle_path /api/v1/* {
+	respond "API v1"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/api/v1/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"strip_path_prefix": "/api/v1"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"body": "API v1",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/handle_path_sorting.txt

@@ -0,0 +1,105 @@
+:80 {
+	handle /api/* {
+		respond "api"
+	}
+
+	handle_path /static/* {
+		respond "static"
+	}
+
+	handle {
+		respond "handle"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"group": "group3",
+							"match": [
+								{
+									"path": [
+										"/static/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"strip_path_prefix": "/static"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"body": "static",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						},
+						{
+							"group": "group3",
+							"match": [
+								{
+									"path": [
+										"/api/*"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "api",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						},
+						{
+							"group": "group3",
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "handle",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_any_address.txt

@@ -0,0 +1,37 @@
+:80 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/version"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "hello from localhost",
+									"handler": "static_response",
+									"status_code": 200
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_domain.txt

@@ -0,0 +1,59 @@
+http://a.caddy.localhost {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"a.caddy.localhost"
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_localhost.txt

@@ -0,0 +1,54 @@
+localhost:80 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/http_only_on_non_standard_port.txt

@@ -0,0 +1,59 @@
+http://a.caddy.localhost:81 {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":81"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"skip": [
+							"a.caddy.localhost"
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/https_on_domain.txt

@@ -0,0 +1,54 @@
+a.caddy.localhost {
+	respond /version 200 {
+		body "hello from localhost"
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.caddy.localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response",
+													"status_code": 200
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/version"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_file.txt

@@ -0,0 +1,49 @@
+example.com
+
+import testdata/import_respond.txt Groot Rocket
+import testdata/import_respond.txt you "the confused man"
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "'I am Groot', hears Rocket",
+													"handler": "static_response"
+												},
+												{
+													"body": "'I am you', hears the confused man",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/import_args_snippet.txt

@@ -0,0 +1,83 @@
+(logging) {
+	log {
+		output file /var/log/caddy/{args.0}.access.log
+	}
+}
+
+a.example.com {
+	import logging a.example.com
+}
+
+b.example.com {
+	import logging b.example.com
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0",
+					"http.log.access.log1"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "/var/log/caddy/a.example.com.access.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			},
+			"log1": {
+				"writer": {
+					"filename": "/var/log/caddy/b.example.com.access.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log1"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"a.example.com"
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"b.example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"logs": {
+						"logger_names": {
+							"a.example.com": "log0",
+							"b.example.com": "log1"
+						}
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_filters.txt

@@ -0,0 +1,69 @@
+:80
+
+log {
+	output stdout
+	format filter {
+		wrap console
+		fields {
+			request>headers>Authorization delete
+			request>headers>Server delete
+			request>remote_addr ip_mask {
+				ipv4 24
+				ipv6 32
+			}
+		}
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0"
+				]
+			},
+			"log0": {
+				"writer": {
+					"output": "stdout"
+				},
+				"encoder": {
+					"fields": {
+						"request\u003eheaders\u003eAuthorization": {
+							"filter": "delete"
+						},
+						"request\u003eheaders\u003eServer": {
+							"filter": "delete"
+						},
+						"request\u003eremote_addr": {
+							"filter": "ip_mask",
+							"ipv4_cidr": 24,
+							"ipv6_cidr": 32
+						}
+					},
+					"format": "filter",
+					"wrap": {
+						"format": "console"
+					}
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"logs": {
+						"default_logger_name": "log0"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/log_roll_days.txt

@@ -0,0 +1,47 @@
+:80
+
+log {
+	output file /var/log/access.log {
+		roll_size 1gb
+		roll_keep 5
+		roll_keep_for 90d
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.log0"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "/var/log/access.log",
+					"output": "file",
+					"roll_keep": 5,
+					"roll_keep_days": 90,
+					"roll_size_mb": 954
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"logs": {
+						"default_logger_name": "log0"
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/matcher_syntax.txt

@@ -0,0 +1,77 @@
+:80 {
+	@matcher {
+		method GET
+	}
+	respond @matcher "get"
+
+	@matcher2 method POST
+	respond @matcher2 "post"
+
+	@matcher3 not method PUT
+	respond @matcher3 "not put"
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"method": [
+										"GET"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "get",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"method": [
+										"POST"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "post",
+									"handler": "static_response"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"not": [
+										{
+											"method": [
+												"PUT"
+											]
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "not put",
+									"handler": "static_response"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/matchers_in_route.txt

@@ -0,0 +1,31 @@
+:80 {
+    route {
+        # unused matchers should not panic
+        # see https://github.com/caddyserver/caddy/issues/3745
+        @matcher1 path /path1
+        @matcher2 path /path2
+    }
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "subroute"
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/not_block_merging.txt

@@ -0,0 +1,49 @@
+:80
+
+@test {
+	not {
+		header Abc "123"
+		header Bcd "123"
+	}
+}
+respond @test 403
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"not": [
+										{
+											"header": {
+												"Abc": [
+													"123"
+												],
+												"Bcd": [
+													"123"
+												]
+											}
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "static_response",
+									"status_code": 403
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_expanded_form.txt

@@ -0,0 +1,132 @@
+:8886
+
+route {
+	# Add trailing slash for directory requests
+	@canonicalPath {
+		file {
+			try_files {path}/index.php
+		}
+		not path */
+	}
+	redir @canonicalPath {path}/ 308
+
+	# If the requested file does not exist, try index files
+	@indexFiles {
+		file {
+			try_files {path} {path}/index.php index.php
+			split_path .php
+		}
+	}
+	rewrite @indexFiles {http.matchers.file.relative}
+
+	# Proxy PHP files to the FastCGI responder
+	@phpFiles {
+		path *.php
+	}
+	reverse_proxy @phpFiles 127.0.0.1:9000 {
+		transport fastcgi {
+			split .php
+		}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8886"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"headers": {
+														"Location": [
+															"{http.request.uri.path}/"
+														]
+													},
+													"status_code": 308
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"try_files": [
+															"{http.request.uri.path}/index.php"
+														]
+													},
+													"not": [
+														{
+															"path": [
+																"*/"
+															]
+														}
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"uri": "{http.matchers.file.relative}"
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"split_path": [
+															".php"
+														],
+														"try_files": [
+															"{http.request.uri.path}",
+															"{http.request.uri.path}/index.php",
+															"index.php"
+														]
+													}
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"transport": {
+														"protocol": "fastcgi",
+														"split_path": [
+															".php"
+														]
+													},
+													"upstreams": [
+														{
+															"dial": "127.0.0.1:9000"
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"*.php"
+													]
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_index_off.txt

@@ -0,0 +1,66 @@
+:8884
+
+php_fastcgi localhost:9000 {
+    # some php_fastcgi-specific subdirectives
+    split .php .php5
+    env VAR1 value1
+    env VAR2 value2
+    root /var/www
+    index off
+
+    # passed through to reverse_proxy (directive order doesn't matter!)
+    lb_policy random
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"*.php",
+										"*.php5"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"load_balancing": {
+										"selection_policy": {
+											"policy": "random"
+										}
+									},
+									"transport": {
+										"env": {
+											"VAR1": "value1",
+											"VAR2": "value2"
+										},
+										"protocol": "fastcgi",
+										"root": "/var/www",
+										"split_path": [
+											".php",
+											".php5"
+										]
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_matcher.txt

@@ -0,0 +1,112 @@
+:8884
+
+@api host example.com
+php_fastcgi @api localhost:9000
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "static_response",
+													"headers": {
+														"Location": [
+															"{http.request.uri.path}/"
+														]
+													},
+													"status_code": 308
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"try_files": [
+															"{http.request.uri.path}/index.php"
+														]
+													},
+													"not": [
+														{
+															"path": [
+																"*/"
+															]
+														}
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "rewrite",
+													"uri": "{http.matchers.file.relative}"
+												}
+											],
+											"match": [
+												{
+													"file": {
+														"split_path": [
+															".php"
+														],
+														"try_files": [
+															"{http.request.uri.path}",
+															"{http.request.uri.path}/index.php",
+															"index.php"
+														]
+													}
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "reverse_proxy",
+													"transport": {
+														"protocol": "fastcgi",
+														"split_path": [
+															".php"
+														]
+													},
+													"upstreams": [
+														{
+															"dial": "localhost:9000"
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"*.php"
+													]
+												}
+											]
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/php_fastcgi_subdirectives.txt

@@ -0,0 +1,118 @@
+:8884
+
+php_fastcgi localhost:9000 {
+    # some php_fastcgi-specific subdirectives
+    split .php .php5
+    env VAR1 value1
+    env VAR2 value2
+    root /var/www
+    index index.php5
+
+    # passed through to reverse_proxy (directive order doesn't matter!)
+    lb_policy random
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}/index.php5"
+										]
+									},
+									"not": [
+										{
+											"path": [
+												"*/"
+											]
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "static_response",
+									"headers": {
+										"Location": [
+											"{http.request.uri.path}/"
+										]
+									},
+									"status_code": 308
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"file": {
+										"try_files": [
+											"{http.request.uri.path}",
+											"{http.request.uri.path}/index.php5",
+											"index.php5"
+										],
+										"split_path": [
+											".php",
+											".php5"
+										]
+									}
+								}
+							],
+							"handle": [
+								{
+									"handler": "rewrite",
+									"uri": "{http.matchers.file.relative}"
+								}
+							]
+						},
+						{
+							"match": [
+								{
+									"path": [
+										"*.php",
+										"*.php5"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"load_balancing": {
+										"selection_policy": {
+											"policy": "random"
+										}
+									},
+									"transport": {
+										"env": {
+											"VAR1": "value1",
+											"VAR2": "value2"
+										},
+										"protocol": "fastcgi",
+										"root": "/var/www",
+										"split_path": [
+											".php",
+											".php5"
+										]
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_empty_non_http_transport.txt

@@ -0,0 +1,36 @@
+:8884
+
+reverse_proxy 127.0.0.1:65535 {
+    transport fastcgi
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8884"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "reverse_proxy",
+									"transport": {
+										"protocol": "fastcgi"
+									},
+									"upstreams": [
+										{
+											"dial": "127.0.0.1:65535"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/reverse_proxy_h2c.txt

@@ -0,0 +1,98 @@
+
+https://example.com {
+	reverse_proxy /path http://localhost:54321 {
+		header_up Host {host}
+		header_up X-Real-IP {remote}
+		header_up X-Forwarded-For {remote}
+		header_up X-Forwarded-Port {server_port}
+		header_up X-Forwarded-Proto "http"
+		transport http {
+			versions h2c 2
+			compression off
+		}
+		buffer_requests
+	}
+}
+
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"buffer_requests": true,
+													"handler": "reverse_proxy",
+													"headers": {
+														"request": {
+															"set": {
+																"Host": [
+																	"{http.request.host}"
+																],
+																"X-Forwarded-For": [
+																	"{http.request.remote}"
+																],
+																"X-Forwarded-Port": [
+																	"{server_port}"
+																],
+																"X-Forwarded-Proto": [
+																	"http"
+																],
+																"X-Real-Ip": [
+																	"{http.request.remote}"
+																]
+															}
+														}
+													},
+													"transport": {
+														"compression": false,
+														"protocol": "http",
+														"versions": [
+															"h2c",
+															"2"
+														]
+													},
+													"upstreams": [
+														{
+															"dial": "localhost:54321"
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/path"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/shorthand_parameterized_placeholders.txt

@@ -0,0 +1,43 @@
+localhost:80
+respond * "{header.content-type} {labels.0} {query.p} {path.0} {re.name.0}"
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "{http.request.header.content-type} {http.request.host.labels.0} {http.request.uri.query.p} {http.request.uri.path.0} {http.regexp.name.0}",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/sort_directives_with_any_matcher_first.txt

@@ -0,0 +1,55 @@
+:80
+
+file_server
+
+@untrusted not remote_ip 10.1.1.0/24
+file_server @untrusted
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"not": [
+										{
+											"remote_ip": {
+												"ranges": [
+													"10.1.1.0/24"
+												]
+											}
+										}
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "file_server",
+									"hide": [
+										"Caddyfile"
+									]
+								}
+							]
+						},
+						{
+							"handle": [
+								{
+									"handler": "file_server",
+									"hide": [
+										"Caddyfile"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_client_auth_cert_file.txt

@@ -0,0 +1,66 @@
+localhost
+
+respond "hello from localhost"
+tls {
+	client_auth {
+		mode request 
+		trusted_ca_cert_file ../caddy.ca.cer
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"tls_connection_policies": [
+						{
+							"match": {
+								"sni": [
+									"localhost"
+								]
+							},
+							"client_authentication": {
+								"trusted_ca_certs": [
+									"MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkwODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl03WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45twOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNxtdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTUApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAdBgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5uNY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfKD66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEOfG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnkoNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdleIh6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ=="
+								],
+								"mode": "request"
+							}
+						},
+						{}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/tls_client_auth_inline_cert.txt

@@ -0,0 +1,66 @@
+localhost
+
+respond "hello from localhost"
+tls {
+	client_auth {
+		mode request 
+		trusted_ca_cert MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkwODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl03WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45twOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNxtdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTUApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAdBgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5uNY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfKD66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEOfG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnkoNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdleIh6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ== 
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"body": "hello from localhost",
+													"handler": "static_response"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"tls_connection_policies": [
+						{
+							"match": {
+								"sni": [
+									"localhost"
+								]
+							},
+							"client_authentication": {
+								"trusted_ca_certs": [
+									"MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkwODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl03WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45twOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNxtdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTUApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAdBgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5uNY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfKD66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEOfG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnkoNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdleIh6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ=="
+								],
+								"mode": "request"
+							}
+						},
+						{}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt_test.go

@@ -0,0 +1,47 @@
+package integration
+
+import (
+	"io/ioutil"
+	"regexp"
+	"strings"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestCaddyfileAdaptToJSON(t *testing.T) {
+	// load the list of test files from the dir
+	files, err := ioutil.ReadDir("./caddyfile_adapt")
+	if err != nil {
+		t.Errorf("failed to read caddyfile_adapt dir: %s", err)
+	}
+
+	// prep a regexp to fix strings on windows
+	winNewlines := regexp.MustCompile(`\r?\n`)
+
+	for _, f := range files {
+		if f.IsDir() {
+			continue
+		}
+
+		// read the test file
+		filename := f.Name()
+		data, err := ioutil.ReadFile("./caddyfile_adapt/" + filename)
+		if err != nil {
+			t.Errorf("failed to read %s dir: %s", filename, err)
+		}
+
+		// split the Caddyfile (first) and JSON (second) parts
+		parts := strings.Split(string(data), "----------")
+		caddyfile, json := strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1])
+
+		// replace windows newlines in the json with unix newlines
+		json = winNewlines.ReplaceAllString(json, "\n")
+
+		// run the test
+		ok := caddytest.CompareAdapt(t, caddyfile, "caddyfile", json)
+		if !ok {
+			t.Errorf("failed to adapt %s", filename)
+		}
+	}
+}

caddytest/integration/caddyfile_test.go

@@ -0,0 +1,103 @@
+package integration
+
+import (
+	"net/http"
+	"net/url"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestRespond(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
+  {
+    http_port     9080
+    https_port    9443
+  }
+  
+  localhost:9080 {
+    respond /version 200 {
+      body "hello from localhost"
+    }	
+    }
+  `, "caddyfile")
+
+	// act and assert
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost")
+}
+
+func TestRedirect(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+  {
+    http_port     9080
+    https_port    9443
+  }
+  
+  localhost:9080 {
+    
+    redir / http://localhost:9080/hello 301
+    
+    respond /hello 200 {
+      body "hello from localhost"
+    }	
+    }
+  `, "caddyfile")
+
+	// act and assert
+	tester.AssertRedirect("http://localhost:9080/", "http://localhost:9080/hello", 301)
+
+	// follow redirect
+	tester.AssertGetResponse("http://localhost:9080/", 200, "hello from localhost")
+}
+
+func TestDuplicateHosts(t *testing.T) {
+
+	// act and assert
+	caddytest.AssertLoadError(t,
+		`
+    localhost:9080 {
+    }
+  
+    localhost:9080 { 
+    }
+    `,
+		"caddyfile",
+		"duplicate site address not allowed")
+}
+
+func TestReadCookie(t *testing.T) {
+
+	localhost, _ := url.Parse("http://localhost")
+	cookie := http.Cookie{
+		Name:  "clientname",
+		Value: "caddytest",
+	}
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.Client.Jar.SetCookies(localhost, []*http.Cookie{&cookie})
+	tester.InitServer(` 
+  {
+    http_port     9080
+    https_port    9443
+  }
+  
+  localhost:9080 {
+    templates {
+      root testdata
+    }
+    file_server {
+      root testdata
+    }
+  }
+  `, "caddyfile")
+
+	// act and assert
+	tester.AssertGetResponse("http://localhost:9080/cookie.html", 200, "<h2>Cookie.ClientName caddytest</h2>")
+}

caddytest/integration/handler_test.go

@@ -0,0 +1,28 @@
+package integration
+
+import (
+	"net/http"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestBrowse(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	http://localhost:9080 {
+		file_server browse
+	}
+  `, "caddyfile")
+
+	req, err := http.NewRequest(http.MethodGet, "http://localhost:9080/", nil)
+	if err != nil {
+		t.Fail()
+		return
+	}
+	tester.AssertResponseCode(req, 200)
+}

caddytest/integration/map_test.go

@@ -0,0 +1,143 @@
+package integration
+
+import (
+	"bytes"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestMap(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`{
+		http_port     9080
+		https_port    9443
+	}
+
+	localhost:9080 {
+
+		map http.request.method dest-name {
+			default unknown
+			G.T get-called
+			POST post-called
+		}
+
+		respond /version 200 {
+			body "hello from localhost {dest-name}"
+		}	
+	}
+	`, "caddyfile")
+
+	// act and assert
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost get-called")
+	tester.AssertPostResponseBody("http://localhost:9080/version", []string{}, bytes.NewBuffer([]byte{}), 200, "hello from localhost post-called")
+}
+
+func TestMapRespondWithDefault(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`{
+		http_port     9080
+		https_port    9443
+		}
+		
+		localhost:9080 {
+	
+			map http.request.method dest-name {
+				default unknown
+				GET get-called
+			}
+		
+			respond /version 200 {
+				body "hello from localhost {dest-name}"
+			}	
+		}
+	`, "caddyfile")
+
+	// act and assert
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost get-called")
+	tester.AssertPostResponseBody("http://localhost:9080/version", []string{}, bytes.NewBuffer([]byte{}), 200, "hello from localhost unknown")
+}
+
+func TestMapAsJson(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`{
+		"apps": {
+			"http": {
+			"http_port": 9080,
+			"https_port": 9443,
+			"servers": {
+				"srv0": {
+				"listen": [
+					":9080"
+				],
+				"routes": [
+					{
+					"handle": [
+						{
+						"handler": "subroute",
+						"routes": [
+							{
+							"handle": [
+								{
+								"handler": "map",
+								"source": "http.request.method",
+								"destination": "dest-name",
+								"default": "unknown",
+								"items": [
+									{
+									"expression": "GET",
+									"value": "get-called"
+									},
+									{
+									"expression": "POST",
+									"value": "post-called"
+									}
+								]
+								}
+							]
+							},
+							{
+							"handle": [
+								{
+								"body": "hello from localhost {dest-name}",
+								"handler": "static_response",
+								"status_code": 200
+								}
+							],
+							"match": [
+								{
+								"path": [
+									"/version"
+								]
+								}
+							]
+							}
+						]
+						}
+					],
+					"match": [
+						{
+						"host": [
+							"localhost"
+						]
+						}
+					],
+					"terminal": true
+					}
+				]
+				}
+			}
+			}
+		}
+		}
+		`, "json")
+
+	tester.AssertGetResponse("http://localhost:9080/version", 200, "hello from localhost get-called")
+	tester.AssertPostResponseBody("http://localhost:9080/version", []string{}, bytes.NewBuffer([]byte{}), 200, "hello from localhost post-called")
+}

caddytest/integration/reverseproxy_test.go

@@ -0,0 +1,97 @@
+package integration
+
+import (
+	"fmt"
+	"io/ioutil"
+	"net"
+	"net/http"
+	"os"
+	"runtime"
+	"strings"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestReverseProxyHealthCheck(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	http://localhost:2020 {
+		respond "Hello, World!"
+	}
+	http://localhost:2021 {
+		respond "ok"
+	}
+	http://localhost:9080 {
+		reverse_proxy {
+			to localhost:2020
+	
+			health_path /health
+			health_port 2021
+			health_interval 2s
+			health_timeout 5s
+		}
+	}
+  `, "caddyfile")
+
+	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")
+}
+
+func TestReverseProxyHealthCheckUnixSocket(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.SkipNow()
+	}
+	tester := caddytest.NewTester(t)
+	f, err := ioutil.TempFile("", "*.sock")
+	if err != nil {
+		t.Errorf("failed to create TempFile: %s", err)
+		return
+	}
+	// a hack to get a file name within a valid path to use as socket
+	socketName := f.Name()
+	os.Remove(f.Name())
+
+	server := http.Server{
+		Handler: http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
+			if strings.HasPrefix(req.URL.Path, "/health") {
+				w.Write([]byte("ok"))
+				return
+			}
+			w.Write([]byte("Hello, World!"))
+		}),
+	}
+
+	unixListener, err := net.Listen("unix", socketName)
+	if err != nil {
+		t.Errorf("failed to listen on the socket: %s", err)
+		return
+	}
+	go server.Serve(unixListener)
+	t.Cleanup(func() {
+		server.Close()
+	})
+	runtime.Gosched() // Allow other goroutines to run
+
+	tester.InitServer(fmt.Sprintf(`
+	{
+		http_port     9080
+		https_port    9443
+	}
+	http://localhost:9080 {
+		reverse_proxy {
+			to unix/%s
+	
+			health_path /health
+			health_port 2021
+			health_interval 2s
+			health_timeout 5s
+		}
+	}
+	`, socketName), "caddyfile")
+
+	tester.AssertGetResponse("http://localhost:9080/", 200, "Hello, World!")
+}

caddytest/integration/sni_test.go

@@ -0,0 +1,320 @@
+package integration
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestDefaultSNI(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`{
+    "apps": {
+      "http": {
+        "http_port": 9080,
+        "https_port": 9443,
+        "servers": {
+          "srv0": {
+            "listen": [
+              ":9443"
+            ],
+            "routes": [
+              {
+                "handle": [
+                  {
+                    "handler": "subroute",
+                    "routes": [
+                      {
+                        "handle": [
+                          {
+                            "body": "hello from a.caddy.localhost",
+                            "handler": "static_response",
+                            "status_code": 200
+                          }
+                        ],
+                        "match": [
+                          {
+                            "path": [
+                              "/version"
+                            ]
+                          }
+                        ]
+                      }
+                    ]
+                  }
+                ],
+                "match": [
+                  {
+                    "host": [
+                      "127.0.0.1"
+                    ]
+                  }
+                ],
+                "terminal": true
+              }
+            ],
+            "tls_connection_policies": [
+              {
+                "certificate_selection": {
+                  "any_tag": ["cert0"]
+                },
+                "match": {
+                  "sni": [
+                    "127.0.0.1"
+                  ]
+                }
+              },
+              {
+                "default_sni": "*.caddy.localhost"
+              }
+            ]
+          }
+        }
+      },
+      "tls": {
+        "certificates": {
+          "load_files": [
+            {
+              "certificate": "/caddy.localhost.crt",
+              "key": "/caddy.localhost.key",
+              "tags": [
+                "cert0"
+              ]
+            }
+          ]
+        }
+      },
+      "pki": {
+        "certificate_authorities" : {
+          "local" : {
+            "install_trust": false
+          }
+        }
+      }
+    }
+  }
+  `, "json")
+
+	// act and assert
+	// makes a request with no sni
+	tester.AssertGetResponse("https://127.0.0.1:9443/version", 200, "hello from a")
+}
+
+func TestDefaultSNIWithNamedHostAndExplicitIP(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
+  {
+    "apps": {
+      "http": {
+        "http_port": 9080,
+        "https_port": 9443,
+        "servers": {
+          "srv0": {
+            "listen": [
+              ":9443"
+            ],
+            "routes": [
+              {
+                "handle": [
+                  {
+                    "handler": "subroute",
+                    "routes": [
+                      {
+                        "handle": [
+                          {
+                            "body": "hello from a",
+                            "handler": "static_response",
+                            "status_code": 200
+                          }
+                        ],
+                        "match": [
+                          {
+                            "path": [
+                              "/version"
+                            ]
+                          }
+                        ]
+                      }
+                    ]
+                  }
+                ],
+                "match": [
+                  {
+                    "host": [
+                      "a.caddy.localhost",
+                      "127.0.0.1"
+                    ]
+                  }
+                ],
+                "terminal": true
+              }
+            ],
+            "tls_connection_policies": [
+              {
+                "certificate_selection": {
+                  "any_tag": ["cert0"]
+                },
+                "default_sni": "a.caddy.localhost",
+                "match": {
+                  "sni": [
+                    "a.caddy.localhost",
+                    "127.0.0.1",
+                    ""
+                  ]
+                }
+              },
+              {
+                "default_sni": "a.caddy.localhost"
+              }
+            ]
+          }
+        }
+      },
+      "tls": {
+        "certificates": {
+          "load_files": [
+            {
+              "certificate": "/a.caddy.localhost.crt",
+              "key": "/a.caddy.localhost.key",
+              "tags": [
+                "cert0"
+              ]
+            }
+          ]
+        }
+      },
+      "pki": {
+        "certificate_authorities" : {
+          "local" : {
+            "install_trust": false
+          }
+        }
+      }
+    }
+  }
+  `, "json")
+
+	// act and assert
+	// makes a request with no sni
+	tester.AssertGetResponse("https://127.0.0.1:9443/version", 200, "hello from a")
+}
+
+func TestDefaultSNIWithPortMappingOnly(t *testing.T) {
+
+	// arrange
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
+  {
+    "apps": {
+      "http": {
+        "http_port": 9080,
+        "https_port": 9443,
+        "servers": {
+          "srv0": {
+            "listen": [
+              ":9443"
+            ],
+            "routes": [
+              {
+                "handle": [
+                  {
+                    "body": "hello from a.caddy.localhost",
+                    "handler": "static_response",
+                    "status_code": 200
+                  }
+                ],
+                "match": [
+                  {
+                    "path": [
+                      "/version"
+                    ]
+                  }
+                ]
+              }
+            ],
+            "tls_connection_policies": [
+              {
+                "certificate_selection": {
+                  "any_tag": ["cert0"]
+                },
+                "default_sni": "a.caddy.localhost"
+              }
+            ]
+          }
+        }
+      },
+      "tls": {
+        "certificates": {
+          "load_files": [
+            {
+              "certificate": "/a.caddy.localhost.crt",
+              "key": "/a.caddy.localhost.key",
+              "tags": [
+                "cert0"
+              ]
+            }
+          ]
+        }
+      },
+      "pki": {
+        "certificate_authorities" : {
+          "local" : {
+            "install_trust": false
+          }
+        }
+      }
+    }
+  }
+  `, "json")
+
+	// act and assert
+	// makes a request with no sni
+	tester.AssertGetResponse("https://127.0.0.1:9443/version", 200, "hello from a")
+}
+
+func TestHttpOnlyOnDomainWithSNI(t *testing.T) {
+	caddytest.AssertAdapt(t, `
+	{
+		default_sni a.caddy.localhost
+	}
+	:80 {
+		respond /version 200 {
+			body "hello from localhost"
+		}
+	}
+	`, "caddyfile", `{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"path": [
+										"/version"
+									]
+								}
+							],
+							"handle": [
+								{
+									"body": "hello from localhost",
+									"handler": "static_response",
+									"status_code": 200
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}`)
+}

caddytest/integration/stream_test.go

@@ -0,0 +1,437 @@
+package integration
+
+import (
+	"compress/gzip"
+	"context"
+	"crypto/rand"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+	"net/http/httputil"
+	"net/url"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+	"golang.org/x/net/http2"
+	"golang.org/x/net/http2/h2c"
+)
+
+// (see https://github.com/caddyserver/caddy/issues/3556 for use case)
+func TestH2ToH2CStream(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
+  {
+    "apps": {
+      "http": {
+        "http_port": 9080,
+        "https_port": 9443,
+        "servers": {
+          "srv0": {
+            "listen": [
+              ":9443"
+            ],
+            "routes": [
+              {
+                "handle": [
+                  {
+                    "handler": "reverse_proxy",
+                    "transport": {
+                      "protocol": "http",
+                      "compression": false,
+                      "versions": [
+                        "h2c",
+                        "2"
+                      ]
+                    },
+                    "upstreams": [
+                      {
+                        "dial": "localhost:54321"
+                      }
+                    ]
+                  }
+                ],
+                "match": [
+                  {
+                    "path": [
+                      "/tov2ray"
+                    ]
+                  }
+                ]
+              }
+            ],
+            "tls_connection_policies": [
+              {
+                "certificate_selection": {
+                  "any_tag": ["cert0"]
+                },
+                "default_sni": "a.caddy.localhost"
+              }
+            ]
+          }
+        }
+      },
+      "tls": {
+        "certificates": {
+          "load_files": [
+            {
+              "certificate": "/a.caddy.localhost.crt",
+              "key": "/a.caddy.localhost.key",
+              "tags": [
+                "cert0"
+              ]
+            }
+          ]
+        }
+      },
+      "pki": {
+        "certificate_authorities" : {
+          "local" : {
+            "install_trust": false
+          }
+        }
+      }
+    }
+  }
+  `, "json")
+
+	expectedBody := "some data to be echoed"
+	// start the server
+	server := testH2ToH2CStreamServeH2C(t)
+	go server.ListenAndServe()
+	defer func() {
+		ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
+		defer cancel()
+		server.Shutdown(ctx)
+	}()
+
+	r, w := io.Pipe()
+	req := &http.Request{
+		Method: "PUT",
+		Body:   ioutil.NopCloser(r),
+		URL: &url.URL{
+			Scheme: "https",
+			Host:   "127.0.0.1:9443",
+			Path:   "/tov2ray",
+		},
+		Proto:      "HTTP/2",
+		ProtoMajor: 2,
+		ProtoMinor: 0,
+		Header:     make(http.Header),
+	}
+	// Disable any compression method from server.
+	req.Header.Set("Accept-Encoding", "identity")
+
+	resp := tester.AssertResponseCode(req, 200)
+	if 200 != resp.StatusCode {
+		return
+	}
+	go func() {
+		fmt.Fprint(w, expectedBody)
+		w.Close()
+	}()
+
+	defer resp.Body.Close()
+	bytes, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		t.Fatalf("unable to read the response body %s", err)
+	}
+
+	body := string(bytes)
+
+	if !strings.Contains(body, expectedBody) {
+		t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", req.RequestURI, expectedBody, body)
+	}
+	return
+}
+
+func testH2ToH2CStreamServeH2C(t *testing.T) *http.Server {
+	h2s := &http2.Server{}
+	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		rstring, err := httputil.DumpRequest(r, false)
+		if err == nil {
+			t.Logf("h2c server received req: %s", rstring)
+		}
+		// We only accept HTTP/2!
+		if r.ProtoMajor != 2 {
+			t.Error("Not a HTTP/2 request, rejected!")
+			w.WriteHeader(http.StatusInternalServerError)
+			return
+		}
+
+		if r.Host != "127.0.0.1:9443" {
+			t.Errorf("r.Host doesn't match, %v!", r.Host)
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+
+		if !strings.HasPrefix(r.URL.Path, "/tov2ray") {
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+
+		w.Header().Set("Cache-Control", "no-store")
+		w.WriteHeader(200)
+		if f, ok := w.(http.Flusher); ok {
+			f.Flush()
+		}
+
+		buf := make([]byte, 4*1024)
+
+		for {
+			n, err := r.Body.Read(buf)
+			if n > 0 {
+				w.Write(buf[:n])
+			}
+
+			if err != nil {
+				if err == io.EOF {
+					r.Body.Close()
+				}
+				break
+			}
+		}
+	})
+
+	server := &http.Server{
+		Addr:    "127.0.0.1:54321",
+		Handler: h2c.NewHandler(handler, h2s),
+	}
+	return server
+}
+
+// (see https://github.com/caddyserver/caddy/issues/3606 for use case)
+func TestH2ToH1ChunkedResponse(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(` 
+{
+  "logging": {
+    "logs": {
+      "default": {
+        "level": "DEBUG"
+      }
+    }
+  },
+  "apps": {
+    "http": {
+      "http_port": 9080,
+      "https_port": 9443,
+      "servers": {
+        "srv0": {
+          "listen": [
+            ":9443"
+          ],
+          "routes": [
+            {
+              "handle": [
+                {
+                  "handler": "subroute",
+                  "routes": [
+                    {
+                      "handle": [
+                        {
+                          "encodings": {
+                            "gzip": {}
+                          },
+                          "handler": "encode"
+                        }
+                      ]
+                    },
+                    {
+                      "handle": [
+                        {
+                          "handler": "reverse_proxy",
+                          "upstreams": [
+                            {
+                              "dial": "localhost:54321"
+                            }
+                          ]
+                        }
+                      ],
+                      "match": [
+                        {
+                          "path": [
+                            "/tov2ray"
+                          ]
+                        }
+                      ]
+                    }
+                  ]
+                }
+              ],
+              "terminal": true
+            }
+          ],
+          "tls_connection_policies": [
+            {
+              "certificate_selection": {
+                "any_tag": [
+                  "cert0"
+                ]
+              },
+              "default_sni": "a.caddy.localhost"
+            }
+          ]
+        }
+      }
+    },
+    "tls": {
+      "certificates": {
+        "load_files": [
+          {
+            "certificate": "/a.caddy.localhost.crt",
+            "key": "/a.caddy.localhost.key",
+            "tags": [
+              "cert0"
+            ]
+          }
+        ]
+      }
+    },
+    "pki": {
+      "certificate_authorities": {
+        "local": {
+          "install_trust": false
+        }
+      }
+    }
+  }
+}
+  `, "json")
+
+	// need a large body here to trigger caddy's compression, larger than gzip.miniLength
+	expectedBody, err := GenerateRandomString(1024)
+	if err != nil {
+		t.Fatalf("generate expected body failed, err: %s", err)
+	}
+
+	// start the server
+	server := testH2ToH1ChunkedResponseServeH1(t)
+	go server.ListenAndServe()
+	defer func() {
+		ctx, cancel := context.WithTimeout(context.Background(), time.Nanosecond)
+		defer cancel()
+		server.Shutdown(ctx)
+	}()
+
+	r, w := io.Pipe()
+	req := &http.Request{
+		Method: "PUT",
+		Body:   ioutil.NopCloser(r),
+		URL: &url.URL{
+			Scheme: "https",
+			Host:   "127.0.0.1:9443",
+			Path:   "/tov2ray",
+		},
+		Proto:      "HTTP/2",
+		ProtoMajor: 2,
+		ProtoMinor: 0,
+		Header:     make(http.Header),
+	}
+	// underlying transport will automaticlly add gzip
+	// req.Header.Set("Accept-Encoding", "gzip")
+	go func() {
+		fmt.Fprint(w, expectedBody)
+		w.Close()
+	}()
+	resp := tester.AssertResponseCode(req, 200)
+	if 200 != resp.StatusCode {
+		return
+	}
+
+	defer resp.Body.Close()
+	bytes, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		t.Fatalf("unable to read the response body %s", err)
+	}
+
+	body := string(bytes)
+
+	if body != expectedBody {
+		t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", req.RequestURI, expectedBody, body)
+	}
+	return
+}
+
+func testH2ToH1ChunkedResponseServeH1(t *testing.T) *http.Server {
+	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+
+		if r.Host != "127.0.0.1:9443" {
+			t.Errorf("r.Host doesn't match, %v!", r.Host)
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+
+		if !strings.HasPrefix(r.URL.Path, "/tov2ray") {
+			w.WriteHeader(http.StatusNotFound)
+			return
+		}
+
+		defer r.Body.Close()
+		bytes, err := ioutil.ReadAll(r.Body)
+		if err != nil {
+			t.Fatalf("unable to read the response body %s", err)
+		}
+
+		n := len(bytes)
+
+		var writer io.Writer
+		if strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
+			gw, err := gzip.NewWriterLevel(w, 5)
+			if err != nil {
+				t.Error("can't return gzip data")
+				w.WriteHeader(http.StatusInternalServerError)
+				return
+			}
+			defer gw.Close()
+			writer = gw
+			w.Header().Set("Content-Encoding", "gzip")
+			w.Header().Del("Content-Length")
+			w.WriteHeader(200)
+		} else {
+			writer = w
+		}
+		if n > 0 {
+			writer.Write(bytes[:])
+		}
+	})
+
+	server := &http.Server{
+		Addr:    "127.0.0.1:54321",
+		Handler: handler,
+	}
+	return server
+}
+
+// GenerateRandomBytes returns securely generated random bytes.
+// It will return an error if the system's secure random
+// number generator fails to function correctly, in which
+// case the caller should not continue.
+func GenerateRandomBytes(n int) ([]byte, error) {
+	b := make([]byte, n)
+	_, err := rand.Read(b)
+	// Note that err == nil only if we read len(b) bytes.
+	if err != nil {
+		return nil, err
+	}
+
+	return b, nil
+}
+
+// GenerateRandomString returns a securely generated random string.
+// It will return an error if the system's secure random
+// number generator fails to function correctly, in which
+// case the caller should not continue.
+func GenerateRandomString(n int) (string, error) {
+	const letters = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-"
+	bytes, err := GenerateRandomBytes(n)
+	if err != nil {
+		return "", err
+	}
+	for i, b := range bytes {
+		bytes[i] = letters[b%byte(len(letters))]
+	}
+	return string(bytes), nil
+}

caddytest/integration/testdata/cookie.html

@@ -0,0 +1 @@
+<h2>Cookie.ClientName {{.Cookie "clientname"}}</h2>

caddytest/integration/testdata/import_respond.txt

@@ -0,0 +1 @@
+respond "'I am {args.0}', hears {args.1}"

cmd/caddy/main.go

@@ -12,36 +12,25 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package main is the entry point of the Caddy application.
+// Most of Caddy's functionality is provided through modules,
+// which can be plugged in by adding their import below.
+//
+// There is no need to modify the Caddy source code to customize your
+// builds. You can easily build a custom Caddy with these simple steps:
+//
+//   1. Copy this file (main.go) into a new folder
+//   2. Edit the imports below to include the modules you want plugged in
+//   3. Run `go mod init caddy`
+//   4. Run `go install` or `go build` - you now have a custom binary!
+//
 package main
 
 import (
 	caddycmd "github.com/caddyserver/caddy/v2/cmd"
 
-	// this is where modules get plugged in
-	_ "github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
-	_ "github.com/caddyserver/caddy/v2/caddyconfig/json5"
-	_ "github.com/caddyserver/caddy/v2/caddyconfig/jsonc"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyauth"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/encode"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/encode/brotli"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/encode/gzip"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/encode/zstd"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/fileserver"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/headers"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/httpcache"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/markdown"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/requestbody"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/rewrite"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/starlarkmw"
-	_ "github.com/caddyserver/caddy/v2/modules/caddyhttp/templates"
-	_ "github.com/caddyserver/caddy/v2/modules/caddytls"
-	_ "github.com/caddyserver/caddy/v2/modules/caddytls/distributedstek"
-	_ "github.com/caddyserver/caddy/v2/modules/caddytls/standardstek"
-	_ "github.com/caddyserver/caddy/v2/modules/filestorage"
-	_ "github.com/caddyserver/caddy/v2/modules/logging"
+	// plug in Caddy modules here
+	_ "github.com/caddyserver/caddy/v2/modules/standard"
 )
 
 func main() {

cmd/commandfuncs.go

@@ -16,6 +16,7 @@ package caddycmd
 
 import (
 	"bytes"
+	"context"
 	"crypto/rand"
 	"encoding/json"
 	"fmt"
@@ -27,19 +28,22 @@ import (
 	"os"
 	"os/exec"
 	"reflect"
+	"runtime"
 	"runtime/debug"
 	"sort"
 	"strings"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
-	"github.com/keybase/go-ps"
-	"github.com/mholt/certmagic"
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	"go.uber.org/zap"
 )
 
 func cmdStart(fl Flags) (int, error) {
 	startCmdConfigFlag := fl.String("config")
 	startCmdConfigAdapterFlag := fl.String("adapter")
+	startCmdPidfileFlag := fl.String("pidfile")
+	startCmdWatchFlag := fl.Bool("watch")
 
 	// open a listener to which the child process will connect when
 	// it is ready to confirm that it has successfully started
@@ -66,6 +70,12 @@ func cmdStart(fl Flags) (int, error) {
 	if startCmdConfigAdapterFlag != "" {
 		cmd.Args = append(cmd.Args, "--adapter", startCmdConfigAdapterFlag)
 	}
+	if startCmdWatchFlag {
+		cmd.Args = append(cmd.Args, "--watch")
+	}
+	if startCmdPidfileFlag != "" {
+		cmd.Args = append(cmd.Args, "--pidfile", startCmdPidfileFlag)
+	}
 	stdinpipe, err := cmd.StdinPipe()
 	if err != nil {
 		return caddy.ExitCodeFailedStartup,
@@ -83,7 +93,7 @@ func cmdStart(fl Flags) (int, error) {
 
 	// begin writing the confirmation bytes to the child's
 	// stdin; use a goroutine since the child hasn't been
-	// started yet, and writing sychronously would result
+	// started yet, and writing synchronously would result
 	// in a deadlock
 	go func() {
 		stdinpipe.Write(expect)
@@ -129,7 +139,7 @@ func cmdStart(fl Flags) (int, error) {
 	// when one of the goroutines unblocks, we're done and can exit
 	select {
 	case <-success:
-		fmt.Printf("Successfully started Caddy (pid=%d)\n", cmd.Process.Pid)
+		fmt.Printf("Successfully started Caddy (pid=%d) - Caddy is running in the background\n", cmd.Process.Pid)
 	case err := <-exit:
 		return caddy.ExitCodeFailedStartup,
 			fmt.Errorf("caddy process exited with error: %v", err)
@@ -139,34 +149,71 @@ func cmdStart(fl Flags) (int, error) {
 }
 
 func cmdRun(fl Flags) (int, error) {
+	caddy.TrapSignals()
+
 	runCmdConfigFlag := fl.String("config")
 	runCmdConfigAdapterFlag := fl.String("adapter")
+	runCmdResumeFlag := fl.Bool("resume")
+	runCmdLoadEnvfileFlag := fl.String("envfile")
 	runCmdPrintEnvFlag := fl.Bool("environ")
+	runCmdWatchFlag := fl.Bool("watch")
+	runCmdPidfileFlag := fl.String("pidfile")
 	runCmdPingbackFlag := fl.String("pingback")
 
+	// load all additional envs as soon as possible
+	if runCmdLoadEnvfileFlag != "" {
+		if err := loadEnvFromFile(runCmdLoadEnvfileFlag); err != nil {
+			return caddy.ExitCodeFailedStartup,
+				fmt.Errorf("loading additional environment variables: %v", err)
+		}
+	}
+
 	// if we are supposed to print the environment, do that first
 	if runCmdPrintEnvFlag {
 		printEnvironment()
 	}
 
-	// get the config in caddy's native format
-	config, err := loadConfig(runCmdConfigFlag, runCmdConfigAdapterFlag)
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, err
+	// TODO: This is TEMPORARY, until the RCs
+	moveStorage()
+
+	// load the config, depending on flags
+	var config []byte
+	var err error
+	if runCmdResumeFlag {
+		config, err = ioutil.ReadFile(caddy.ConfigAutosavePath)
+		if os.IsNotExist(err) {
+			// not a bad error; just can't resume if autosave file doesn't exist
+			caddy.Log().Info("no autosave file exists", zap.String("autosave_file", caddy.ConfigAutosavePath))
+			runCmdResumeFlag = false
+		} else if err != nil {
+			return caddy.ExitCodeFailedStartup, err
+		} else {
+			if runCmdConfigFlag == "" {
+				caddy.Log().Info("resuming from last configuration",
+					zap.String("autosave_file", caddy.ConfigAutosavePath))
+			} else {
+				// if they also specified a config file, user should be aware that we're not
+				// using it (doing so could lead to data/config loss by overwriting!)
+				caddy.Log().Warn("--config and --resume flags were used together; ignoring --config and resuming from last configuration",
+					zap.String("autosave_file", caddy.ConfigAutosavePath))
+			}
+		}
+	}
+	// we don't use 'else' here since this value might have been changed in 'if' block; i.e. not mutually exclusive
+	var configFile string
+	if !runCmdResumeFlag {
+		config, configFile, err = loadConfig(runCmdConfigFlag, runCmdConfigAdapterFlag)
+		if err != nil {
+			return caddy.ExitCodeFailedStartup, err
+		}
 	}
 
-	// set a fitting User-Agent for ACME requests
-	goModule := caddy.GoModule()
-	cleanModVersion := strings.TrimPrefix(goModule.Version, "v")
-	certmagic.UserAgent = "Caddy/" + cleanModVersion
-
-	// start the admin endpoint along with any initial config
-	err = caddy.StartAdmin(config)
+	// run the initial config
+	err = caddy.Load(config, true)
 	if err != nil {
-		return caddy.ExitCodeFailedStartup,
-			fmt.Errorf("starting caddy administration endpoint: %v", err)
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("loading initial config: %v", err)
 	}
-	defer caddy.StopAdmin()
+	caddy.Log().Info("serving initial configuration")
 
 	// if we are to report to another process the successful start
 	// of the server, do so now by echoing back contents of stdin
@@ -189,31 +236,53 @@ func cmdRun(fl Flags) (int, error) {
 		}
 	}
 
+	// if enabled, reload config file automatically on changes
+	// (this better only be used in dev!)
+	if runCmdWatchFlag {
+		go watchConfigFile(configFile, runCmdConfigAdapterFlag)
+	}
+
+	// create pidfile
+	if runCmdPidfileFlag != "" {
+		err := caddy.PIDFile(runCmdPidfileFlag)
+		if err != nil {
+			caddy.Log().Error("unable to write PID file",
+				zap.String("pidfile", runCmdPidfileFlag),
+				zap.Error(err))
+		}
+	}
+
+	// warn if the environment does not provide enough information about the disk
+	hasXDG := os.Getenv("XDG_DATA_HOME") != "" &&
+		os.Getenv("XDG_CONFIG_HOME") != "" &&
+		os.Getenv("XDG_CACHE_HOME") != ""
+	switch runtime.GOOS {
+	case "windows":
+		if os.Getenv("HOME") == "" && os.Getenv("USERPROFILE") == "" && !hasXDG {
+			caddy.Log().Warn("neither HOME nor USERPROFILE environment variables are set - please fix; some assets might be stored in ./caddy")
+		}
+	case "plan9":
+		if os.Getenv("home") == "" && !hasXDG {
+			caddy.Log().Warn("$home environment variable is empty - please fix; some assets might be stored in ./caddy")
+		}
+	default:
+		if os.Getenv("HOME") == "" && !hasXDG {
+			caddy.Log().Warn("$HOME environment variable is empty - please fix; some assets might be stored in ./caddy")
+		}
+	}
+
 	select {}
 }
 
-func cmdStop(_ Flags) (int, error) {
-	processList, err := ps.Processes()
-	if err != nil {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("listing processes: %v", err)
-	}
-	thisProcName := getProcessName()
-	var found bool
-	for _, p := range processList {
-		// the process we're looking for should have the same name but different PID
-		if p.Executable() == thisProcName && p.Pid() != os.Getpid() {
-			found = true
-			fmt.Printf("pid=%d\n", p.Pid())
+func cmdStop(fl Flags) (int, error) {
+	stopCmdAddrFlag := fl.String("address")
 
-			if err := gracefullyStopProcess(p.Pid()); err != nil {
-				return caddy.ExitCodeFailedStartup, err
-			}
-		}
+	err := apiRequest(stopCmdAddrFlag, http.MethodPost, "/stop", nil)
+	if err != nil {
+		caddy.Log().Warn("failed using API to stop instance", zap.Error(err))
+		return caddy.ExitCodeFailedStartup, err
 	}
-	if !found {
-		return caddy.ExitCodeFailedStartup, fmt.Errorf("Caddy is not running")
-	}
-	fmt.Println(" success")
+
 	return caddy.ExitCodeSuccess, nil
 }
 
@@ -222,21 +291,18 @@ func cmdReload(fl Flags) (int, error) {
 	reloadCmdConfigAdapterFlag := fl.String("adapter")
 	reloadCmdAddrFlag := fl.String("address")
 
-	// a configuration is required
-	if reloadCmdConfigFlag == "" {
-		return caddy.ExitCodeFailedStartup,
-			fmt.Errorf("no configuration to load (use --config)")
-	}
-
 	// get the config in caddy's native format
-	config, err := loadConfig(reloadCmdConfigFlag, reloadCmdConfigAdapterFlag)
+	config, configFile, err := loadConfig(reloadCmdConfigFlag, reloadCmdConfigAdapterFlag)
 	if err != nil {
 		return caddy.ExitCodeFailedStartup, err
 	}
+	if configFile == "" {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("no config file to load")
+	}
 
-	// get the address of the admin listener and craft endpoint URL
+	// get the address of the admin listener; use flag if specified
 	adminAddr := reloadCmdAddrFlag
-	if adminAddr == "" {
+	if adminAddr == "" && len(config) > 0 {
 		var tmpStruct struct {
 			Admin caddy.AdminConfig `json:"admin"`
 		}
@@ -247,28 +313,10 @@ func cmdReload(fl Flags) (int, error) {
 		}
 		adminAddr = tmpStruct.Admin.Listen
 	}
-	if adminAddr == "" {
-		adminAddr = caddy.DefaultAdminListen
-	}
-	adminEndpoint := fmt.Sprintf("http://%s/load", adminAddr)
 
-	// send the configuration to the instance
-	resp, err := http.Post(adminEndpoint, "application/json", bytes.NewReader(config))
+	err = apiRequest(adminAddr, http.MethodPost, "/load", bytes.NewReader(config))
 	if err != nil {
-		return caddy.ExitCodeFailedStartup,
-			fmt.Errorf("sending configuration to instance: %v", err)
-	}
-	defer resp.Body.Close()
-
-	// if it didn't work, let the user know
-	if resp.StatusCode >= 400 {
-		respBody, err := ioutil.ReadAll(io.LimitReader(resp.Body, 1024*10))
-		if err != nil {
-			return caddy.ExitCodeFailedStartup,
-				fmt.Errorf("HTTP %d: reading error message: %v", resp.StatusCode, err)
-		}
-		return caddy.ExitCodeFailedStartup,
-			fmt.Errorf("caddy responded with error: HTTP %d: %s", resp.StatusCode, respBody)
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("sending configuration to instance: %v", err)
 	}
 
 	return caddy.ExitCodeSuccess, nil
@@ -276,11 +324,37 @@ func cmdReload(fl Flags) (int, error) {
 
 func cmdVersion(_ Flags) (int, error) {
 	goModule := caddy.GoModule()
+	fmt.Print(goModule.Version)
 	if goModule.Sum != "" {
 		// a build with a known version will also have a checksum
-		fmt.Printf("%s %s\n", goModule.Version, goModule.Sum)
-	} else {
-		fmt.Println(goModule.Version)
+		fmt.Printf(" %s", goModule.Sum)
+	}
+	if goModule.Replace != nil {
+		fmt.Printf(" => %s", goModule.Replace.Path)
+		if goModule.Replace.Version != "" {
+			fmt.Printf(" %s", goModule.Replace.Version)
+		}
+	}
+	fmt.Println()
+	return caddy.ExitCodeSuccess, nil
+}
+
+func cmdBuildInfo(fl Flags) (int, error) {
+	bi, ok := debug.ReadBuildInfo()
+	if !ok {
+		return caddy.ExitCodeFailedStartup, fmt.Errorf("no build information")
+	}
+
+	fmt.Printf("path: %s\n", bi.Path)
+	fmt.Printf("main: %s %s %s\n", bi.Main.Path, bi.Main.Version, bi.Main.Sum)
+	fmt.Println("dependencies:")
+
+	for _, goMod := range bi.Deps {
+		fmt.Printf("%s %s %s", goMod.Path, goMod.Version, goMod.Sum)
+		if goMod.Replace != nil {
+			fmt.Printf(" => %s %s %s", goMod.Replace.Path, goMod.Replace.Version, goMod.Replace.Sum)
+		}
+		fmt.Println()
 	}
 	return caddy.ExitCodeSuccess, nil
 }
@@ -298,11 +372,11 @@ func cmdListModules(fl Flags) (int, error) {
 		return caddy.ExitCodeSuccess, nil
 	}
 
-	for _, modName := range caddy.Modules() {
-		modInfo, err := caddy.GetModule(modName)
+	for _, modID := range caddy.Modules() {
+		modInfo, err := caddy.GetModule(modID)
 		if err != nil {
 			// that's weird
-			fmt.Println(modName)
+			fmt.Println(modID)
 			continue
 		}
 
@@ -331,13 +405,13 @@ func cmdListModules(fl Flags) (int, error) {
 		}
 
 		// if we could find no matching module, just print out
-		// the module name instead
+		// the module ID instead
 		if matched == nil {
-			fmt.Println(modName)
+			fmt.Println(modID)
 			continue
 		}
 
-		fmt.Printf("%s %s\n", modName, matched.Version)
+		fmt.Printf("%s %s\n", modID, matched.Version)
 	}
 
 	return caddy.ExitCodeSuccess, nil
@@ -354,9 +428,27 @@ func cmdAdaptConfig(fl Flags) (int, error) {
 	adaptCmdPrettyFlag := fl.Bool("pretty")
 	adaptCmdValidateFlag := fl.Bool("validate")
 
-	if adaptCmdAdapterFlag == "" || adaptCmdInputFlag == "" {
+	// if no input file was specified, try a default
+	// Caddyfile if the Caddyfile adapter is plugged in
+	if adaptCmdInputFlag == "" && caddyconfig.GetAdapter("caddyfile") != nil {
+		_, err := os.Stat("Caddyfile")
+		if err == nil {
+			// default Caddyfile exists
+			adaptCmdInputFlag = "Caddyfile"
+			caddy.Log().Info("using adjacent Caddyfile")
+		} else if !os.IsNotExist(err) {
+			// default Caddyfile exists, but error accessing it
+			return caddy.ExitCodeFailedStartup, fmt.Errorf("accessing default Caddyfile: %v", err)
+		}
+	}
+
+	if adaptCmdInputFlag == "" {
 		return caddy.ExitCodeFailedStartup,
-			fmt.Errorf("--adapter and --config flags are required")
+			fmt.Errorf("input file required when there is no Caddyfile in current directory (use --config flag)")
+	}
+	if adaptCmdAdapterFlag == "" {
+		return caddy.ExitCodeFailedStartup,
+			fmt.Errorf("adapter name is required (use --adapt flag or leave unspecified for default)")
 	}
 
 	cfgAdapter := caddyconfig.GetAdapter(adaptCmdAdapterFlag)
@@ -375,6 +467,7 @@ func cmdAdaptConfig(fl Flags) (int, error) {
 	if adaptCmdPrettyFlag {
 		opts["pretty"] = "true"
 	}
+	opts["filename"] = adaptCmdInputFlag
 
 	adaptedConfig, warnings, err := cfgAdapter.Adapt(input, opts)
 	if err != nil {
@@ -413,34 +506,11 @@ func cmdValidateConfig(fl Flags) (int, error) {
 	validateCmdConfigFlag := fl.String("config")
 	validateCmdAdapterFlag := fl.String("adapter")
 
-	input, err := ioutil.ReadFile(validateCmdConfigFlag)
+	input, _, err := loadConfig(validateCmdConfigFlag, validateCmdAdapterFlag)
 	if err != nil {
-		return caddy.ExitCodeFailedStartup,
-			fmt.Errorf("reading input file: %v", err)
-	}
-
-	if validateCmdAdapterFlag != "" {
-		cfgAdapter := caddyconfig.GetAdapter(validateCmdAdapterFlag)
-		if cfgAdapter == nil {
-			return caddy.ExitCodeFailedStartup,
-				fmt.Errorf("unrecognized config adapter: %s", validateCmdAdapterFlag)
-		}
-
-		adaptedConfig, warnings, err := cfgAdapter.Adapt(input, nil)
-		if err != nil {
-			return caddy.ExitCodeFailedStartup, err
-		}
-		// print warnings to stderr
-		for _, warn := range warnings {
-			msg := warn.Message
-			if warn.Directive != "" {
-				msg = fmt.Sprintf("%s: %s", warn.Directive, warn.Message)
-			}
-			fmt.Fprintf(os.Stderr, "[WARNING][%s] %s:%d: %s\n", validateCmdAdapterFlag, warn.File, warn.Line, msg)
-		}
-
-		input = adaptedConfig
+		return caddy.ExitCodeFailedStartup, err
 	}
+	input = caddy.RemoveMetaFields(input)
 
 	var cfg *caddy.Config
 	err = json.Unmarshal(input, &cfg)
@@ -458,9 +528,45 @@ func cmdValidateConfig(fl Flags) (int, error) {
 	return caddy.ExitCodeSuccess, nil
 }
 
+func cmdFmt(fl Flags) (int, error) {
+	formatCmdConfigFile := fl.Arg(0)
+	if formatCmdConfigFile == "" {
+		formatCmdConfigFile = "Caddyfile"
+	}
+
+	// as a special case, read from stdin if the file name is "-"
+	if formatCmdConfigFile == "-" {
+		input, err := ioutil.ReadAll(os.Stdin)
+		if err != nil {
+			return caddy.ExitCodeFailedStartup,
+				fmt.Errorf("reading stdin: %v", err)
+		}
+		fmt.Print(string(caddyfile.Format(input)))
+		return caddy.ExitCodeSuccess, nil
+	}
+
+	input, err := ioutil.ReadFile(formatCmdConfigFile)
+	if err != nil {
+		return caddy.ExitCodeFailedStartup,
+			fmt.Errorf("reading input file: %v", err)
+	}
+
+	output := caddyfile.Format(input)
+
+	if fl.Bool("overwrite") {
+		if err := ioutil.WriteFile(formatCmdConfigFile, output, 0600); err != nil {
+			return caddy.ExitCodeFailedStartup, nil
+		}
+	} else {
+		fmt.Print(string(output))
+	}
+
+	return caddy.ExitCodeSuccess, nil
+}
+
 func cmdHelp(fl Flags) (int, error) {
 	const fullDocs = `Full documentation is available at:
-https://github.com/caddyserver/caddy/wiki/v2:-Documentation`
+https://caddyserver.com/docs/command-line`
 
 	args := fl.Args()
 	if len(args) == 0 {
@@ -521,3 +627,76 @@ commands:
 
 	return caddy.ExitCodeSuccess, nil
 }
+
+// apiRequest makes an API request to the endpoint adminAddr with the
+// given HTTP method and request URI. If body is non-nil, it will be
+// assumed to be Content-Type application/json.
+func apiRequest(adminAddr, method, uri string, body io.Reader) error {
+	// parse the admin address
+	if adminAddr == "" {
+		adminAddr = caddy.DefaultAdminListen
+	}
+	parsedAddr, err := caddy.ParseNetworkAddress(adminAddr)
+	if err != nil || parsedAddr.PortRangeSize() > 1 {
+		return fmt.Errorf("invalid admin address %s: %v", adminAddr, err)
+	}
+	origin := parsedAddr.JoinHostPort(0)
+	if parsedAddr.IsUnixNetwork() {
+		origin = "unixsocket" // hack so that http.NewRequest() is happy
+	}
+
+	// form the request
+	req, err := http.NewRequest(method, "http://"+origin+uri, body)
+	if err != nil {
+		return fmt.Errorf("making request: %v", err)
+	}
+	if parsedAddr.IsUnixNetwork() {
+		// When listening on a unix socket, the admin endpoint doesn't
+		// accept any Host header because there is no host:port for
+		// a unix socket's address. The server's host check is fairly
+		// strict for security reasons, so we don't allow just any
+		// Host header. For unix sockets, the Host header must be
+		// empty. Unfortunately, Go makes it impossible to make HTTP
+		// requests with an empty Host header... except with this one
+		// weird trick. (Hopefully they don't fix it. It's already
+		// hard enough to use HTTP over unix sockets.)
+		//
+		// An equivalent curl command would be something like:
+		// $ curl --unix-socket caddy.sock http:/:$REQUEST_URI
+		req.URL.Host = " "
+		req.Host = ""
+	} else {
+		req.Header.Set("Origin", origin)
+	}
+	if body != nil {
+		req.Header.Set("Content-Type", "application/json")
+	}
+
+	// make an HTTP client that dials our network type, since admin
+	// endpoints aren't always TCP, which is what the default transport
+	// expects; reuse is not of particular concern here
+	client := http.Client{
+		Transport: &http.Transport{
+			DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
+				return net.Dial(parsedAddr.Network, parsedAddr.JoinHostPort(0))
+			},
+		},
+	}
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return fmt.Errorf("performing request: %v", err)
+	}
+	defer resp.Body.Close()
+
+	// if it didn't work, let the user know
+	if resp.StatusCode >= 400 {
+		respBody, err := ioutil.ReadAll(io.LimitReader(resp.Body, 1024*10))
+		if err != nil {
+			return fmt.Errorf("HTTP %d: reading error message: %v", resp.StatusCode, err)
+		}
+		return fmt.Errorf("caddy responded with error: HTTP %d: %s", resp.StatusCode, respBody)
+	}
+
+	return nil
+}

cmd/commands.go

@@ -27,7 +27,7 @@ type Command struct {
 	// Required.
 	Name string
 
-	// Run is a function that executes a subcommand using
+	// Func is a function that executes a subcommand using
 	// the parsed flags. It returns an exit code and any
 	// associated error.
 	// Required.
@@ -74,7 +74,7 @@ func init() {
 	RegisterCommand(Command{
 		Name:  "start",
 		Func:  cmdStart,
-		Usage: "[--config <path> [[--adapter <name>]]",
+		Usage: "[--config <path> [--adapter <name>]] [--watch] [--pidfile <file>]",
 		Short: "Starts the Caddy process in the background and then returns",
 		Long: `
 Starts the Caddy process, optionally bootstrapped with an initial config file.
@@ -87,6 +87,8 @@ using 'caddy run' instead to keep it in the foreground.`,
 			fs := flag.NewFlagSet("start", flag.ExitOnError)
 			fs.String("config", "", "Configuration file")
 			fs.String("adapter", "", "Name of config adapter to apply")
+			fs.String("pidfile", "", "Path of file to which to write process ID")
+			fs.Bool("watch", false, "Reload changed config file automatically")
 			return fs
 		}(),
 	})
@@ -94,7 +96,7 @@ using 'caddy run' instead to keep it in the foreground.`,
 	RegisterCommand(Command{
 		Name:  "run",
 		Func:  cmdRun,
-		Usage: "[--config <path> [--adapter <name>]] [--environ]",
+		Usage: "[--config <path> [--adapter <name>]] [--envfile <path>] [--environ] [--resume] [--watch] [--pidfile <fil>]",
 		Short: `Starts the Caddy process and blocks indefinitely`,
 		Long: `
 Starts the Caddy process, optionally bootstrapped with an initial config file,
@@ -114,14 +116,28 @@ As a special case, if the current working directory has a file called
 that file will be loaded and used to configure Caddy, even without any command
 line flags.
 
+If --envfile is specified, an environment file with environment variables in
+the KEY=VALUE format will be loaded into the Caddy process.
+
 If --environ is specified, the environment as seen by the Caddy process will
 be printed before starting. This is the same as the environ command but does
-not quit after printing, and can be useful for troubleshooting.`,
+not quit after printing, and can be useful for troubleshooting.
+
+The --resume flag will override the --config flag if there is a config auto-
+save file. It is not an error if --resume is used and no autosave file exists.
+
+If --watch is specified, the config file will be loaded automatically after
+changes. ⚠️ This is dangerous in production! Only use this option in a local
+development environment.`,
 		Flags: func() *flag.FlagSet {
 			fs := flag.NewFlagSet("run", flag.ExitOnError)
 			fs.String("config", "", "Configuration file")
 			fs.String("adapter", "", "Name of config adapter to apply")
+			fs.String("envfile", "", "Environment file to load")
 			fs.Bool("environ", false, "Print environment")
+			fs.Bool("resume", false, "Use saved config, if any (and prefer over --config file)")
+			fs.Bool("watch", false, "Watch config file for changes and reload it automatically")
+			fs.String("pidfile", "", "Path of file to which to write process ID")
 			fs.String("pingback", "", "Echo confirmation bytes to this address on success")
 			return fs
 		}(),
@@ -134,11 +150,14 @@ not quit after printing, and can be useful for troubleshooting.`,
 		Long: `
 Stops the background Caddy process as gracefully as possible.
 
-On Windows, this stop is forceful and Caddy will not have an opportunity to
-clean up any active locks; for a graceful shutdown on Windows, use Ctrl+C
-or the /stop API endpoint.
-
-Note: this will stop any process named the same as the executable (os.Args[0]).`,
+It requires that the admin API is enabled and accessible, since it will
+use the API's /stop endpoint. The address of this request can be
+customized using the --address flag if it is not the default.`,
+		Flags: func() *flag.FlagSet {
+			fs := flag.NewFlagSet("stop", flag.ExitOnError)
+			fs.String("address", "", "The address to use to reach the admin API endpoint, if not the default")
+			return fs
+		}(),
 	})
 
 	RegisterCommand(Command{
@@ -181,6 +200,12 @@ config file; otherwise the default is assumed.`,
 		}(),
 	})
 
+	RegisterCommand(Command{
+		Name:  "build-info",
+		Func:  cmdBuildInfo,
+		Short: "Prints information about this build",
+	})
+
 	RegisterCommand(Command{
 		Name:  "environ",
 		Func:  cmdEnviron,
@@ -229,6 +254,28 @@ provisioning stages.`,
 		}(),
 	})
 
+	RegisterCommand(Command{
+		Name:  "fmt",
+		Func:  cmdFmt,
+		Usage: "[--overwrite] [<path>]",
+		Short: "Formats a Caddyfile",
+		Long: `
+Formats the Caddyfile by adding proper indentation and spaces to improve
+human readability. It prints the result to stdout.
+
+If --overwrite is specified, the output will be written to the config file
+directly instead of printing it.
+
+If you wish you use stdin instead of a regular file, use - as the path.
+When reading from stdin, the --overwrite flag has no effect: the result
+is always printed to stdout.`,
+		Flags: func() *flag.FlagSet {
+			fs := flag.NewFlagSet("format", flag.ExitOnError)
+			fs.Bool("overwrite", false, "Overwrite the input file with the results")
+			return fs
+		}(),
+	})
+
 }
 
 // RegisterCommand registers the command cmd.
@@ -243,6 +290,8 @@ provisioning stages.`,
 // This function panics if the name is already registered,
 // if the name does not meet the described format, or if
 // any of the fields are missing from cmd.
+//
+// This function should be used in init().
 func RegisterCommand(cmd Command) {
 	if cmd.Name == "" {
 		panic("command name is required")

cmd/main.go

@@ -15,26 +15,42 @@
 package caddycmd
 
 import (
+	"bufio"
 	"bytes"
 	"flag"
 	"fmt"
 	"io"
 	"io/ioutil"
+	"log"
 	"net"
 	"os"
+	"path/filepath"
+	"runtime"
+	"runtime/debug"
 	"strconv"
 	"strings"
 	"time"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/certmagic"
+	"go.uber.org/zap"
 )
 
+func init() {
+	// set a fitting User-Agent for ACME requests
+	goModule := caddy.GoModule()
+	cleanModVersion := strings.TrimPrefix(goModule.Version, "v")
+	certmagic.UserAgent = "Caddy/" + cleanModVersion
+
+	// by using Caddy, user indicates agreement to CA terms
+	// (very important, or ACME account creation will fail!)
+	certmagic.DefaultACME.Agreed = true
+}
+
 // Main implements the main function of the caddy command.
 // Call this if Caddy is to be the main() if your program.
 func Main() {
-	caddy.TrapSignals()
-
 	switch len(os.Args) {
 	case 0:
 		fmt.Printf("[FATAL] no arguments provided by OS; args[0] must be command\n")
@@ -68,7 +84,7 @@ func Main() {
 
 	exitCode, err := subcommand.Func(Flags{fs})
 	if err != nil {
-		fmt.Printf("%s: %v\n", subcommand.Name, err)
+		fmt.Fprintf(os.Stderr, "%s: %v\n", subcommand.Name, err)
 	}
 
 	os.Exit(exitCode)
@@ -90,12 +106,16 @@ func handlePingbackConn(conn net.Conn, expect []byte) error {
 
 // loadConfig loads the config from configFile and adapts it
 // using adapterName. If adapterName is specified, configFile
-// must be also. It prints any warnings to stderr, and returns
-// the resulting JSON config bytes.
-func loadConfig(configFile, adapterName string) ([]byte, error) {
+// must be also. If no configFile is specified, it tries
+// loading a default config file. The lack of a config file is
+// not treated as an error, but false will be returned if
+// there is no config available. It prints any warnings to stderr,
+// and returns the resulting JSON config bytes along with
+// whether a config file was loaded or not.
+func loadConfig(configFile, adapterName string) ([]byte, string, error) {
 	// specifying an adapter without a config file is ambiguous
-	if configFile == "" && adapterName != "" {
-		return nil, fmt.Errorf("cannot adapt config without config file (use --config)")
+	if adapterName != "" && configFile == "" {
+		return nil, "", fmt.Errorf("cannot adapt config without config file (use --config)")
 	}
 
 	// load initial config and adapter
@@ -105,8 +125,11 @@ func loadConfig(configFile, adapterName string) ([]byte, error) {
 	if configFile != "" {
 		config, err = ioutil.ReadFile(configFile)
 		if err != nil {
-			return nil, fmt.Errorf("reading config file: %v", err)
+			return nil, "", fmt.Errorf("reading config file: %v", err)
 		}
+		caddy.Log().Info("using provided configuration",
+			zap.String("config_file", configFile),
+			zap.String("config_adapter", adapterName))
 	} else if adapterName == "" {
 		// as a special case when no config file or adapter
 		// is specified, see if the Caddyfile adapter is
@@ -119,19 +142,29 @@ func loadConfig(configFile, adapterName string) ([]byte, error) {
 				cfgAdapter = nil
 			} else if err != nil {
 				// default Caddyfile exists, but error reading it
-				return nil, fmt.Errorf("reading default Caddyfile: %v", err)
+				return nil, "", fmt.Errorf("reading default Caddyfile: %v", err)
 			} else {
 				// success reading default Caddyfile
 				configFile = "Caddyfile"
+				caddy.Log().Info("using adjacent Caddyfile")
 			}
 		}
 	}
 
+	// as a special case, if a config file called "Caddyfile" was
+	// specified, and no adapter is specified, assume caddyfile adapter
+	// for convenience
+	if strings.HasPrefix(filepath.Base(configFile), "Caddyfile") &&
+		filepath.Ext(configFile) != ".json" &&
+		adapterName == "" {
+		adapterName = "caddyfile"
+	}
+
 	// load config adapter
 	if adapterName != "" {
 		cfgAdapter = caddyconfig.GetAdapter(adapterName)
 		if cfgAdapter == nil {
-			return nil, fmt.Errorf("unrecognized config adapter: %s", adapterName)
+			return nil, "", fmt.Errorf("unrecognized config adapter: %s", adapterName)
 		}
 	}
 
@@ -141,7 +174,7 @@ func loadConfig(configFile, adapterName string) ([]byte, error) {
 			"filename": configFile,
 		})
 		if err != nil {
-			return nil, fmt.Errorf("adapting config using %s: %v", adapterName, err)
+			return nil, "", fmt.Errorf("adapting config using %s: %v", adapterName, err)
 		}
 		for _, warn := range warnings {
 			msg := warn.Message
@@ -153,7 +186,91 @@ func loadConfig(configFile, adapterName string) ([]byte, error) {
 		config = adaptedConfig
 	}
 
-	return config, nil
+	return config, configFile, nil
+}
+
+// watchConfigFile watches the config file at filename for changes
+// and reloads the config if the file was updated. This function
+// blocks indefinitely; it only quits if the poller has errors for
+// long enough time. The filename passed in must be the actual
+// config file used, not one to be discovered.
+func watchConfigFile(filename, adapterName string) {
+	defer func() {
+		if err := recover(); err != nil {
+			log.Printf("[PANIC] watching config file: %v\n%s", err, debug.Stack())
+		}
+	}()
+
+	// make our logger; since config reloads can change the
+	// default logger, we need to get it dynamically each time
+	logger := func() *zap.Logger {
+		return caddy.Log().
+			Named("watcher").
+			With(zap.String("config_file", filename))
+	}
+
+	// get the initial timestamp on the config file
+	info, err := os.Stat(filename)
+	if err != nil {
+		logger().Error("cannot watch config file", zap.Error(err))
+		return
+	}
+	lastModified := info.ModTime()
+
+	logger().Info("watching config file for changes")
+
+	// if the file disappears or something, we can
+	// stop polling if the error lasts long enough
+	var lastErr time.Time
+	finalError := func(err error) bool {
+		if lastErr.IsZero() {
+			lastErr = time.Now()
+			return false
+		}
+		if time.Since(lastErr) > 30*time.Second {
+			logger().Error("giving up watching config file; too many errors",
+				zap.Error(err))
+			return true
+		}
+		return false
+	}
+
+	// begin poller
+	for range time.Tick(1 * time.Second) {
+		// get the file info
+		info, err := os.Stat(filename)
+		if err != nil {
+			if finalError(err) {
+				return
+			}
+			continue
+		}
+		lastErr = time.Time{} // no error, so clear any memory of one
+
+		// if it hasn't changed, nothing to do
+		if !info.ModTime().After(lastModified) {
+			continue
+		}
+
+		logger().Info("config file changed; reloading")
+
+		// remember this timestamp
+		lastModified = info.ModTime()
+
+		// load the contents of the file
+		config, _, err := loadConfig(filename, adapterName)
+		if err != nil {
+			logger().Error("unable to load latest config", zap.Error(err))
+			continue
+		}
+
+		// apply the updated config
+		err = caddy.Load(config, false)
+		if err != nil {
+			logger().Error("applying latest config", zap.Error(err))
+			continue
+		}
+	}
 }
 
 // Flags wraps a FlagSet so that typed values
@@ -201,7 +318,7 @@ func (f Flags) Float64(name string) float64 {
 // is not a duration type. It panics if the flag is
 // not in the flag set.
 func (f Flags) Duration(name string) time.Duration {
-	val, _ := time.ParseDuration(f.String(name))
+	val, _ := caddy.ParseDuration(f.String(name))
 	return val
 }
 
@@ -221,8 +338,159 @@ func flagHelp(fs *flag.FlagSet) string {
 	return buf.String()
 }
 
+func loadEnvFromFile(envFile string) error {
+	file, err := os.Open(envFile)
+	if err != nil {
+		return fmt.Errorf("reading environment file: %v", err)
+	}
+	defer file.Close()
+
+	envMap, err := parseEnvFile(file)
+	if err != nil {
+		return fmt.Errorf("parsing environment file: %v", err)
+	}
+
+	for k, v := range envMap {
+		if err := os.Setenv(k, v); err != nil {
+			return fmt.Errorf("setting environment variables: %v", err)
+		}
+	}
+
+	return nil
+}
+
+func parseEnvFile(envInput io.Reader) (map[string]string, error) {
+	envMap := make(map[string]string)
+
+	scanner := bufio.NewScanner(envInput)
+	var line string
+	lineNumber := 0
+
+	for scanner.Scan() {
+		line = strings.TrimSpace(scanner.Text())
+		lineNumber++
+
+		// skip lines starting with comment
+		if strings.HasPrefix(line, "#") {
+			continue
+		}
+
+		// skip empty line
+		if len(line) == 0 {
+			continue
+		}
+
+		fields := strings.SplitN(line, "=", 2)
+		if len(fields) != 2 {
+			return nil, fmt.Errorf("can't parse line %d; line should be in KEY=VALUE format", lineNumber)
+		}
+
+		if strings.Contains(fields[0], " ") {
+			return nil, fmt.Errorf("bad key on line %d: contains whitespace", lineNumber)
+		}
+
+		key := fields[0]
+		val := fields[1]
+
+		if key == "" {
+			return nil, fmt.Errorf("missing or empty key on line %d", lineNumber)
+		}
+		envMap[key] = val
+	}
+
+	if err := scanner.Err(); err != nil {
+		return nil, err
+	}
+
+	return envMap, nil
+}
+
 func printEnvironment() {
+	fmt.Printf("caddy.HomeDir=%s\n", caddy.HomeDir())
+	fmt.Printf("caddy.AppDataDir=%s\n", caddy.AppDataDir())
+	fmt.Printf("caddy.AppConfigDir=%s\n", caddy.AppConfigDir())
+	fmt.Printf("caddy.ConfigAutosavePath=%s\n", caddy.ConfigAutosavePath)
+	fmt.Printf("caddy.Version=%s\n", caddy.GoModule().Version)
+	fmt.Printf("runtime.GOOS=%s\n", runtime.GOOS)
+	fmt.Printf("runtime.GOARCH=%s\n", runtime.GOARCH)
+	fmt.Printf("runtime.Compiler=%s\n", runtime.Compiler)
+	fmt.Printf("runtime.NumCPU=%d\n", runtime.NumCPU())
+	fmt.Printf("runtime.GOMAXPROCS=%d\n", runtime.GOMAXPROCS(0))
+	fmt.Printf("runtime.Version=%s\n", runtime.Version())
+	cwd, err := os.Getwd()
+	if err != nil {
+		cwd = fmt.Sprintf("<error: %v>", err)
+	}
+	fmt.Printf("os.Getwd=%s\n\n", cwd)
 	for _, v := range os.Environ() {
 		fmt.Println(v)
 	}
 }
+
+// moveStorage moves the old default dataDir to the new default dataDir.
+// TODO: This is TEMPORARY until the release candidates.
+func moveStorage() {
+	// get the home directory (the old way)
+	oldHome := os.Getenv("HOME")
+	if oldHome == "" && runtime.GOOS == "windows" {
+		drive := os.Getenv("HOMEDRIVE")
+		path := os.Getenv("HOMEPATH")
+		oldHome = drive + path
+		if drive == "" || path == "" {
+			oldHome = os.Getenv("USERPROFILE")
+		}
+	}
+	if oldHome == "" {
+		oldHome = "."
+	}
+	oldDataDir := filepath.Join(oldHome, ".local", "share", "caddy")
+
+	// nothing to do if old data dir doesn't exist
+	_, err := os.Stat(oldDataDir)
+	if os.IsNotExist(err) {
+		return
+	}
+
+	// nothing to do if the new data dir is the same as the old one
+	newDataDir := caddy.AppDataDir()
+	if oldDataDir == newDataDir {
+		return
+	}
+
+	logger := caddy.Log().Named("automigrate").With(
+		zap.String("old_dir", oldDataDir),
+		zap.String("new_dir", newDataDir))
+
+	logger.Info("beginning one-time data directory migration",
+		zap.String("details", "https://github.com/caddyserver/caddy/issues/2955"))
+
+	// if new data directory exists, avoid auto-migration as a conservative safety measure
+	_, err = os.Stat(newDataDir)
+	if !os.IsNotExist(err) {
+		logger.Error("new data directory already exists; skipping auto-migration as conservative safety measure",
+			zap.Error(err),
+			zap.String("instructions", "https://github.com/caddyserver/caddy/issues/2955#issuecomment-570000333"))
+		return
+	}
+
+	// construct the new data directory's parent folder
+	err = os.MkdirAll(filepath.Dir(newDataDir), 0700)
+	if err != nil {
+		logger.Error("unable to make new datadirectory - follow link for instructions",
+			zap.String("instructions", "https://github.com/caddyserver/caddy/issues/2955#issuecomment-570000333"),
+			zap.Error(err))
+		return
+	}
+
+	// folder structure is same, so just try to rename (move) it;
+	// this fails if the new path is on a separate device
+	err = os.Rename(oldDataDir, newDataDir)
+	if err != nil {
+		logger.Error("new data directory already exists; skipping auto-migration as conservative safety measure - follow link for instructions",
+			zap.String("instructions", "https://github.com/caddyserver/caddy/issues/2955#issuecomment-570000333"),
+			zap.Error(err))
+	}
+
+	logger.Info("successfully completed one-time migration of data directory",
+		zap.String("details", "https://github.com/caddyserver/caddy/issues/2955"))
+}

cmd/proc_posix.go

@@ -24,7 +24,7 @@ import (
 )
 
 func gracefullyStopProcess(pid int) error {
-	fmt.Printf("Graceful stop...\n")
+	fmt.Print("Graceful stop... ")
 	err := syscall.Kill(pid, syscall.SIGINT)
 	if err != nil {
 		return fmt.Errorf("kill: %v", err)

cmd/proc_windows.go

@@ -23,7 +23,7 @@ import (
 )
 
 func gracefullyStopProcess(pid int) error {
-	fmt.Printf("Forceful Stop...\n")
+	fmt.Print("Forceful stop... ")
 	// process on windows will not stop unless forced with /f
 	cmd := exec.Command("taskkill", "/pid", strconv.Itoa(pid), "/f")
 	if err := cmd.Run(); err != nil {

context.go

@@ -21,7 +21,7 @@ import (
 	"log"
 	"reflect"
 
-	"github.com/mholt/certmagic"
+	"github.com/caddyserver/certmagic"
 	"go.uber.org/zap"
 )
 
@@ -30,7 +30,7 @@ import (
 // that spawned the modules which are loaded. It should be used
 // with care and wrapped with derivation functions from the
 // standard context package only if you don't need the Caddy
-// specific features. These contexts are cancelled when the
+// specific features. These contexts are canceled when the
 // lifetime of the modules loaded from it is over.
 //
 // Use NewContext() to get a valid value (but most modules will
@@ -75,29 +75,240 @@ func NewContext(ctx Context) (Context, context.CancelFunc) {
 	return newCtx, wrappedCancel
 }
 
-// OnCancel executes f when ctx is cancelled.
+// OnCancel executes f when ctx is canceled.
 func (ctx *Context) OnCancel(f func()) {
 	ctx.cleanupFuncs = append(ctx.cleanupFuncs, f)
 }
 
-// LoadModule decodes rawMsg into a new instance of mod and
-// returns the value. If mod.New() does not return a pointer
-// value, it is converted to one so that it is unmarshaled
-// into the underlying concrete type. If mod.New is nil, an
-// error is returned. If the module implements Validator or
-// Provisioner interfaces, those methods are invoked to
-// ensure the module is fully configured and valid before
-// being used.
-func (ctx Context) LoadModule(name string, rawMsg json.RawMessage) (interface{}, error) {
-	modulesMu.Lock()
-	mod, ok := modules[name]
-	modulesMu.Unlock()
+// LoadModule loads the Caddy module(s) from the specified field of the parent struct
+// pointer and returns the loaded module(s). The struct pointer and its field name as
+// a string are necessary so that reflection can be used to read the struct tag on the
+// field to get the module namespace and inline module name key (if specified).
+//
+// The field can be any one of the supported raw module types: json.RawMessage,
+// []json.RawMessage, map[string]json.RawMessage, or []map[string]json.RawMessage.
+// ModuleMap may be used in place of map[string]json.RawMessage. The return value's
+// underlying type mirrors the input field's type:
+//
+//    json.RawMessage              => interface{}
+//    []json.RawMessage            => []interface{}
+//    [][]json.RawMessage          => [][]interface{}
+//    map[string]json.RawMessage   => map[string]interface{}
+//    []map[string]json.RawMessage => []map[string]interface{}
+//
+// The field must have a "caddy" struct tag in this format:
+//
+//    caddy:"key1=val1 key2=val2"
+//
+// To load modules, a "namespace" key is required. For example, to load modules
+// in the "http.handlers" namespace, you'd put: `namespace=http.handlers` in the
+// Caddy struct tag.
+//
+// The module name must also be available. If the field type is a map or slice of maps,
+// then key is assumed to be the module name if an "inline_key" is NOT specified in the
+// caddy struct tag. In this case, the module name does NOT need to be specified in-line
+// with the module itself.
+//
+// If not a map, or if inline_key is non-empty, then the module name must be embedded
+// into the values, which must be objects; then there must be a key in those objects
+// where its associated value is the module name. This is called the "inline key",
+// meaning the key containing the module's name that is defined inline with the module
+// itself. You must specify the inline key in a struct tag, along with the namespace:
+//
+//    caddy:"namespace=http.handlers inline_key=handler"
+//
+// This will look for a key/value pair like `"handler": "..."` in the json.RawMessage
+// in order to know the module name.
+//
+// To make use of the loaded module(s) (the return value), you will probably want
+// to type-assert each interface{} value(s) to the types that are useful to you
+// and store them on the same struct. Storing them on the same struct makes for
+// easy garbage collection when your host module is no longer needed.
+//
+// Loaded modules have already been provisioned and validated. Upon returning
+// successfully, this method clears the json.RawMessage(s) in the field since
+// the raw JSON is no longer needed, and this allows the GC to free up memory.
+func (ctx Context) LoadModule(structPointer interface{}, fieldName string) (interface{}, error) {
+	val := reflect.ValueOf(structPointer).Elem().FieldByName(fieldName)
+	typ := val.Type()
+
+	field, ok := reflect.TypeOf(structPointer).Elem().FieldByName(fieldName)
 	if !ok {
-		return nil, fmt.Errorf("unknown module: %s", name)
+		panic(fmt.Sprintf("field %s does not exist in %#v", fieldName, structPointer))
+	}
+
+	opts, err := ParseStructTag(field.Tag.Get("caddy"))
+	if err != nil {
+		panic(fmt.Sprintf("malformed tag on field %s: %v", fieldName, err))
+	}
+
+	moduleNamespace, ok := opts["namespace"]
+	if !ok {
+		panic(fmt.Sprintf("missing 'namespace' key in struct tag on field %s", fieldName))
+	}
+	inlineModuleKey := opts["inline_key"]
+
+	var result interface{}
+
+	switch val.Kind() {
+	case reflect.Slice:
+		if isJSONRawMessage(typ) {
+			// val is `json.RawMessage` ([]uint8 under the hood)
+
+			if inlineModuleKey == "" {
+				panic("unable to determine module name without inline_key when type is not a ModuleMap")
+			}
+			val, err := ctx.loadModuleInline(inlineModuleKey, moduleNamespace, val.Interface().(json.RawMessage))
+			if err != nil {
+				return nil, err
+			}
+			result = val
+
+		} else if isJSONRawMessage(typ.Elem()) {
+			// val is `[]json.RawMessage`
+
+			if inlineModuleKey == "" {
+				panic("unable to determine module name without inline_key because type is not a ModuleMap")
+			}
+			var all []interface{}
+			for i := 0; i < val.Len(); i++ {
+				val, err := ctx.loadModuleInline(inlineModuleKey, moduleNamespace, val.Index(i).Interface().(json.RawMessage))
+				if err != nil {
+					return nil, fmt.Errorf("position %d: %v", i, err)
+				}
+				all = append(all, val)
+			}
+			result = all
+
+		} else if typ.Elem().Kind() == reflect.Slice && isJSONRawMessage(typ.Elem().Elem()) {
+			// val is `[][]json.RawMessage`
+
+			if inlineModuleKey == "" {
+				panic("unable to determine module name without inline_key because type is not a ModuleMap")
+			}
+			var all [][]interface{}
+			for i := 0; i < val.Len(); i++ {
+				innerVal := val.Index(i)
+				var allInner []interface{}
+				for j := 0; j < innerVal.Len(); j++ {
+					innerInnerVal, err := ctx.loadModuleInline(inlineModuleKey, moduleNamespace, innerVal.Index(j).Interface().(json.RawMessage))
+					if err != nil {
+						return nil, fmt.Errorf("position %d: %v", j, err)
+					}
+					allInner = append(allInner, innerInnerVal)
+				}
+				all = append(all, allInner)
+			}
+			result = all
+
+		} else if isModuleMapType(typ.Elem()) {
+			// val is `[]map[string]json.RawMessage`
+
+			var all []map[string]interface{}
+			for i := 0; i < val.Len(); i++ {
+				thisSet, err := ctx.loadModulesFromSomeMap(moduleNamespace, inlineModuleKey, val.Index(i))
+				if err != nil {
+					return nil, err
+				}
+				all = append(all, thisSet)
+			}
+			result = all
+		}
+
+	case reflect.Map:
+		// val is a ModuleMap or some other kind of map
+		result, err = ctx.loadModulesFromSomeMap(moduleNamespace, inlineModuleKey, val)
+		if err != nil {
+			return nil, err
+		}
+
+	default:
+		return nil, fmt.Errorf("unrecognized type for module: %s", typ)
+	}
+
+	// we're done with the raw bytes; allow GC to deallocate
+	val.Set(reflect.Zero(typ))
+
+	return result, nil
+}
+
+// loadModulesFromSomeMap loads modules from val, which must be a type of map[string]interface{}.
+// Depending on inlineModuleKey, it will be interpreted as either a ModuleMap (key is the module
+// name) or as a regular map (key is not the module name, and module name is defined inline).
+func (ctx Context) loadModulesFromSomeMap(namespace, inlineModuleKey string, val reflect.Value) (map[string]interface{}, error) {
+	// if no inline_key is specified, then val must be a ModuleMap,
+	// where the key is the module name
+	if inlineModuleKey == "" {
+		if !isModuleMapType(val.Type()) {
+			panic(fmt.Sprintf("expected ModuleMap because inline_key is empty; but we do not recognize this type: %s", val.Type()))
+		}
+		return ctx.loadModuleMap(namespace, val)
+	}
+
+	// otherwise, val is a map with modules, but the module name is
+	// inline with each value (the key means something else)
+	return ctx.loadModulesFromRegularMap(namespace, inlineModuleKey, val)
+}
+
+// loadModulesFromRegularMap loads modules from val, where val is a map[string]json.RawMessage.
+// Map keys are NOT interpreted as module names, so module names are still expected to appear
+// inline with the objects.
+func (ctx Context) loadModulesFromRegularMap(namespace, inlineModuleKey string, val reflect.Value) (map[string]interface{}, error) {
+	mods := make(map[string]interface{})
+	iter := val.MapRange()
+	for iter.Next() {
+		k := iter.Key()
+		v := iter.Value()
+		mod, err := ctx.loadModuleInline(inlineModuleKey, namespace, v.Interface().(json.RawMessage))
+		if err != nil {
+			return nil, fmt.Errorf("key %s: %v", k, err)
+		}
+		mods[k.String()] = mod
+	}
+	return mods, nil
+}
+
+// loadModuleMap loads modules from a ModuleMap, i.e. map[string]interface{}, where the key is the
+// module name. With a module map, module names do not need to be defined inline with their values.
+func (ctx Context) loadModuleMap(namespace string, val reflect.Value) (map[string]interface{}, error) {
+	all := make(map[string]interface{})
+	iter := val.MapRange()
+	for iter.Next() {
+		k := iter.Key().Interface().(string)
+		v := iter.Value().Interface().(json.RawMessage)
+		moduleName := namespace + "." + k
+		if namespace == "" {
+			moduleName = k
+		}
+		val, err := ctx.LoadModuleByID(moduleName, v)
+		if err != nil {
+			return nil, fmt.Errorf("module name '%s': %v", k, err)
+		}
+		all[k] = val
+	}
+	return all, nil
+}
+
+// LoadModuleByID decodes rawMsg into a new instance of mod and
+// returns the value. If mod.New is nil, an error is returned.
+// If the module implements Validator or Provisioner interfaces,
+// those methods are invoked to ensure the module is fully
+// configured and valid before being used.
+//
+// This is a lower-level method and will usually not be called
+// directly by most modules. However, this method is useful when
+// dynamically loading/unloading modules in their own context,
+// like from embedded scripts, etc.
+func (ctx Context) LoadModuleByID(id string, rawMsg json.RawMessage) (interface{}, error) {
+	modulesMu.RLock()
+	mod, ok := modules[id]
+	modulesMu.RUnlock()
+	if !ok {
+		return nil, fmt.Errorf("unknown module: %s", id)
 	}
 
 	if mod.New == nil {
-		return nil, fmt.Errorf("module '%s' has no constructor", mod.Name)
+		return nil, fmt.Errorf("module '%s' has no constructor", mod.ID)
 	}
 
 	val := mod.New().(interface{})
@@ -108,7 +319,7 @@ func (ctx Context) LoadModule(name string, rawMsg json.RawMessage) (interface{},
 	if rv := reflect.ValueOf(val); rv.Kind() != reflect.Ptr {
 		log.Printf("[WARNING] ModuleInfo.New() for module '%s' did not return a pointer,"+
 			" so we are using reflection to make a pointer instead; please fix this by"+
-			" using new(Type) or &Type notation in your module's New() function.", name)
+			" using new(Type) or &Type notation in your module's New() function.", id)
 		val = reflect.New(rv.Type()).Elem().Addr().Interface().(Module)
 	}
 
@@ -116,7 +327,7 @@ func (ctx Context) LoadModule(name string, rawMsg json.RawMessage) (interface{},
 	if len(rawMsg) > 0 {
 		err := strictUnmarshalJSON(rawMsg, &val)
 		if err != nil {
-			return nil, fmt.Errorf("decoding module config: %s: %v", mod.Name, err)
+			return nil, fmt.Errorf("decoding module config: %s: %v", mod, err)
 		}
 	}
 
@@ -124,8 +335,8 @@ func (ctx Context) LoadModule(name string, rawMsg json.RawMessage) (interface{},
 		// returned module values are almost always type-asserted
 		// before being used, so a nil value would panic; and there
 		// is no good reason to explicitly declare null modules in
-		// a config; it might be because the user is trying to
-		// achieve a result they aren't expecting, which is a smell
+		// a config; it might be because the user is trying to achieve
+		// a result the developer isn't expecting, which is a smell
 		return nil, fmt.Errorf("module value cannot be null")
 	}
 
@@ -140,7 +351,7 @@ func (ctx Context) LoadModule(name string, rawMsg json.RawMessage) (interface{},
 					err = fmt.Errorf("%v; additionally, cleanup: %v", err, err2)
 				}
 			}
-			return nil, fmt.Errorf("provision %s: %v", mod.Name, err)
+			return nil, fmt.Errorf("provision %s: %v", mod, err)
 		}
 	}
 
@@ -154,33 +365,33 @@ func (ctx Context) LoadModule(name string, rawMsg json.RawMessage) (interface{},
 					err = fmt.Errorf("%v; additionally, cleanup: %v", err, err2)
 				}
 			}
-			return nil, fmt.Errorf("%s: invalid configuration: %v", mod.Name, err)
+			return nil, fmt.Errorf("%s: invalid configuration: %v", mod, err)
 		}
 	}
 
-	ctx.moduleInstances[name] = append(ctx.moduleInstances[name], val)
+	ctx.moduleInstances[id] = append(ctx.moduleInstances[id], val)
 
 	return val, nil
 }
 
-// LoadModuleInline loads a module from a JSON raw message which decodes
-// to a map[string]interface{}, where one of the keys is moduleNameKey
-// and the corresponding value is the module name as a string, which
-// can be found in the given scope.
+// loadModuleInline loads a module from a JSON raw message which decodes to
+// a map[string]interface{}, where one of the object keys is moduleNameKey
+// and the corresponding value is the module name (as a string) which can
+// be found in the given scope. In other words, the module name is declared
+// in-line with the module itself.
 //
-// This allows modules to be decoded into their concrete types and
-// used when their names cannot be the unique key in a map, such as
-// when there are multiple instances in the map or it appears in an
-// array (where there are no custom keys). In other words, the key
-// containing the module name is treated special/separate from all
-// the other keys.
-func (ctx Context) LoadModuleInline(moduleNameKey, moduleScope string, raw json.RawMessage) (interface{}, error) {
+// This allows modules to be decoded into their concrete types and used when
+// their names cannot be the unique key in a map, such as when there are
+// multiple instances in the map or it appears in an array (where there are
+// no custom keys). In other words, the key containing the module name is
+// treated special/separate from all the other keys in the object.
+func (ctx Context) loadModuleInline(moduleNameKey, moduleScope string, raw json.RawMessage) (interface{}, error) {
 	moduleName, raw, err := getModuleNameInline(moduleNameKey, raw)
 	if err != nil {
 		return nil, err
 	}
 
-	val, err := ctx.LoadModule(moduleScope+"."+moduleName, raw)
+	val, err := ctx.LoadModuleByID(moduleScope+"."+moduleName, raw)
 	if err != nil {
 		return nil, fmt.Errorf("loading module '%s': %v", moduleName, err)
 	}
@@ -188,16 +399,25 @@ func (ctx Context) LoadModuleInline(moduleNameKey, moduleScope string, raw json.
 	return val, nil
 }
 
-// App returns the configured app named name. If no app with
-// that name is currently configured, a new empty one will be
-// instantiated. (The app module must still be registered.)
+// App returns the configured app named name. If that app has
+// not yet been loaded and provisioned, it will be immediately
+// loaded and provisioned. If no app with that name is
+// configured, a new empty one will be instantiated instead.
+// (The app module must still be registered.) This must not be
+// called during the Provision/Validate phase to reference a
+// module's own host app (since the parent app module is still
+// in the process of being provisioned, it is not yet ready).
 func (ctx Context) App(name string) (interface{}, error) {
 	if app, ok := ctx.cfg.apps[name]; ok {
 		return app, nil
 	}
-	modVal, err := ctx.LoadModule(name, nil)
+	appRaw := ctx.cfg.AppsRaw[name]
+	modVal, err := ctx.LoadModuleByID(name, appRaw)
 	if err != nil {
-		return nil, fmt.Errorf("instantiating new module %s: %v", name, err)
+		return nil, fmt.Errorf("loading %s app module: %v", name, err)
+	}
+	if appRaw != nil {
+		ctx.cfg.AppsRaw[name] = nil // allow GC to deallocate
 	}
 	ctx.cfg.apps[name] = modVal.(App)
 	return modVal, nil

context_test.go

@@ -0,0 +1,118 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddy
+
+import (
+	"encoding/json"
+	"io"
+)
+
+func ExampleContext_LoadModule() {
+	// this whole first part is just setting up for the example;
+	// note the struct tags - very important; we specify inline_key
+	// because that is the only way to know the module name
+	var ctx Context
+	myStruct := &struct {
+		// This godoc comment will appear in module documentation.
+		GuestModuleRaw json.RawMessage `json:"guest_module,omitempty" caddy:"namespace=example inline_key=name"`
+
+		// this is where the decoded module will be stored; in this
+		// example, we pretend we need an io.Writer but it can be
+		// any interface type that is useful to you
+		guestModule io.Writer
+	}{
+		GuestModuleRaw: json.RawMessage(`{"name":"module_name","foo":"bar"}`),
+	}
+
+	// if a guest module is provided, we can load it easily
+	if myStruct.GuestModuleRaw != nil {
+		mod, err := ctx.LoadModule(myStruct, "GuestModuleRaw")
+		if err != nil {
+			// you'd want to actually handle the error here
+			// return fmt.Errorf("loading guest module: %v", err)
+		}
+		// mod contains the loaded and provisioned module,
+		// it is now ready for us to use
+		myStruct.guestModule = mod.(io.Writer)
+	}
+
+	// use myStruct.guestModule from now on
+}
+
+func ExampleContext_LoadModule_array() {
+	// this whole first part is just setting up for the example;
+	// note the struct tags - very important; we specify inline_key
+	// because that is the only way to know the module name
+	var ctx Context
+	myStruct := &struct {
+		// This godoc comment will appear in module documentation.
+		GuestModulesRaw []json.RawMessage `json:"guest_modules,omitempty" caddy:"namespace=example inline_key=name"`
+
+		// this is where the decoded module will be stored; in this
+		// example, we pretend we need an io.Writer but it can be
+		// any interface type that is useful to you
+		guestModules []io.Writer
+	}{
+		GuestModulesRaw: []json.RawMessage{
+			json.RawMessage(`{"name":"module1_name","foo":"bar1"}`),
+			json.RawMessage(`{"name":"module2_name","foo":"bar2"}`),
+		},
+	}
+
+	// since our input is []json.RawMessage, the output will be []interface{}
+	mods, err := ctx.LoadModule(myStruct, "GuestModulesRaw")
+	if err != nil {
+		// you'd want to actually handle the error here
+		// return fmt.Errorf("loading guest modules: %v", err)
+	}
+	for _, mod := range mods.([]interface{}) {
+		myStruct.guestModules = append(myStruct.guestModules, mod.(io.Writer))
+	}
+
+	// use myStruct.guestModules from now on
+}
+
+func ExampleContext_LoadModule_map() {
+	// this whole first part is just setting up for the example;
+	// note the struct tags - very important; we don't specify
+	// inline_key because the map key is the module name
+	var ctx Context
+	myStruct := &struct {
+		// This godoc comment will appear in module documentation.
+		GuestModulesRaw ModuleMap `json:"guest_modules,omitempty" caddy:"namespace=example"`
+
+		// this is where the decoded module will be stored; in this
+		// example, we pretend we need an io.Writer but it can be
+		// any interface type that is useful to you
+		guestModules map[string]io.Writer
+	}{
+		GuestModulesRaw: ModuleMap{
+			"module1_name": json.RawMessage(`{"foo":"bar1"}`),
+			"module2_name": json.RawMessage(`{"foo":"bar2"}`),
+		},
+	}
+
+	// since our input is map[string]json.RawMessage, the output will be map[string]interface{}
+	mods, err := ctx.LoadModule(myStruct, "GuestModulesRaw")
+	if err != nil {
+		// you'd want to actually handle the error here
+		// return fmt.Errorf("loading guest modules: %v", err)
+	}
+	for modName, mod := range mods.(map[string]interface{}) {
+		myStruct.guestModules[modName] = mod.(io.Writer)
+	}
+
+	// use myStruct.guestModules from now on
+}

dynamicconfig.go

@@ -1,358 +0,0 @@
-// Copyright 2015 Matthew Holt and The Caddy Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package caddy
-
-import (
-	"bytes"
-	"encoding/json"
-	"fmt"
-	"log"
-	"net/http"
-	"path"
-	"strconv"
-	"strings"
-	"sync"
-)
-
-func init() {
-	RegisterModule(router{})
-}
-
-type router []AdminRoute
-
-// CaddyModule returns the Caddy module information.
-func (router) CaddyModule() ModuleInfo {
-	return ModuleInfo{
-		Name: "admin.routers.dynamic_config",
-		New: func() Module {
-			return router{
-				{
-					Pattern: "/" + rawConfigKey + "/",
-					Handler: http.HandlerFunc(handleConfig),
-				},
-				{
-					Pattern: "/id/",
-					Handler: http.HandlerFunc(handleConfigID),
-				},
-			}
-		},
-	}
-}
-
-func (r router) Routes() []AdminRoute { return r }
-
-// handleConfig handles config changes or exports according to r.
-// This function is safe for concurrent use.
-func handleConfig(w http.ResponseWriter, r *http.Request) {
-	rawCfgMu.Lock()
-	defer rawCfgMu.Unlock()
-	unsyncedHandleConfig(w, r)
-}
-
-// handleConfigID accesses the config through a user-assigned ID
-// that is mapped to its full/expanded path in the JSON structure.
-// It is the same as handleConfig except it replaces the ID in
-// the request path with the full, expanded URL path.
-// This function is safe for concurrent use.
-func handleConfigID(w http.ResponseWriter, r *http.Request) {
-	parts := strings.Split(r.URL.Path, "/")
-	if len(parts) < 3 || parts[2] == "" {
-		http.Error(w, "request path is missing object ID", http.StatusBadRequest)
-		return
-	}
-	id := parts[2]
-
-	rawCfgMu.Lock()
-	defer rawCfgMu.Unlock()
-
-	// map the ID to the expanded path
-	expanded, ok := rawCfgIndex[id]
-	if !ok {
-		http.Error(w, "unknown object ID: "+id, http.StatusBadRequest)
-		return
-	}
-
-	// piece the full URL path back together
-	parts = append([]string{expanded}, parts[3:]...)
-	r.URL.Path = path.Join(parts...)
-
-	unsyncedHandleConfig(w, r)
-}
-
-// configIndex recurisvely searches ptr for object fields named "@id"
-// and maps that ID value to the full configPath in the index.
-// This function is NOT safe for concurrent access; use rawCfgMu.
-func configIndex(ptr interface{}, configPath string, index map[string]string) error {
-	switch val := ptr.(type) {
-	case map[string]interface{}:
-		for k, v := range val {
-			if k == "@id" {
-				switch idVal := v.(type) {
-				case string:
-					index[idVal] = configPath
-				case float64: // all JSON numbers decode as float64
-					index[fmt.Sprintf("%v", idVal)] = configPath
-				default:
-					return fmt.Errorf("%s: @id field must be a string or number", configPath)
-				}
-				delete(val, "@id") // field is no longer needed, and will break config if not removed
-				continue
-			}
-			// traverse this object property recursively
-			err := configIndex(val[k], path.Join(configPath, k), index)
-			if err != nil {
-				return err
-			}
-		}
-	case []interface{}:
-		// traverse each element of the array recursively
-		for i := range val {
-			err := configIndex(val[i], path.Join(configPath, strconv.Itoa(i)), index)
-			if err != nil {
-				return err
-			}
-		}
-	}
-
-	return nil
-}
-
-// unsycnedHandleConfig handles config accesses without a lock
-// on rawCfgMu. This is NOT safe for concurrent use, so be sure
-// to acquire a lock on rawCfgMu before calling this.
-func unsyncedHandleConfig(w http.ResponseWriter, r *http.Request) {
-	switch r.Method {
-	case http.MethodGet, http.MethodPost, http.MethodPut, http.MethodPatch, http.MethodDelete:
-	default:
-		http.Error(w, http.StatusText(http.StatusMethodNotAllowed), http.StatusMethodNotAllowed)
-		return
-	}
-
-	// perform the mutation with our decoded representation
-	// (the map), which may change pointers deep within it
-	err := mutateConfig(w, r)
-	if err != nil {
-		http.Error(w, "mutating config: "+err.Error(), http.StatusBadRequest)
-		return
-	}
-
-	if r.Method != http.MethodGet {
-		// find any IDs in this config and index them
-		idx := make(map[string]string)
-		err = configIndex(rawCfg[rawConfigKey], "/config", idx)
-		if err != nil {
-			http.Error(w, "indexing config: "+err.Error(), http.StatusInternalServerError)
-			return
-		}
-
-		// the mutation is complete, so encode the entire config as JSON
-		newCfg, err := json.Marshal(rawCfg[rawConfigKey])
-		if err != nil {
-			http.Error(w, "encoding new config: "+err.Error(), http.StatusBadRequest)
-			return
-		}
-
-		// if nothing changed, no need to do a whole reload unless the client forces it
-		if r.Header.Get("Cache-Control") != "must-revalidate" && bytes.Equal(rawCfgJSON, newCfg) {
-			log.Printf("[ADMIN][INFO] Config is unchanged")
-			return
-		}
-
-		// load this new config; if it fails, we need to revert to
-		// our old representation of caddy's actual config
-		err = Load(bytes.NewReader(newCfg))
-		if err != nil {
-			// restore old config state to keep it consistent
-			// with what caddy is still running; we need to
-			// unmarshal it again because it's likely that
-			// pointers deep in our rawCfg map were modified
-			var oldCfg interface{}
-			err2 := json.Unmarshal(rawCfgJSON, &oldCfg)
-			if err2 != nil {
-				err = fmt.Errorf("%v; additionally, restoring old config: %v", err, err2)
-			}
-			rawCfg[rawConfigKey] = oldCfg
-
-			// report error
-			log.Printf("[ADMIN][ERROR] loading config: %v", err)
-			http.Error(w, err.Error(), http.StatusBadRequest)
-			return
-		}
-
-		// success, so update our stored copy of the encoded
-		// config to keep it consistent with what caddy is now
-		// running (storing an encoded copy is not strictly
-		// necessary, but avoids an extra json.Marshal for
-		// each config change)
-		rawCfgJSON = newCfg
-		rawCfgIndex = idx
-	}
-}
-
-// mutateConfig changes the rawCfg according to r. It is NOT
-// safe for concurrent use; use rawCfgMu. If the request's
-// method is GET, the config will not be changed.
-func mutateConfig(w http.ResponseWriter, r *http.Request) error {
-	var err error
-	var val interface{}
-
-	// if there is a request body, make sure we recognize its content-type and decode it
-	if r.Method != http.MethodGet && r.Method != http.MethodDelete {
-		if ct := r.Header.Get("Content-Type"); !strings.Contains(ct, "/json") {
-			return fmt.Errorf("unacceptable content-type: %v; 'application/json' required", ct)
-		}
-		err = json.NewDecoder(r.Body).Decode(&val)
-		if err != nil {
-			return fmt.Errorf("decoding request body: %v", err)
-		}
-	}
-
-	buf := new(bytes.Buffer)
-	enc := json.NewEncoder(buf)
-
-	cleanPath := strings.Trim(r.URL.Path, "/")
-	if cleanPath == "" {
-		return fmt.Errorf("no traversable path")
-	}
-
-	parts := strings.Split(cleanPath, "/")
-	if len(parts) == 0 {
-		return fmt.Errorf("path missing")
-	}
-
-	var ptr interface{} = rawCfg
-
-traverseLoop:
-	for i, part := range parts {
-		switch v := ptr.(type) {
-		case map[string]interface{}:
-			// if the next part enters a slice, and the slice is our destination,
-			// handle it specially (because appending to the slice copies the slice
-			// header, which does not replace the original one like we want)
-			if arr, ok := v[part].([]interface{}); ok && i == len(parts)-2 {
-				var idx int
-				if r.Method != http.MethodPost {
-					idxStr := parts[len(parts)-1]
-					idx, err = strconv.Atoi(idxStr)
-					if err != nil {
-						return fmt.Errorf("[%s] invalid array index '%s': %v",
-							r.URL.Path, idxStr, err)
-					}
-					if idx < 0 || idx >= len(arr) {
-						return fmt.Errorf("[%s] array index out of bounds: %s", r.URL.Path, idxStr)
-					}
-				}
-
-				switch r.Method {
-				case http.MethodGet:
-					err = enc.Encode(arr[idx])
-					if err != nil {
-						return fmt.Errorf("encoding config: %v", err)
-					}
-				case http.MethodPost:
-					v[part] = append(arr, val)
-				case http.MethodPut:
-					// avoid creation of new slice and a second copy (see
-					// https://github.com/golang/go/wiki/SliceTricks#insert)
-					arr = append(arr, nil)
-					copy(arr[idx+1:], arr[idx:])
-					arr[idx] = val
-					v[part] = arr
-				case http.MethodPatch:
-					arr[idx] = val
-				case http.MethodDelete:
-					v[part] = append(arr[:idx], arr[idx+1:]...)
-				default:
-					return fmt.Errorf("unrecognized method %s", r.Method)
-				}
-				break traverseLoop
-			}
-
-			if i == len(parts)-1 {
-				switch r.Method {
-				case http.MethodGet:
-					err = enc.Encode(v[part])
-					if err != nil {
-						return fmt.Errorf("encoding config: %v", err)
-					}
-				case http.MethodPost:
-					if arr, ok := v[part].([]interface{}); ok {
-						// if the part is an existing list, POST appends to it
-						// TODO: Do we ever reach this point, since we handle arrays
-						// separately above?
-						v[part] = append(arr, val)
-					} else {
-						// otherwise, it simply sets the value
-						v[part] = val
-					}
-				case http.MethodPut:
-					if _, ok := v[part]; ok {
-						return fmt.Errorf("[%s] key already exists: %s", r.URL.Path, part)
-					}
-					v[part] = val
-				case http.MethodPatch:
-					if _, ok := v[part]; !ok {
-						return fmt.Errorf("[%s] key does not exist: %s", r.URL.Path, part)
-					}
-					v[part] = val
-				case http.MethodDelete:
-					delete(v, part)
-				default:
-					return fmt.Errorf("unrecognized method %s", r.Method)
-				}
-			} else {
-				ptr = v[part]
-			}
-
-		case []interface{}:
-			partInt, err := strconv.Atoi(part)
-			if err != nil {
-				return fmt.Errorf("[/%s] invalid array index '%s': %v",
-					strings.Join(parts[:i+1], "/"), part, err)
-			}
-			if partInt < 0 || partInt >= len(v) {
-				return fmt.Errorf("[/%s] array index out of bounds: %s",
-					strings.Join(parts[:i+1], "/"), part)
-			}
-			ptr = v[partInt]
-
-		default:
-			return fmt.Errorf("invalid path: %s", parts[:i+1])
-		}
-	}
-
-	if r.Method == http.MethodGet {
-		w.Header().Set("Content-Type", "application/json")
-		w.Write(buf.Bytes())
-	}
-
-	return nil
-}
-
-var (
-	// rawCfg is the current, generic-decoded configuration;
-	// we initialize it as a map with one field ("config")
-	// to maintain parity with the API endpoint and to avoid
-	// the special case of having to access/mutate the variable
-	// directly without traversing into it
-	rawCfg = map[string]interface{}{
-		rawConfigKey: nil,
-	}
-	rawCfgJSON  []byte            // keeping the encoded form avoids an extra Marshal on changes
-	rawCfgIndex map[string]string // map of user-assigned ID to expanded path
-	rawCfgMu    sync.Mutex        // protects rawCfg, rawCfgJSON, and rawCfgIndex
-)
-
-const rawConfigKey = "config"

dynamicconfig_test.go

@@ -1,123 +0,0 @@
-// Copyright 2015 Matthew Holt and The Caddy Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package caddy
-
-import (
-	"encoding/json"
-	"net/http"
-	"net/http/httptest"
-	"reflect"
-	"strings"
-	"testing"
-)
-
-func TestMutateConfig(t *testing.T) {
-	// each test is performed in sequence, so
-	// each change builds on the previous ones;
-	// the config is not reset between tests
-	for i, tc := range []struct {
-		method    string
-		path      string // rawConfigKey will be prepended
-		payload   string
-		expect    string // JSON representation of what the whole config is expected to be after the request
-		shouldErr bool
-	}{
-		{
-			method:  "POST",
-			path:    "",
-			payload: `{"foo": "bar", "list": ["a", "b", "c"]}`, // starting value
-			expect:  `{"foo": "bar", "list": ["a", "b", "c"]}`,
-		},
-		{
-			method:  "POST",
-			path:    "/foo",
-			payload: `"jet"`,
-			expect:  `{"foo": "jet", "list": ["a", "b", "c"]}`,
-		},
-		{
-			method:  "POST",
-			path:    "/bar",
-			payload: `{"aa": "bb", "qq": "zz"}`,
-			expect:  `{"foo": "jet", "bar": {"aa": "bb", "qq": "zz"}, "list": ["a", "b", "c"]}`,
-		},
-		{
-			method: "DELETE",
-			path:   "/bar/qq",
-			expect: `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c"]}`,
-		},
-		{
-			method:  "POST",
-			path:    "/list",
-			payload: `"e"`,
-			expect:  `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "e"]}`,
-		},
-		{
-			method:  "PUT",
-			path:    "/list/3",
-			payload: `"d"`,
-			expect:  `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "d", "e"]}`,
-		},
-		{
-			method: "DELETE",
-			path:   "/list/3",
-			expect: `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "e"]}`,
-		},
-		{
-			method:  "PATCH",
-			path:    "/list/3",
-			payload: `"d"`,
-			expect:  `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c", "d"]}`,
-		},
-	} {
-		req, err := http.NewRequest(tc.method, rawConfigKey+tc.path, strings.NewReader(tc.payload))
-		if err != nil {
-			t.Fatalf("Test %d: making test request: %v", i, err)
-		}
-		req.Header.Set("Content-Type", "application/json")
-
-		w := httptest.NewRecorder()
-
-		err = mutateConfig(w, req)
-
-		if tc.shouldErr && err == nil {
-			t.Fatalf("Test %d: Expected error return value, but got: %v", i, err)
-		}
-		if !tc.shouldErr && err != nil {
-			t.Fatalf("Test %d: Should not have had error return value, but got: %v", i, err)
-		}
-
-		if tc.shouldErr && w.Code == http.StatusOK {
-			t.Fatalf("Test %d: Expected error, but got HTTP %d: %s",
-				i, w.Code, w.Body.String())
-		}
-		if !tc.shouldErr && w.Code != http.StatusOK {
-			t.Fatalf("Test %d: Should not have errored, but got HTTP %d: %s",
-				i, w.Code, w.Body.String())
-		}
-
-		// decode the expected config so we can do a convenient DeepEqual
-		var expectedDecoded interface{}
-		err = json.Unmarshal([]byte(tc.expect), &expectedDecoded)
-		if err != nil {
-			t.Fatalf("Test %d: Unmarshaling expected config: %v", i, err)
-		}
-
-		// make sure the resulting config is as we expect it
-		if !reflect.DeepEqual(rawCfg[rawConfigKey], expectedDecoded) {
-			t.Fatalf("Test %d:\nExpected:\n\t%#v\nActual:\n\t%#v",
-				i, expectedDecoded, rawCfg[rawConfigKey])
-		}
-	}
-}