r/RegisterType/RegisterNamespace/g

Co-Author: Matt Holt

.editorconfig

@@ -0,0 +1,5 @@
+[*]
+end_of_line = lf
+
+[caddytest/integration/caddyfile_adapt/*.txt]
+indent_style = tab

.gitattributes

@@ -0,0 +1 @@
+*.go text eol=lf

.github/CONTRIBUTING.md

@@ -0,0 +1,197 @@
+Contributing to Caddy
+=====================
+
+Welcome! Thank you for choosing to be a part of our community. Caddy wouldn't be nearly as excellent without your involvement!
+
+For starters, we invite you to join [the Caddy forum](https://caddy.community) where you can hang out with other Caddy users and developers.
+
+## Common Tasks
+
+- [Contributing code](#contributing-code)
+- [Writing a Caddy module](#writing-a-caddy-module)
+- [Asking or answering questions for help using Caddy](#getting-help-using-caddy)
+- [Reporting a bug](#reporting-bugs)
+- [Suggesting an enhancement or a new feature](#suggesting-features)
+- [Improving documentation](#improving-documentation)
+
+Other menu items:
+
+- [Values](#values)
+- [Coordinated Disclosure](#coordinated-disclosure)
+- [Thank You](#thank-you)
+
+
+### Contributing code
+
+You can have a huge impact on the project by helping with its code. To contribute code to Caddy, first submit or comment in an issue to discuss your contribution, then open a [pull request](https://github.com/caddyserver/caddy/pulls) (PR). If you're new to our community, that's okay: **we gladly welcome pull requests from anyone, regardless of your native language or coding experience.** You can get familiar with Caddy's code base by using [code search at Sourcegraph](https://sourcegraph.com/github.com/caddyserver/caddy).
+
+We hold contributions to a high standard for quality :bowtie:, so don't be surprised if we ask for revisions—even if it seems small or insignificant. Please don't take it personally. :blue_heart: If your change is on the right track, we can guide you to make it mergable.
+
+Here are some of the expectations we have of contributors:
+
+- **Open an issue to propose your change first.** This way we can avoid confusion, coordinate what everyone is working on, and ensure that any changes are in-line with the project's goals and the best interests of its users. We can also discuss the best possible implementation. If there's already an issue about it, comment on the existing issue to claim it. A lot of valuable time can be saved by discussing a proposal first.
+
+- **Keep pull requests small.** Smaller PRs are more likely to be merged because they are easier to review! We might ask you to break up large PRs into smaller ones. [An example of what we want to avoid.](https://twitter.com/iamdevloper/status/397664295875805184)
+
+- **Keep related commits together in a PR.** We do want pull requests to be small, but you should also keep multiple related commits in the same PR if they rely on each other.
+
+- **Write tests.** Good, automated tests are very valuable! Written properly, they ensure your change works, and that other changes in the future won't break your change. CI checks should pass.
+
+- **Benchmarks should be included for optimizations.** Optimizations sometimes make code harder to read or have changes that are less than obvious. They should be proven with benchmarks and profiling.
+
+- **[Squash](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html) insignificant commits.** Every commit should be significant. Commits which merely rewrite a comment or fix a typo can be combined into another commit that has more substance. Interactive rebase can do this, or a simpler way is `git reset --soft <diverging-commit>` then `git commit -s`.
+
+- **Be responsible for and maintain your contributions.** Caddy is a growing project, and it's much better when individual contributors help maintain their change after it is merged.
+
+- **Use comments properly.** We expect good godoc comments for package-level functions, types, and values. Comments are also useful whenever the purpose for a line of code is not obvious.
+
+- **Pull requests may still get closed.** The longer a PR stays open and idle, the more likely it is to be closed. If we haven't reviewed it in a while, it probably means the change is not a priority. Please don't take this personally, we're trying to balance a lot of tasks! If nobody else has commented or reacted to the PR, it likely means your change is useful only to you. The reality is this happens quite a lot. We don't tend to accept PRs that aren't generally helpful. For these reasons or others, the PR may get closed even after a review. We are not obligated to accept all proposed changes, even if the best justification we can give is something vague like, "It doesn't sit right." Sometimes PRs are just the wrong thing or the wrong time. Because it is open source, you can always build your own modified version of Caddy with a change you need, even if we reject it in the official repo. Plus, because Caddy is extensible, it's possible your feature could make a great plugin instead!
+
+- **You certify that you wrote and comprehend the code you submit.** The Caddy project welcomes original contributions that comply with [our CLA](https://cla-assistant.io/caddyserver/caddy), meaning that authors must be able to certify that they created or have rights to the code they are contributing. In addition, we require that code is not simply copy-pasted from Q/A sites or AI language models without full comprehension and rigorous testing. In other words: contributors are allowed to refer to communities for assistance and use AI tools such as language models for inspiration, but code which originates from or is assisted by these resources MUST be:
+
+	- Licensed for you to freely share
+	- Fully comprehended by you (be able to explain every line of code)
+	- Verified by automated tests when feasible, or thorough manual tests otherwise
+
+	We have found that current language models (LLMs, like ChatGPT) may understand code syntax and even problem spaces to an extent, but often fail in subtle ways to convey true knowledge and produce correct algorithms. Integrated tools such as GitHub Copilot and Sourcegraph Cody may be used for inspiration, but code generated by these tools still needs to meet our criteria for licensing, human comprehension, and testing. These tools may be used to help write code comments and tests as long as you can certify they are accurate and correct. Note that it is often more trouble than it's worth to certify that Copilot (for example) is not giving you code that is possibly plagiarised, unlicensed, or licensed with incompatible terms -- as the Caddy project cannot accept such contributions. If that's too difficult for you (or impossible), then we recommend using these resources only for inspiration and write your own code. Ultimately, you (the contributor) are responsible for the code you're submitting.
+
+	As a courtesy to reviewers, we kindly ask that you disclose when contributing code that was generated by an AI tool or copied from another website so we can be aware of what to look for in code review.
+
+We often grant [collaborator status](#collaborator-instructions) to contributors who author one or more significant, high-quality PRs that are merged into the code base.
+
+
+#### HOW TO MAKE A PULL REQUEST TO CADDY
+
+Contributing to Go projects on GitHub is fun and easy. After you have proposed your change in an issue, we recommend the following workflow:
+
+1. [Fork this repo](https://github.com/caddyserver/caddy). This makes a copy of the code you can write to.
+
+2. If you don't already have this repo (caddyserver/caddy.git) repo on your computer, clone it down: `git clone https://github.com/caddyserver/caddy.git`
+
+3. Tell git that it can push the caddyserver/caddy.git repo to your fork by adding a remote: `git remote add myfork https://github.com/<your-username>/caddy.git`
+
+4. Make your changes in the caddyserver/caddy.git repo on your computer.
+
+5. Push your changes to your fork: `git push myfork`
+
+6. [Create a pull request](https://github.com/caddyserver/caddy/pull/new/master) to merge your changes into caddyserver/caddy @ master. (Click "compare across forks" and change the head fork.)
+
+This workflow is nice because you don't have to change import paths. You can get fancier by using different branches if you want.
+
+
+### Writing a Caddy module
+
+Caddy can do more with modules! Anyone can write one. Caddy modules are Go libraries that get compiled into Caddy, extending its feature set. They can add directives to the Caddyfile, add new configuration adapters, and even implement new server types (e.g. HTTP, DNS).
+
+[Learn how to write a module here](https://caddyserver.com/docs/extending-caddy). You should also share and discuss your module idea [on the forums](https://caddy.community) to have people test it out. We don't use the Caddy issue tracker for third-party modules.
+
+
+### Getting help using Caddy
+
+If you have a question about using Caddy, [ask on our forum](https://caddy.community)! There will be more people there who can help you than just the Caddy developers who follow our issue tracker. Issues are not the place for usage questions.
+
+Many people on the forums could benefit from your experience and expertise, too. Once you've been helped, consider giving back by answering other people's questions and participating in other discussions.
+
+
+### Reporting bugs
+
+Like every software, Caddy has its flaws. If you find one, [search the issues](https://github.com/caddyserver/caddy/issues) to see if it has already been reported. If not, [open a new issue](https://github.com/caddyserver/caddy/issues/new) and describe the bug, and somebody will look into it! (This repository is only for Caddy and its standard modules.)
+
+**You can help us fix bugs!** Speed up the patch by identifying the bug in the code. This can sometimes be done by adding `fmt.Println()` statements (or similar) in relevant code paths to narrow down where the problem may be. It's a good way to [introduce yourself to the Go language](https://tour.golang.org), too.
+
+We may reply with an issue template. Please follow the template so we have all the needed information. Unredacted&mdash;yes, actual values matter. We need to be able to repeat the bug using your instructions. Please simplify the issue as much as possible. If you don't, we might close your report. The burden is on you to make it easily reproducible and to convince us that it is actually a bug in Caddy. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you!
+
+We suggest reading [How to Report Bugs Effectively](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html).
+
+Please be kind. :smile: Remember that Caddy comes at no cost to you, and you're getting free support when we fix your issues. If we helped you, please consider helping someone else!
+
+#### Bug reporting expectations
+
+Maintainers---or more generally, developers---need three things to act on bugs:
+
+1. To agree or be convinced that it's a bug (reporter's responsibility).
+	- A bug is unintentional, undesired, or surprising behavior which violates documentation or relevant spec. It might be either a mistake in the documentation or a bug in the code.
+	- This project usually does not work around bugs in other software, systems, and dependencies; instead, we recommend that those bugs are fixed at their source. This sometimes means we close issues or reject PRs that attempt to fix, workaround, or hide bugs in other projects.
+
+2. To be able to understand what is happening (mostly reporter's responsibility).
+	- If the reporter can provide satisfactory instructions such that a developer can reproduce the bug, the developer will likely be able to understand the bug, write a test case, and implement a fix. This is the least amount of work for everyone and path to the fastest resolution.
+	- Otherwise, the burden is on the reporter to test possible solutions. This is less preferable because it loosens the feedback loop, slows down debugging efforts, obscures the true nature of the problem from the developers, and is unlikely to result in new test cases.
+
+3. A solution, or ideas toward a solution (mostly maintainer's responsibility).
+	- Sometimes the best solution is a documentation change.
+	- Usually the developers have the best domain knowledge for inventing a solution, but reporters may have ideas or preferences for how they would like the software to work.
+	- Security, correctness, and project goals/vision all take priority over a user's preferences.
+	- It's simply good business to yield a solution that satisfies the users, and it's even better business to leave them impressed.
+
+Thus, at the very least, the reporter is expected to:
+
+1. Convince the reader that it's a bug in Caddy (if it's not obvious).
+2. Reduce the problem down to the minimum specific steps required to reproduce it.
+
+The maintainer is usually able to do the rest; but of course the reporter may invest additional effort to speed up the process.
+
+
+
+### Suggesting features
+
+First, [search to see if your feature has already been requested](https://github.com/caddyserver/caddy/issues). If it has, you can add a :+1: reaction to vote for it. If your feature idea is new, open an issue to request the feature. Please describe your idea thoroughly so that we know how to implement it! Really vague requests may not be helpful or actionable and, without clarification, will have to be closed.
+
+While we really do value your requests and implement many of them, not all features are a good fit for Caddy. Most of those [make good modules](#writing-a-caddy-module), which can be made by anyone! But if a feature is not in the best interest of the Caddy project or its users in general, we may politely decline to implement it into Caddy core. Additionally, some features are bad ideas altogether (for either obvious or non-obvious reasons) which may be rejected. We'll try to explain why we reject a feature, but sometimes the best we can do is, "It's not a good fit for the project."
+
+
+### Improving documentation
+
+Caddy's documentation is available at [https://caddyserver.com/docs](https://caddyserver.com/docs) and its source is in the [website repo](https://github.com/caddyserver/website). If you would like to make a fix to the docs, please submit an issue there describing the change to make.
+
+Note that third-party module documentation is not hosted by the Caddy website, other than basic usage examples. They are managed by the individual module authors, and you will have to contact them to change their documentation.
+
+Our documentation is scoped to the Caddy project only: it is not for describing how other software or systems work, even if they relate to Caddy or web servers. That kind of content [can be found in our community wiki](https://caddy.community/c/wiki/13), however.
+
+## Collaborator Instructions
+
+Collaborators have push rights to the repository. We grant this permission after one or more successful, high-quality PRs are merged! We thank them for their help. The expectations we have of collaborators are:
+
+- **Help review pull requests.** Be meticulous, but also kind. We love our contributors, but we critique the contribution to make it better. Multiple, thorough reviews make for the best contributions! Here are some questions to consider:
+	- Can the change be made more elegant?
+	- Is this a maintenance burden?
+	- What assumptions does the code make?
+	- Is it well-tested?
+	- Is the change a good fit for the project?
+	- Does it actually fix the problem or is it creating a special case instead?
+	- Does the change incur any new dependencies? (Avoid these!)
+
+- **Answer issues.** If every collaborator helped out with issues, we could count the number of open issues on two hands. This means getting involved in the discussion, investigating the code, and yes, debugging it. It's fun. Really! :smile: Please, please help with open issues. Granted, some issues need to be done before others. And of course some are larger than others: you don't have to do it all yourself. Work with other collaborators as a team!
+
+- **Do not merge pull requests until they have been approved by one or two other collaborators.** If a project owner approves the PR, it can be merged (as long as the conversation has finished too).
+
+- **Prefer squashed commits over a messy merge.** If there are many little commits, please [squash the commits](https://stackoverflow.com/a/11732910/1048862) so we don't clutter the commit history.
+
+- **Don't accept new dependencies lightly.** Dependencies can make the world crash and burn, but they are sometimes necessary. Choose carefully. Extremely small dependencies (a few lines of code) can be inlined. The rest may not be needed. For those that are, Caddy uses [go modules](https://github.com/golang/go/wiki/Modules). All external dependencies must be installed as modules, and _Caddy must not export any types defined by those dependencies_. Check this diligently!
+
+- **Be extra careful in some areas of the code.** There are some critical areas in the Caddy code base that we review extra meticulously: the `caddyhttp` and `caddytls` packages especially.
+
+- **Make sure tests test the actual thing.** Double-check that the tests fail without the change, and pass with it. It's important that they assert what they're purported to assert.
+
+- **Recommended reading**
+	- [CodeReviewComments](https://github.com/golang/go/wiki/CodeReviewComments) for an idea of what we look for in good, clean Go code
+	- [Linus Torvalds describes a good commit message](https://gist.github.com/matthewhudson/1475276)
+	- [Best Practices for Maintainers](https://opensource.guide/best-practices/)
+	- [Shrinking Code Review](https://alexgaynor.net/2015/dec/29/shrinking-code-review/)
+
+
+
+## Values (WIP)
+
+- A person is always more important than code. People don't like being handled "efficiently". But we can still process issues and pull requests efficiently while being kind, patient, and considerate.
+
+- The ends justify the means, if the means are good. A good tree won't produce bad fruit. But if we cut corners or are hasty in our process, the end result will not be good.
+
+
+## Security Policy
+
+If you think you've found a security vulnerability, please refer to our [Security Policy](https://github.com/caddyserver/caddy/security/policy) document.
+
+
+## Thank you
+
+Thanks for your help! Caddy would not be what it is today without your contributions.

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: [mholt] # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.github/SECURITY.md

@@ -0,0 +1,59 @@
+# Security Policy
+
+The Caddy project would like to make sure that it stays on top of all practically-exploitable vulnerabilities.
+
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x     | ✔️ |
+| 1.x     | :x:                |
+| < 1.x   | :x:                |
+
+
+## Acceptable Scope
+
+A security report must demonstrate a security bug in the source code from this repository.
+
+Some security problems are the result of interplay between different components of the Web, rather than a vulnerability in the web server itself. Please only report vulnerabilities in the web server itself, as we cannot coerce the rest of the Web to be fixed (for example, we do not consider IP spoofing, BGP hijacks, or missing/misconfigured HTTP headers a vulnerability in the Caddy web server).
+
+Vulnerabilities caused by misconfigurations are out of scope. Yes, it is entirely possible to craft and use a configuration that is unsafe, just like with every other web server; we recommend against doing that.
+
+We do not accept reports if the steps imply or require a compromised system or third-party software, as we cannot control those. We expect that users secure their own systems and keep all their software patched. For example, if untrusted users are able to upload/write/host arbitrary files in the web root directory, it is NOT a security bug in Caddy if those files get served to clients; however, it _would_ be a valid report if a bug in Caddy's source code unintentionally gave unauthorized users the ability to upload unsafe files or delete files without relying on an unpatched system or piece of software.
+
+Client-side exploits are out of scope. In other words, it is not a bug in Caddy if the web browser does something unsafe, even if the downloaded content was served by Caddy. (Those kinds of exploits can generally be mitigated by proper configuration of HTTP headers.) As a general rule, the content served by Caddy is not considered in scope because content is configurable by the site owner or the associated web application.
+
+Security bugs in code dependencies (including Go's standard library) are out of scope. Instead, if a dependency has patched a relevant security bug, please feel free to open a public issue or pull request to update that dependency in our code.
+
+
+## Reporting a Vulnerability
+
+We get a lot of difficult reports that turn out to be invalid. Clear, obvious reports tend to be the most credible (but are also rare).
+
+First please ensure your report falls within the accepted scope of security bugs (above).
+
+We'll need enough information to verify the bug and make a patch. To speed things up, please include:
+
+- Most minimal possible config (without redactions!)
+- Command(s)
+- Precise HTTP requests (`curl -v` and its output please)
+- Full log output (please enable debug mode)
+- Specific minimal steps to reproduce the issue from scratch
+- A working patch
+
+Please DO NOT use containers, VMs, cloud instances or services, or any other complex infrastructure in your steps. Always prefer `curl -v` instead of web browsers.
+
+We consider publicly-registered domain names to be public information. This necessary in order to maintain the integrity of certificate transparency, public DNS, and other public trust systems. Do not redact domain names from your reports. The actual content of your domain name affects Caddy's behavior, so we need the exact domain name(s) to reproduce with, or your report will be ignored.
+
+It will speed things up if you suggest a working patch, such as a code diff, and explain why and how it works. Reports that are not actionable, do not contain enough information, are too pushy/demanding, or are not able to convince us that it is a viable and practical attack on the web server itself may be deferred to a later time or possibly ignored, depending on available resources. Priority will be given to credible, responsible reports that are constructive, specific, and actionable. (We get a lot of invalid reports.) Thank you for understanding.
+
+When you are ready, please email Matt Holt (the author) directly: matt at dyanim dot com.
+
+Please don't encrypt the email body. It only makes the process more complicated.
+
+Please also understand that due to our nature as an open source project, we do not have a budget to award security bounties. We can only thank you.
+
+If your report is valid and a patch is released, we will not reveal your identity by default. If you wish to be credited, please give us the name to use and/or your GitHub username. If you don't provide this we can't credit you.
+
+Thanks for responsibly helping Caddy&mdash;and thousands of websites&mdash;be more secure!

.github/dependabot.yml

@@ -0,0 +1,7 @@
+---
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "monthly"

.github/workflows/ci.yml

@@ -0,0 +1,172 @@
+# Used as inspiration: https://github.com/mvdan/github-actions-golang
+
+name: Tests
+
+on:
+  push:
+    branches:
+      - master
+      - 2.*
+  pull_request:
+    branches:
+      - master
+      - 2.*
+
+jobs:
+  test:
+    strategy:
+      # Default is true, cancels jobs for other platforms in the matrix if one fails
+      fail-fast: false
+      matrix:
+        os: 
+          - ubuntu-latest
+          - macos-latest
+          - windows-latest
+        go: 
+          - '1.20'
+          - '1.21'
+
+        include:
+        # Set the minimum Go patch version for the given Go minor
+        # Usable via ${{ matrix.GO_SEMVER }}
+        - go: '1.20'
+          GO_SEMVER: '~1.20.6'
+
+        - go: '1.21'
+          GO_SEMVER: '~1.21.0'
+
+        # Set some variables per OS, usable via ${{ matrix.VAR }}
+        # CADDY_BIN_PATH: the path to the compiled Caddy binary, for artifact publishing
+        # SUCCESS: the typical value for $? per OS (Windows/pwsh returns 'True')
+        - os: ubuntu-latest
+          CADDY_BIN_PATH: ./cmd/caddy/caddy
+          SUCCESS: 0
+
+        - os: macos-latest
+          CADDY_BIN_PATH: ./cmd/caddy/caddy
+          SUCCESS: 0
+
+        - os: windows-latest
+          CADDY_BIN_PATH: ./cmd/caddy/caddy.exe
+          SUCCESS: 'True'
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Install Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: ${{ matrix.GO_SEMVER }}
+        check-latest: true
+
+    # These tools would be useful if we later decide to reinvestigate
+    # publishing test/coverage reports to some tool for easier consumption
+    # - name: Install test and coverage analysis tools
+    #   run: |
+    #     go get github.com/axw/gocov/gocov
+    #     go get github.com/AlekSi/gocov-xml
+    #     go get -u github.com/jstemmer/go-junit-report
+    #     echo "$(go env GOPATH)/bin" >> $GITHUB_PATH
+
+    - name: Print Go version and environment
+      id: vars
+      shell: bash
+      run: |
+        printf "Using go at: $(which go)\n"
+        printf "Go version: $(go version)\n"
+        printf "\n\nGo environment:\n\n"
+        go env
+        printf "\n\nSystem environment:\n\n"
+        env
+        printf "Git version: $(git version)\n\n"
+        # Calculate the short SHA1 hash of the git commit
+        echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+
+    - name: Get dependencies
+      run: |
+        go get -v -t -d ./...
+        # mkdir test-results
+
+    - name: Build Caddy
+      working-directory: ./cmd/caddy
+      env:
+        CGO_ENABLED: 0
+      run: |
+        go build -trimpath -ldflags="-w -s" -v
+
+    - name: Publish Build Artifact
+      uses: actions/upload-artifact@v3
+      with:
+        name: caddy_${{ runner.os }}_go${{ matrix.go }}_${{ steps.vars.outputs.short_sha }}
+        path: ${{ matrix.CADDY_BIN_PATH }}
+
+    # Commented bits below were useful to allow the job to continue
+    # even if the tests fail, so we can publish the report separately
+    # For info about set-output, see https://stackoverflow.com/questions/57850553/github-actions-check-steps-status
+    - name: Run tests
+      # id: step_test
+      # continue-on-error: true
+      run: |
+        # (go test -v -coverprofile=cover-profile.out -race ./... 2>&1) > test-results/test-result.out
+        go test -v -coverprofile="cover-profile.out" -short -race ./...
+        # echo "status=$?" >> $GITHUB_OUTPUT
+
+    # Relevant step if we reinvestigate publishing test/coverage reports
+    # - name: Prepare coverage reports
+    #   run: |
+    #     mkdir coverage
+    #     gocov convert cover-profile.out > coverage/coverage.json
+    #     # Because Windows doesn't work with input redirection like *nix, but output redirection works.
+    #     (cat ./coverage/coverage.json | gocov-xml) > coverage/coverage.xml
+
+    # To return the correct result even though we set 'continue-on-error: true'
+    # - name: Coerce correct build result
+    #   if: matrix.os != 'windows-latest' && steps.step_test.outputs.status != ${{ matrix.SUCCESS }}
+    #   run: |
+    #     echo "step_test ${{ steps.step_test.outputs.status }}\n"
+    #     exit 1
+
+  s390x-test:
+    name: test (s390x on IBM Z)
+    runs-on: ubuntu-latest
+    if: github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]'
+    continue-on-error: true  # August 2020: s390x VM is down due to weather and power issues
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      - name: Run Tests
+        run: |
+          mkdir -p ~/.ssh && echo -e "${SSH_KEY//_/\\n}" > ~/.ssh/id_ecdsa && chmod og-rwx ~/.ssh/id_ecdsa
+
+          # short sha is enough?
+          short_sha=$(git rev-parse --short HEAD)
+
+          # The environment is fresh, so there's no point in keeping accepting and adding the key.
+          rsync -arz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress --delete --exclude '.git' . "$CI_USER"@ci-s390x.caddyserver.com:/var/tmp/"$short_sha"
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -t "$CI_USER"@ci-s390x.caddyserver.com "cd /var/tmp/$short_sha; go version; go env; printf "\n\n";CGO_ENABLED=0 go test -v ./..."
+          test_result=$?
+
+          # There's no need leaving the files around
+          ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$CI_USER"@ci-s390x.caddyserver.com "rm -rf /var/tmp/'$short_sha'"
+
+          echo "Test exit code: $test_result"
+          exit $test_result
+        env:
+          SSH_KEY: ${{ secrets.S390X_SSH_KEY }}
+          CI_USER: ${{ secrets.CI_USER }}
+
+  goreleaser-check:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      
+      - uses: goreleaser/goreleaser-action@v5
+        with:
+          version: latest
+          args: check
+        env:
+          TAG: ${{ steps.vars.outputs.version_tag }}

.github/workflows/cross-build.yml

@@ -0,0 +1,75 @@
+name: Cross-Build
+
+on:
+  push:
+    branches:
+      - master
+      - 2.*
+  pull_request:
+    branches:
+      - master
+      - 2.*
+
+jobs:
+  cross-build-test:
+    strategy:
+      fail-fast: false
+      matrix:
+        goos: 
+          - 'aix'
+          - 'android'
+          - 'linux'
+          - 'solaris'
+          - 'illumos'
+          - 'dragonfly'
+          - 'freebsd'
+          - 'openbsd'
+          - 'plan9'
+          - 'windows'
+          - 'darwin'
+          - 'netbsd'
+        go: 
+          - '1.21'
+
+        include:
+        # Set the minimum Go patch version for the given Go minor
+        # Usable via ${{ matrix.GO_SEMVER }}
+        - go: '1.21'
+          GO_SEMVER: '~1.21.0'
+
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: ${{ matrix.GO_SEMVER }}
+          check-latest: true
+
+      - name: Print Go version and environment
+        id: vars
+        run: |
+          printf "Using go at: $(which go)\n"
+          printf "Go version: $(go version)\n"
+          printf "\n\nGo environment:\n\n"
+          go env
+          printf "\n\nSystem environment:\n\n"
+          env
+
+      - name: Run Build
+        env:
+          CGO_ENABLED: 0
+          GOOS: ${{ matrix.goos }}
+          GOARCH: ${{ matrix.goos == 'aix' && 'ppc64' || 'amd64' }}
+        shell: bash
+        continue-on-error: true
+        working-directory: ./cmd/caddy
+        run: |
+          GOOS=$GOOS GOARCH=$GOARCH go build -trimpath -o caddy-"$GOOS"-$GOARCH 2> /dev/null
+          if [ $? -ne 0 ]; then
+            echo "::warning ::$GOOS Build Failed"
+            exit 0
+          fi

.github/workflows/lint.yml

@@ -0,0 +1,61 @@
+name: Lint
+
+on:
+  push:
+    branches:
+      - master
+      - 2.*
+  pull_request:
+    branches:
+      - master
+      - 2.*
+
+permissions:
+  contents: read
+
+jobs:
+  # From https://github.com/golangci/golangci-lint-action
+  golangci:
+    permissions:
+      contents: read # for actions/checkout to fetch code
+      pull-requests: read # for golangci/golangci-lint-action to fetch pull requests
+    name: lint
+    strategy:
+      matrix:
+        os:
+          - ubuntu-latest
+          - macos-latest
+          - windows-latest
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v4
+        with:
+          go-version: '~1.21.0'
+          check-latest: true
+
+          # Workaround for https://github.com/golangci/golangci-lint-action/issues/135
+          skip-pkg-cache: true
+
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@v3
+        with:
+          version: v1.54
+
+          # Workaround for https://github.com/golangci/golangci-lint-action/issues/135
+          skip-pkg-cache: true
+
+          # Windows times out frequently after about 5m50s if we don't set a longer timeout.
+          args: --timeout 10m
+
+          # Optional: show only new issues if it's a pull request. The default value is `false`.
+          # only-new-issues: true
+
+  govulncheck:
+    runs-on: ubuntu-latest
+    steps:
+      - name: govulncheck
+        uses: golang/govulncheck-action@v1
+        with:
+          go-version-input: '~1.21.0'
+          check-latest: true

.github/workflows/release.yml

@@ -0,0 +1,174 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - 'v*.*.*'
+
+jobs:
+  release:
+    name: Release
+    strategy:
+      matrix:
+        os: 
+          - ubuntu-latest
+        go: 
+          - '1.21'
+
+        include:
+        # Set the minimum Go patch version for the given Go minor
+        # Usable via ${{ matrix.GO_SEMVER }}
+        - go: '1.21'
+          GO_SEMVER: '~1.21.0'
+
+    runs-on: ${{ matrix.os }}
+    # https://github.com/sigstore/cosign/issues/1258#issuecomment-1002251233
+    # https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#adding-permissions-settings
+    permissions:
+      id-token: write
+      # https://docs.github.com/en/rest/overview/permissions-required-for-github-apps#permission-on-contents
+      # "Releases" is part of `contents`, so it needs the `write`
+      contents: write
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+      with:
+        fetch-depth: 0
+
+    - name: Install Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: ${{ matrix.GO_SEMVER }}
+        check-latest: true
+
+    # Force fetch upstream tags -- because 65 minutes
+    # tl;dr: actions/checkout@v4 runs this line:
+    #   git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +ebc278ec98bb24f2852b61fde2a9bf2e3d83818b:refs/tags/
+    # which makes its own local lightweight tag, losing all the annotations in the process. Our earlier script ran:
+    #   git fetch --prune --unshallow
+    # which doesn't overwrite that tag because that would be destructive.
+    # Credit to @francislavoie for the investigation.
+    # https://github.com/actions/checkout/issues/290#issuecomment-680260080
+    - name: Force fetch upstream tags
+      run: git fetch --tags --force
+
+    # https://github.community/t5/GitHub-Actions/How-to-get-just-the-tag-name/m-p/32167/highlight/true#M1027
+    - name: Print Go version and environment
+      id: vars
+      run: |
+        printf "Using go at: $(which go)\n"
+        printf "Go version: $(go version)\n"
+        printf "\n\nGo environment:\n\n"
+        go env
+        printf "\n\nSystem environment:\n\n"
+        env
+        echo "version_tag=${GITHUB_REF/refs\/tags\//}" >> $GITHUB_OUTPUT
+        echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+
+        # Add "pip install" CLI tools to PATH
+        echo ~/.local/bin >> $GITHUB_PATH
+
+        # Parse semver
+        TAG=${GITHUB_REF/refs\/tags\//}
+        SEMVER_RE='[^0-9]*\([0-9]*\)[.]\([0-9]*\)[.]\([0-9]*\)\([0-9A-Za-z\.-]*\)'
+        TAG_MAJOR=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\1#"`
+        TAG_MINOR=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\2#"`
+        TAG_PATCH=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\3#"`
+        TAG_SPECIAL=`echo ${TAG#v} | sed -e "s#$SEMVER_RE#\4#"`
+        echo "tag_major=${TAG_MAJOR}" >> $GITHUB_OUTPUT
+        echo "tag_minor=${TAG_MINOR}" >> $GITHUB_OUTPUT
+        echo "tag_patch=${TAG_PATCH}" >> $GITHUB_OUTPUT
+        echo "tag_special=${TAG_SPECIAL}" >> $GITHUB_OUTPUT
+
+    # Cloudsmith CLI tooling for pushing releases
+    # See https://help.cloudsmith.io/docs/cli
+    - name: Install Cloudsmith CLI
+      run: pip install --upgrade cloudsmith-cli
+
+    - name: Validate commits and tag signatures
+      run: |
+        
+        # Import Matt Holt's key
+        curl 'https://github.com/mholt.gpg' | gpg --import
+
+        echo "Verifying the tag: ${{ steps.vars.outputs.version_tag }}"
+        # tags are only accepted if signed by Matt's key
+        git verify-tag "${{ steps.vars.outputs.version_tag }}" || exit 1
+
+    - name: Install Cosign
+      uses: sigstore/cosign-installer@main
+    - name: Cosign version
+      run: cosign version
+    - name: Install Syft
+      uses: anchore/sbom-action/download-syft@main
+    - name: Syft version
+      run: syft version
+    # GoReleaser will take care of publishing those artifacts into the release
+    - name: Run GoReleaser
+      uses: goreleaser/goreleaser-action@v5
+      with:
+        version: latest
+        args: release --clean --timeout 60m
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        TAG: ${{ steps.vars.outputs.version_tag }}
+        COSIGN_EXPERIMENTAL: 1
+
+    # Only publish on non-special tags (e.g. non-beta)
+    # We will continue to push to Gemfury for the foreseeable future, although
+    # Cloudsmith is probably better, to not break things for existing users of Gemfury.
+    # See https://gemfury.com/caddy/deb:caddy
+    - name: Publish .deb to Gemfury
+      if: ${{ steps.vars.outputs.tag_special == '' }}
+      env:
+        GEMFURY_PUSH_TOKEN: ${{ secrets.GEMFURY_PUSH_TOKEN }}
+      run: |
+        for filename in dist/*.deb; do
+          # armv6 and armv7 are both "armhf" so we can skip the duplicate
+          if [[ "$filename" == *"armv6"* ]]; then
+            echo "Skipping $filename"
+            continue
+          fi
+
+          curl -F package=@"$filename" https://${GEMFURY_PUSH_TOKEN}:@push.fury.io/caddy/
+        done
+
+    # Publish only special tags (unstable/beta/rc) to the "testing" repo
+    # See https://cloudsmith.io/~caddy/repos/testing/
+    - name: Publish .deb to Cloudsmith (special tags)
+      if: ${{ steps.vars.outputs.tag_special != '' }}
+      env:
+        CLOUDSMITH_API_KEY: ${{ secrets.CLOUDSMITH_API_KEY }}
+      run: |
+        for filename in dist/*.deb; do
+          # armv6 and armv7 are both "armhf" so we can skip the duplicate
+          if [[ "$filename" == *"armv6"* ]]; then
+            echo "Skipping $filename"
+            continue
+          fi
+
+          echo "Pushing $filename to 'testing'"
+          cloudsmith push deb caddy/testing/any-distro/any-version $filename
+        done
+
+    # Publish stable tags to Cloudsmith to both repos, "stable" and "testing"
+    # See https://cloudsmith.io/~caddy/repos/stable/
+    - name: Publish .deb to Cloudsmith (stable tags)
+      if: ${{ steps.vars.outputs.tag_special == '' }}
+      env:
+        CLOUDSMITH_API_KEY: ${{ secrets.CLOUDSMITH_API_KEY }}
+      run: |
+        for filename in dist/*.deb; do
+          # armv6 and armv7 are both "armhf" so we can skip the duplicate
+          if [[ "$filename" == *"armv6"* ]]; then
+            echo "Skipping $filename"
+            continue
+          fi
+
+          echo "Pushing $filename to 'stable'"
+          cloudsmith push deb caddy/stable/any-distro/any-version $filename
+
+          echo "Pushing $filename to 'testing'"
+          cloudsmith push deb caddy/testing/any-distro/any-version $filename
+        done

.github/workflows/release_published.yml

@@ -0,0 +1,35 @@
+name: Release Published
+
+# Event payload: https://developer.github.com/webhooks/event-payloads/#release
+on:
+  release:
+    types: [published]
+
+jobs:
+  release:
+    name: Release Published
+    strategy:
+      matrix:
+        os: 
+          - ubuntu-latest
+    runs-on: ${{ matrix.os }}
+
+    steps:
+
+    # See https://github.com/peter-evans/repository-dispatch
+    - name: Trigger event on caddyserver/dist
+      uses: peter-evans/repository-dispatch@v2
+      with:
+        token: ${{ secrets.REPO_DISPATCH_TOKEN }}
+        repository: caddyserver/dist
+        event-type: release-tagged
+        client-payload: '{"tag": "${{ github.event.release.tag_name }}"}'
+
+    - name: Trigger event on caddyserver/caddy-docker
+      uses: peter-evans/repository-dispatch@v2
+      with:
+        token: ${{ secrets.REPO_DISPATCH_TOKEN }}
+        repository: caddyserver/caddy-docker
+        event-type: release-tagged
+        client-payload: '{"tag": "${{ github.event.release.tag_name }}"}'
+

.gitignore

@@ -1,18 +1,30 @@
 _gitignore/
 *.log
 Caddyfile
+Caddyfile.*
 !caddyfile/
 
 # artifacts from pprof tooling
 *.prof
 *.test
 
-# build artifacts
+# build artifacts and helpers
 cmd/caddy/caddy
 cmd/caddy/caddy.exe
+cmd/caddy/tmp/*.exe
+cmd/caddy/.env
 
 # mac specific
 .DS_Store
 
 # go modules
 vendor
+
+# goreleaser artifacts
+dist
+caddy-build
+caddy-dist
+
+# IDE files
+.idea/
+.vscode/

.golangci.yml

@@ -1,19 +1,78 @@
 linters-settings:
   errcheck:
-    ignore: fmt:.*,io/ioutil:^Read.*,github.com/caddyserver/caddy/v2/caddyconfig:RegisterAdapter,github.com/caddyserver/caddy/v2:RegisterModule
+    ignore: fmt:.*,go.uber.org/zap/zapcore:^Add.*
     ignoretests: true
-  misspell:
+  gci:
-    locale: US
+    sections:
+      - standard # Standard section: captures all standard packages.
+      - default # Default section: contains all imports that could not be matched to another section type.
+      - prefix(github.com/caddyserver/caddy/v2/cmd) # ensure that this is always at the top and always has a line break.
+      - prefix(github.com/caddyserver/caddy) # Custom section: groups all imports with the specified Prefix.
+    # Skip generated files.
+    # Default: true
+    skip-generated: true
+    # Enable custom order of sections.
+    # If `true`, make the section order the same as the order of `sections`.
+    # Default: false
+    custom-order: true
 
 linters:
+  disable-all: true
   enable:
     - bodyclose
     - errcheck
-    - gofmt
+    - gci
-    - goimports
+    - gofumpt
     - gosec
+    - gosimple
+    - govet
     - ineffassign
     - misspell
+    - prealloc
+    - staticcheck
+    - typecheck
+    - unconvert
+    - unused
+  # these are implicitly disabled:
+  # - asciicheck
+  # - depguard
+  # - dogsled
+  # - dupl
+  # - exhaustive
+  # - exportloopref
+  # - funlen
+  # - gci
+  # - gochecknoglobals
+  # - gochecknoinits
+  # - gocognit
+  # - goconst
+  # - gocritic
+  # - gocyclo
+  # - godot
+  # - godox
+  # - goerr113
+  # - gofumpt
+  # - goheader
+  # - golint
+  # - gomnd
+  # - gomodguard
+  # - goprintffuncname
+  # - interfacer
+  # - lll
+  # - maligned
+  # - nakedret
+  # - nestif
+  # - nlreturn
+  # - noctx
+  # - nolintlint
+  # - rowserrcheck
+  # - scopelint
+  # - sqlclosecheck
+  # - stylecheck
+  # - testpackage
+  # - unparam
+  # - whitespace
+  # - wsl
 
 run:
   # default concurrency is a available CPU number.
@@ -31,19 +90,23 @@ output:
 issues:
   exclude-rules:
     # we aren't calling unknown URL
-    - text: "G107" # G107: Url provided to HTTP request as taint input
+    - text: 'G107' # G107: Url provided to HTTP request as taint input
       linters:
         - gosec
     # as a web server that's expected to handle any template, this is totally in the hands of the user.
-    - text: "G203" # G203: Use of unescaped data in HTML templates
+    - text: 'G203' # G203: Use of unescaped data in HTML templates
       linters:
         - gosec
     # we're shelling out to known commands, not relying on user-defined input.
-    - text: "G204" # G204: Audit use of command execution
+    - text: 'G204' # G204: Audit use of command execution
       linters:
         - gosec
     # the choice of weakrand is deliberate, hence the named import "weakrand"
     - path: modules/caddyhttp/reverseproxy/selectionpolicies.go
-      text: "G404" # G404: Insecure random number source (rand)
+      text: 'G404' # G404: Insecure random number source (rand)
+      linters:
+        - gosec
+    - path: modules/caddyhttp/reverseproxy/streaming.go
+      text: 'G404' # G404: Insecure random number source (rand)
       linters:
         - gosec

.goreleaser.yml

@@ -0,0 +1,203 @@
+before:
+  hooks:
+    # The build is done in this particular way to build Caddy in a designated directory named in .gitignore.
+    # This is so we can run goreleaser on tag without Git complaining of being dirty. The main.go in cmd/caddy directory 
+    # cannot be built within that directory due to changes necessary for the build causing Git to be dirty, which
+    # subsequently causes gorleaser to refuse running.
+    - rm -rf caddy-build caddy-dist vendor
+    # vendor Caddy deps
+    - go mod vendor
+    - mkdir -p caddy-build
+    - cp cmd/caddy/main.go caddy-build/main.go
+    - /bin/sh -c 'cd ./caddy-build && go mod init caddy'
+    # GoReleaser doesn't seem to offer {{.Tag}} at this stage, so we have to embed it into the env
+    # so we run: TAG=$(git describe --abbrev=0) goreleaser release --rm-dist --skip-publish --skip-validate
+    - go mod edit -require=github.com/caddyserver/caddy/v2@{{.Env.TAG}} ./caddy-build/go.mod
+    # as of Go 1.16, `go` commands no longer automatically change go.{mod,sum}. We now have to explicitly
+    # run `go mod tidy`. The `/bin/sh -c '...'` is because goreleaser can't find cd in PATH without shell invocation.
+    - /bin/sh -c 'cd ./caddy-build && go mod tidy'
+    # vendor the deps of the prepared to-build module
+    - /bin/sh -c 'cd ./caddy-build && go mod vendor'
+    - git clone --depth 1 https://github.com/caddyserver/dist caddy-dist
+    - mkdir -p caddy-dist/man
+    - go mod download
+    - go run cmd/caddy/main.go manpage --directory ./caddy-dist/man
+    - gzip -r ./caddy-dist/man/
+    - /bin/sh -c 'go run cmd/caddy/main.go completion bash > ./caddy-dist/scripts/bash-completion'
+
+builds:
+- env:
+  - CGO_ENABLED=0
+  - GO111MODULE=on
+  main: main.go
+  dir: ./caddy-build
+  binary: caddy
+  goos:
+  - darwin
+  - linux
+  - windows
+  - freebsd
+  goarch:
+  - amd64
+  - arm
+  - arm64
+  - s390x
+  - ppc64le
+  - riscv64
+  goarm:
+  - "5"
+  - "6"
+  - "7"
+  ignore:
+    - goos: darwin
+      goarch: arm
+    - goos: darwin
+      goarch: ppc64le
+    - goos: darwin
+      goarch: s390x
+    - goos: darwin
+      goarch: riscv64
+    - goos: windows
+      goarch: ppc64le
+    - goos: windows
+      goarch: s390x
+    - goos: windows
+      goarch: riscv64
+    - goos: freebsd
+      goarch: ppc64le
+    - goos: freebsd
+      goarch: s390x
+    - goos: freebsd
+      goarch: riscv64
+    - goos: freebsd
+      goarch: arm
+      goarm: "5"
+  flags:
+  - -trimpath
+  - -mod=readonly
+  ldflags:
+  - -s -w
+
+signs:
+  - cmd: cosign
+    signature: "${artifact}.sig"
+    certificate: '{{ trimsuffix (trimsuffix .Env.artifact ".zip") ".tar.gz" }}.pem'
+    args: ["sign-blob", "--yes", "--output-signature=${signature}", "--output-certificate", "${certificate}", "${artifact}"]
+    artifacts: all
+
+sboms:
+  - artifacts: binary
+    documents:
+      - >-
+        {{ .ProjectName }}_
+        {{- .Version }}_
+        {{- if eq .Os "darwin" }}mac{{ else }}{{ .Os }}{{ end }}_
+        {{- .Arch }}
+        {{- with .Arm }}v{{ . }}{{ end }}
+        {{- with .Mips }}_{{ . }}{{ end }}
+        {{- if not (eq .Amd64 "v1") }}{{ .Amd64 }}{{ end }}.sbom
+    cmd: syft
+    args: ["$artifact", "--file", "${document}", "--output", "cyclonedx-json"]
+
+archives:
+  - id: default
+    format_overrides:
+      - goos: windows
+        format: zip
+    name_template: >-
+      {{ .ProjectName }}_
+      {{- .Version }}_
+      {{- if eq .Os "darwin" }}mac{{ else }}{{ .Os }}{{ end }}_
+      {{- .Arch }}
+      {{- with .Arm }}v{{ . }}{{ end }}
+      {{- with .Mips }}_{{ . }}{{ end }}
+      {{- if not (eq .Amd64 "v1") }}{{ .Amd64 }}{{ end }}
+  
+  # package the 'caddy-build' directory into a tarball,
+  # allowing users to build the exact same set of files as ours.
+  - id: source
+    meta: true
+    name_template: "{{ .ProjectName }}_{{ .Version }}_buildable-artifact"
+    files:
+      - src: LICENSE
+        dst: ./LICENSE
+      - src: README.md
+        dst: ./README.md
+      - src: AUTHORS
+        dst: ./AUTHORS
+      - src: ./caddy-build
+        dst: ./
+
+source:
+  enabled: true
+  name_template: '{{ .ProjectName }}_{{ .Version }}_src'
+  format: 'tar.gz'
+
+  # Additional files/template/globs you want to add to the source archive.
+  #
+  # Default: empty.
+  files:
+    - vendor
+
+
+checksum:
+  algorithm: sha512
+
+nfpms:
+  - id: default
+    package_name: caddy
+
+    vendor: Dyanim
+    homepage: https://caddyserver.com
+    maintainer: Matthew Holt <mholt@users.noreply.github.com>
+    description: |
+      Caddy - Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go
+    license: Apache 2.0
+
+    formats:
+      - deb
+      # - rpm
+
+    bindir: /usr/bin
+    contents:
+      - src: ./caddy-dist/init/caddy.service
+        dst: /lib/systemd/system/caddy.service
+        
+      - src: ./caddy-dist/init/caddy-api.service
+        dst: /lib/systemd/system/caddy-api.service
+      
+      - src: ./caddy-dist/welcome/index.html
+        dst: /usr/share/caddy/index.html
+      
+      - src: ./caddy-dist/scripts/bash-completion
+        dst: /etc/bash_completion.d/caddy
+    
+      - src: ./caddy-dist/config/Caddyfile
+        dst: /etc/caddy/Caddyfile
+        type: config
+
+      - src: ./caddy-dist/man/*
+        dst: /usr/share/man/man8/
+
+    scripts:
+      postinstall: ./caddy-dist/scripts/postinstall.sh
+      preremove: ./caddy-dist/scripts/preremove.sh
+      postremove: ./caddy-dist/scripts/postremove.sh
+
+release:
+  github:
+    owner: caddyserver
+    name: caddy
+  draft: true
+  prerelease: auto
+
+changelog:
+  sort: asc
+  filters:
+    exclude:
+    - '^chore:'
+    - '^ci:'
+    - '^docs?:'
+    - '^readme:'
+    - '^tests?:'
+    - '^\w+\s+' # a hack to remove commit messages without colons thus don't correspond to a package

README.md

@@ -1,384 +1,200 @@
-Caddy 2 Development Branch
+<p align="center">
-===========================
+	<a href="https://caddyserver.com">
+		<picture>
+			<source media="(prefers-color-scheme: dark)" srcset="https://user-images.githubusercontent.com/1128849/210187358-e2c39003-9a5e-4dd5-a783-6deb6483ee72.svg">
+			<source media="(prefers-color-scheme: light)" srcset="https://user-images.githubusercontent.com/1128849/210187356-dfb7f1c5-ac2e-43aa-bb23-fc014280ae1f.svg">
+			<img src="https://user-images.githubusercontent.com/1128849/210187356-dfb7f1c5-ac2e-43aa-bb23-fc014280ae1f.svg" alt="Caddy" width="550">
+		</picture>
+	</a>
+	<br>
+	<h3 align="center">a <a href="https://zerossl.com"><img src="https://user-images.githubusercontent.com/55066419/208327323-2770dc16-ec09-43a0-9035-c5b872c2ad7f.svg" height="28" style="vertical-align: -7.7px" valign="middle"></a> project</h3>
+</p>
+<hr>
+<h3 align="center">Every site on HTTPS</h3>
+<p align="center">Caddy is an extensible server platform that uses TLS by default.</p>
+<p align="center">
+	<a href="https://github.com/caddyserver/caddy/actions/workflows/ci.yml"><img src="https://github.com/caddyserver/caddy/actions/workflows/ci.yml/badge.svg"></a>
+	<a href="https://pkg.go.dev/github.com/caddyserver/caddy/v2"><img src="https://img.shields.io/badge/godoc-reference-%23007d9c.svg"></a>
+	<br>
+	<a href="https://twitter.com/caddyserver" title="@caddyserver on Twitter"><img src="https://img.shields.io/badge/twitter-@caddyserver-55acee.svg" alt="@caddyserver on Twitter"></a>
+	<a href="https://caddy.community" title="Caddy Forum"><img src="https://img.shields.io/badge/community-forum-ff69b4.svg" alt="Caddy Forum"></a>
+	<br>
+	<a href="https://sourcegraph.com/github.com/caddyserver/caddy?badge" title="Caddy on Sourcegraph"><img src="https://sourcegraph.com/github.com/caddyserver/caddy/-/badge.svg" alt="Caddy on Sourcegraph"></a>
+	<a href="https://cloudsmith.io/~caddy/repos/"><img src="https://img.shields.io/badge/OSS%20hosting%20by-cloudsmith-blue?logo=cloudsmith" alt="Cloudsmith"></a>
+</p>
+<p align="center">
+	<a href="https://github.com/caddyserver/caddy/releases">Releases</a> ·
+	<a href="https://caddyserver.com/docs/">Documentation</a> ·
+	<a href="https://caddy.community">Get Help</a>
+</p>
 
-[![Build Status](https://dev.azure.com/mholt-dev/Caddy/_apis/build/status/Multiplatform%20Tests?branchName=v2)](https://dev.azure.com/mholt-dev/Caddy/_build/latest?definitionId=5&branchName=v2)
-[![fuzzit](https://app.fuzzit.dev/badge?org_id=caddyserver-gh)](https://app.fuzzit.dev/orgs/caddyserver-gh/dashboard)
-
-This is the development branch for Caddy 2. This code (version 2) is not yet feature-complete or production-ready, but is already being used in production, and we encourage you to deploy it today on sites that are not very visible or important so that it can obtain crucial experience in the field.
-
-Please file issues to propose new features and report bugs, and after the bug or feature has been discussed, submit a pull request! We need your help to build this web server into what you want it to be. (Caddy 2 issues and pull requests receive priority over Caddy 1 issues and pull requests.)
-
-**Caddy 2 is the web server of the Go community.** We are looking for maintainers to represent the community! Please become involved (issues, PRs, [our forum](https://caddy.community) etc.) and express interest if you are committed to being a collaborator on the Caddy project.
 
 
 ### Menu
 
+- [Features](#features)
+- [Install](#install)
 - [Build from source](#build-from-source)
-- [Quick Start](#quick-start)
+	- [For development](#for-development)
-- [Configuration](#configuration)
+	- [With version information and/or plugins](#with-version-information-andor-plugins)
-- [Full Documentation](#full-documentation)
+- [Quick start](#quick-start)
-- [List of Improvements](#list-of-improvements)
+- [Overview](#overview)
-- [FAQ](#faq)
+- [Full documentation](#full-documentation)
+- [Getting help](#getting-help)
+- [About](#about)
 
+<p align="center">
+	<b>Powered by</b>
+	<br>
+	<a href="https://github.com/caddyserver/certmagic">
+		<picture>
+			<source media="(prefers-color-scheme: dark)" srcset="https://user-images.githubusercontent.com/55066419/206946718-740b6371-3df3-4d72-a822-47e4c48af999.png">
+			<source media="(prefers-color-scheme: light)" srcset="https://user-images.githubusercontent.com/1128849/49704830-49d37200-fbd5-11e8-8385-767e0cd033c3.png">
+			<img src="https://user-images.githubusercontent.com/1128849/49704830-49d37200-fbd5-11e8-8385-767e0cd033c3.png" alt="CertMagic" width="250">
+		</picture>
+	</a>
+</p>
+
+
+## [Features](https://caddyserver.com/v2)
+
+- **Easy configuration** with the [Caddyfile](https://caddyserver.com/docs/caddyfile)
+- **Powerful configuration** with its [native JSON config](https://caddyserver.com/docs/json/)
+- **Dynamic configuration** with the [JSON API](https://caddyserver.com/docs/api)
+- [**Config adapters**](https://caddyserver.com/docs/config-adapters) if you don't like JSON
+- **Automatic HTTPS** by default
+	- [ZeroSSL](https://zerossl.com) and [Let's Encrypt](https://letsencrypt.org) for public names
+	- Fully-managed local CA for internal names & IPs
+	- Can coordinate with other Caddy instances in a cluster
+	- Multi-issuer fallback
+- **Stays up when other servers go down** due to TLS/OCSP/certificate-related issues
+- **Production-ready** after serving trillions of requests and managing millions of TLS certificates
+- **Scales to hundreds of thousands of sites** as proven in production
+- **HTTP/1.1, HTTP/2, and HTTP/3** all supported by default
+- **Highly extensible** [modular architecture](https://caddyserver.com/docs/architecture) lets Caddy do anything without bloat
+- **Runs anywhere** with **no external dependencies** (not even libc)
+- Written in Go, a language with higher **memory safety guarantees** than other servers
+- Actually **fun to use**
+- So much more to [discover](https://caddyserver.com/v2)
+
+## Install
+
+The simplest, cross-platform way to get started is to download Caddy from [GitHub Releases](https://github.com/caddyserver/caddy/releases) and place the executable file in your PATH.
+
+See [our online documentation](https://caddyserver.com/docs/install) for other install instructions.
 
 ## Build from source
 
 Requirements:
 
-- [Go 1.13 or newer](https://golang.org/dl/)
+- [Go 1.20 or newer](https://golang.org/dl/)
-- Do NOT disable [Go modules](https://github.com/golang/go/wiki/Modules) (`export GO111MODULE=auto`)
 
-Download the `v2` source code:
+### For development
-
+
-```bash
+_**Note:** These steps [will not embed proper version information](https://github.com/golang/go/issues/29228). For that, please follow the instructions in the next section._
-$ git clone -b v2 "https://github.com/caddyserver/caddy.git"
-```
-
-Build:
 
 ```bash
+$ git clone "https://github.com/caddyserver/caddy.git"
 $ cd caddy/cmd/caddy/
 $ go build
 ```
 
-That will put a `caddy(.exe)` binary into the current directory. You can move it into your PATH or use `go install` to do that automatically (assuming `$GOPATH/bin` is already in your PATH). You can also use `go run main.go` for quick, temporary builds while developing.
+When you run Caddy, it may try to bind to low ports unless otherwise specified in your config. If your OS requires elevated privileges for this, you will need to give your new binary permission to do so. On Linux, this can be done easily with: `sudo setcap cap_net_bind_service=+ep ./caddy`
 
-The initial build may be slow as dependencies are downloaded. Subsequent builds should be very fast. If you encounter any Go-module-related errors, try clearing your Go module cache (`$GOPATH/pkg/mod`) and Go package cache (`$GOPATH/pkg`) and read [the Go wiki page about modules for help](https://github.com/golang/go/wiki/Modules). If you have issues with Go modules, please consult the Go community for help. But if there is an actual error in Caddy, please report it to us.
+If you prefer to use `go run` which only creates temporary binaries, you can still do this with the included `setcap.sh` like so:
-
-
-## Quick Start
-
-(Until the stable 2.0 release, there may be breaking changes in v2, please be aware!)
-
-These instructions assume an executable build of Caddy 2 is named `caddy` in the current folder. If it's in your PATH, you may omit the path to the binary (`./`).
-
-Start Caddy:
 
 ```bash
-$ ./caddy start
+$ go run -exec ./setcap.sh main.go
 ```
 
-There are no config files with Caddy 2. Instead, you POST configuration to it:
+If you don't want to type your password for `setcap`, use `sudo visudo` to edit your sudoers file and allow your user account to run that command without a password, for example:
 
-```bash
+```
-$ curl -X POST "http://localhost:2019/load" \
+username ALL=(ALL:ALL) NOPASSWD: /usr/sbin/setcap
-    -H "Content-Type: application/json" \
-    -d @- << EOF
-    {
-        "apps": {
-            "http": {
-                "servers": {
-                    "example": {
-                        "listen": ["127.0.0.1:2080"],
-                        "routes": [
-                            {
-                                "handle": [{
-                                    "handler": "file_server",
-                                    "browse": {}
-                                }]
-                            }
-                        ]
-                    }
-                }
-            }
-        }
-    }
-EOF
 ```
 
-Now visit http://localhost:2080 in your browser and you will see the contents of the current directory displayed.
+replacing `username` with your actual username. Please be careful and only do this if you know what you are doing! We are only qualified to document how to use Caddy, not Go tooling or your computer, and we are providing these instructions for convenience only; please learn how to use your own computer at your own risk and make any needful adjustments.
 
-To change Caddy's configuration, simply POST a new payload to that endpoint. Config changes are extremely lightweight and efficient, and should be graceful on all platforms -- _even Windows_.
+### With version information and/or plugins
 
-Updating configuration using heredoc can be tedious, so you can still use a config file if you prefer. Put your configuration in any file (`caddy.json` for example) and then POST that instead:
+Using [our builder tool, `xcaddy`](https://github.com/caddyserver/xcaddy)...
 
-```bash
+```
-$ curl -X POST "http://localhost:2019/load" \
+$ xcaddy build
-    -H "Content-Type: application/json" \
-    -d @caddy.json
 ```
 
-Or you can tell Caddy to load its configuration from a file in the first place (this simply does the work of the above curl command for you):
+...the following steps are automated:
 
-```bash
+1. Create a new folder: `mkdir caddy`
-$ ./caddy start --config caddy.json
+2. Change into it: `cd caddy`
-```
+3. Copy [Caddy's main.go](https://github.com/caddyserver/caddy/blob/master/cmd/caddy/main.go) into the empty folder. Add imports for any custom plugins you want to add.
-
+4. Initialize a Go module: `go mod init caddy`
-To stop Caddy:
+5. (Optional) Pin Caddy version: `go get github.com/caddyserver/caddy/v2@version` replacing `version` with a git tag, commit, or branch name.
-
+6. (Optional) Add plugins by adding their import: `_ "import/path/here"`
-```bash
+7. Compile: `go build`
-$ ./caddy stop
-```
-
-Note that this will stop any process named the same as `os.Args[0]`.
-
-For other commands, please see [the Caddy 2 documentation](https://caddyserver.com/docs/command-line).
-
-### Caddyfile
-
-Caddy 2 can be configured with a Caddyfile, much like in v1, for example:
-
-```plain
-example.com
-
-try_files {path}.html {path}
-encode gzip zstd
-reverse_proxy /api  localhost:9005
-php_fastcgi   /blog unix//path/to/socket
-file_server
-```
-
-Instead of being its primary mode of configuration, an internal _config adapter_ adapts the Caddyfile to Caddy's native JSON structure. You can see it in action with the [`adapt` command](https://caddyserver.com/docs/command-line#caddy-adapt):
-
-```bash
-$ ./caddy adapt --config path/to/Caddyfile --adapter caddyfile --pretty
-```
-
-If you just want to run Caddy with your Caddyfile directly, the CLI wraps this up for you nicely. Either of the following commands:
-
-```bash
-$ ./caddy start
-$ ./caddy run
-```
-
-will use your Caddyfile if it is called `Caddyfile` in the current directory.
-
-If your Caddyfile is somewhere else, you can still use it:
-
-```bash
-$ ./caddy start|run --config path/to/Caddyfile --adapter caddyfile
-```
-
-[Learn more about the Caddyfile in v2.](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#caddyfile-adapter)
 
 
-## Configuration
-
-Caddy 2 exposes an unprecedented level of control compared to any web server in existence. In Caddy 2, you are usually setting the actual values of the initialized types in memory that power everything from your HTTP handlers and TLS handshakes to your storage medium. Caddy 2 is also ridiculously extensible, with a module system that makes vast improvements over Caddy 1's plugin system.
-
-Nearly all of Caddy 2's configuration is contained in a single config document, rather than being spread across CLI flags and env variables and a configuration file as with other web servers (and Caddy 1).
-
-To wield the power of this design, you need to know how the config document is structured. Please see the [the Caddy 2 documentation in our wiki](https://caddyserver.com/docs/json/) for details about Caddy's config structure.
-
-Configuration is normally given to Caddy through an API endpoint, which is likewise documented in the wiki pages. However, you can also use config files of various formats with [config adapters](https://caddyserver.com/docs/config-adapters).
 
 
-## Full Documentation
+## Quick start
 
-Caddy 2 is very much in development, and so is its documentation. You can find it here:
+The [Caddy website](https://caddyserver.com/docs/) has documentation that includes tutorials, quick-start guides, reference, and more.
+
+**We recommend that all users -- regardless of experience level -- do our [Getting Started](https://caddyserver.com/docs/getting-started) guide to become familiar with using Caddy.**
+
+If you've only got a minute, [the website has several quick-start tutorials](https://caddyserver.com/docs/quick-starts) to choose from! However, after finishing a quick-start tutorial, please read more documentation to understand how the software works. 🙂
+
+
+
+
+## Overview
+
+Caddy is most often used as an HTTPS server, but it is suitable for any long-running Go program. First and foremost, it is a platform to run Go applications. Caddy "apps" are just Go programs that are implemented as Caddy modules. Two apps -- `tls` and `http` -- ship standard with Caddy.
+
+Caddy apps instantly benefit from [automated documentation](https://caddyserver.com/docs/json/), graceful on-line [config changes via API](https://caddyserver.com/docs/api), and unification with other Caddy apps.
+
+Although [JSON](https://caddyserver.com/docs/json/) is Caddy's native config language, Caddy can accept input from [config adapters](https://caddyserver.com/docs/config-adapters) which can essentially convert any config format of your choice into JSON: Caddyfile, JSON 5, YAML, TOML, NGINX config, and more.
+
+The primary way to configure Caddy is through [its API](https://caddyserver.com/docs/api), but if you prefer config files, the [command-line interface](https://caddyserver.com/docs/command-line) supports those too.
+
+Caddy exposes an unprecedented level of control compared to any web server in existence. In Caddy, you are usually setting the actual values of the initialized types in memory that power everything from your HTTP handlers and TLS handshakes to your storage medium. Caddy is also ridiculously extensible, with a powerful plugin system that makes vast improvements over other web servers.
+
+To wield the power of this design, you need to know how the config document is structured. Please see [our documentation site](https://caddyserver.com/docs/) for details about [Caddy's config structure](https://caddyserver.com/docs/json/).
+
+Nearly all of Caddy's configuration is contained in a single config document, rather than being scattered across CLI flags and env variables and a configuration file as with other web servers. This makes managing your server config more straightforward and reduces hidden variables/factors.
+
+
+## Full documentation
+
+Our website has complete documentation:
 
 **https://caddyserver.com/docs/**
 
-Note that breaking changes are expected until the stable 2.0 release. The v2 Caddyfile will probably be the last thing documented, as it is rapidly changing.
+The docs are also open source. You can contribute to them here: https://github.com/caddyserver/website
-
-
-## List of Improvements
-
-The following is a non-comprehensive list of significant improvements over Caddy 1. Not everything in this list is finished yet, but they will be finished or at least will be possible with Caddy 2:
-
-- Centralized configuration. No more disparate use of environment variables, config files (potentially multiple!), CLI flags, etc.
-- REST API. Control Caddy with HTTP requests to an administration endpoint. Changes are applied immediately and efficiently.
-- Dynamic configuration. Any and all specific config values can be modified directly through the admin API with a REST endpoint.
-    - Change only specific configuration settings instead of needing to specify the whole config each time. This makes it safe and easy to change Caddy's config with manually-crafted curl commands, for example.
-- No configuration files. Except optionally to bootstrap its configuration at startup. You can still use config files if you wish, and we expect that most people will.
-- Export the current Caddy configuration with an API GET request.
-- Silky-smooth graceful reloads. Update the configuration up to dozens of times per second with no dropped requests and very little memory cost. Our unique graceful reload technology is lighter and faster **and works on all platforms, including Windows**.
-- An embedded scripting language! Caddy2 has native Starlark integration. Do things you never thought possible with higher performance than Lua, JavaScript, and other VMs. Starlark is expressive, familiar (dialect of Python), _almost_ Turing-complete, and highly efficient. (We're still improving performance here.)
-- Using [XDG standards](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html#variables) instead of dumping all assets in `$HOME/.caddy`.
-- Caddy plugins are now called "Caddy modules" (although the terms "plugin" and "module" may be used interchangeably). Caddy modules are a concept unrelated [Go modules](https://github.com/golang/go/wiki/Modules), except that Caddy modules may be implemented by Go modules. Caddy modules are centrally-registered, properly namespaced, and generically loaded & configured, as opposed to how scattered and unorganized Caddy 1-era plugins are.
-- Modules are easier to write, since they do not have to both deserialize their own configuration from a configuration DSL and provision themselves like plugins did. Modules are initialized pre-configured and have the ability to validate the configuration and perform provisioning steps if necessary.
-- Can specify different storage mechanisms in different parts of the configuration, if more than one is needed.
-- "Top-level" Caddy modules are simply called "apps" because literally any long-lived application can be served by Caddy 2.
-- Even more of Caddy is made of modules, allowing for unparalleled extensibility, flexibility, and control. Caddy 2 is arguably the most flexible, extensible, programmable web server ever made.
-- TLS improvements!
-	- TLS configuration is now centralized and decoupled from specific sites
-	- A single certificate cache is used process-wide, reducing duplication and improving memory use
-	- Customize how to manage each certificate ("automation policies") based on the hostname
-	- Automation policy doesn't have to be limited to just ACME - could be any way to manage certificates
-	- Fine-grained control over TLS handshakes
-	- If an ACME challenge fails, other enabled challenges will be tried (no other web server does this)
-    - TLS Session Ticket Ephemeral Keys (STEKs) can be rotated in a cluster for increased performance (no other web server does this either!)
-    - Ability to select a specific certificate per ClientHello given multiple qualifying certificates
-    - Provide TLS certificates without persisting them to disk; keep private keys entirely in memory
-	- Certificate management at startup is now asynchronous and much easier to use through machine reboots and in unsupervised settings
-- All-new HTTP server core
-	- Listeners can be configured for any network type, address, and port range
-	- Customizable TLS connection policies
-	- HTTP handlers are configured by "routes" which consist of matcher and handler components. Match matches an HTTP request, and handle defines the list of handlers to invoke as a result of the match.
-	- Some matchers are regular expressions, which expose capture groups to placeholders.
-	- New matchers include negation and matching based on remote IP address / CIDR ranges.
-    - Placeholders are vastly improved generally
-	- Placeholders (variables) are more properly namespaced.
-	- Multiple routes may match an HTTP request, creating a "composite route" quickly on the fly.
-	- The actual handler for any given request is its composite route.
-	- User defines the order of middlewares (careful! easy to break things).
-	- Adding middlewares no longer requires changes to Caddy's code base (there is no authoritative list).
-	- Routes may be marked as terminal, meaning no more routes will be matched.
-	- Routes may be grouped so that only the first matching route in a group is applied.
-	- Requests may be "re-handled" if they are modified and need to be sent through the chain again (internal redirect).
-	- Vastly more powerful static file server, with native content-negotiation abilities
-    - Done away with URL-rewriting hacks often needed in Caddy 1
-	- Highly descriptive/traceable errors
-	- Very flexible error handling, with the ability to specify a whole list of routes just for error cases
-	- The proxy has numerous improvements, including dynamic backends and more configurable health checks
-	- FastCGI support integrated with the reverse proxy
-	- More control over automatic HTTPS: disable entirely, disable only HTTP->HTTPS redirects, disable only cert management, and for certain names, etc.
-    - Use Starlark to build custom, dynamic HTTP handlers at request-time
-        - We are finding that -- on average -- Caddy 2's Starlark handlers are ~1.25-2x faster than NGINX+Lua.
-
-And a few major features still being worked on:
-
-- Logging
-- Kubernetes ingress controller (mostly done, just polishing it -- and it's amazing)
-- More config adapters. Caddy's native JSON config structure is powerful and complex. Config adapters upsample various formats to Caddy's native config. There are already adapters for Caddyfile, JSON 5, and JSON-C. Planned are NGINX config, YAML, and TOML. The community might be interested in building Traefik and Apache config adapters!
 
 
 
-## FAQ
+## Getting help
 
-### How do I configure Caddy 2?
+- We advise companies using Caddy to secure a support contract through [Ardan Labs](https://www.ardanlabs.com/my/contact-us?dd=caddy) before help is needed.
 
-Caddy's primary mode of configuration is a REST API, which accepts a JSON document. The JSON structure is described [interactively in the docs](https://caddyserver.com/docs/json/). The advantages of exposing this low-level structure are 1) it has near-parity with actual memory initialization, 2) it allows us to offer wrappers over this configuration to any degree of convenience that is needed, and 3) it performs very well under rapid config changes.
+- A [sponsorship](https://github.com/sponsors/mholt) goes a long way! We can offer private help to sponsors. If Caddy is benefitting your company, please consider a sponsorship. This not only helps fund full-time work to ensure the longevity of the project, it provides your company the resources, support, and discounts you need; along with being a great look for your company to your customers and potential customers!
 
-Basically, you will [start Caddy](https://caddyserver.com/docs/command-line#caddy-run), then [POST a JSON config to its API endpoint](https://caddyserver.com/docs/api#post-load).
+- Individuals can exchange help for free on our community forum at https://caddy.community. Remember that people give help out of their spare time and good will. The best way to get help is to give it first!
 
-Although this makes Caddy 2 highly programmable, not everyone will want to configure Caddy via JSON with an API. Sometimes we just want to give Caddy a simple, static config file and have it do its thing. That's what **[config adapters](https://caddyserver.com/docs/config-adapters)** are for! You can configure Caddy more ways than one, depending on your needs and preferences. See the next questions that explain this more.
+Please use our [issue tracker](https://github.com/caddyserver/caddy/issues) only for bug reports and feature requests, i.e. actionable development items (support questions will usually be referred to the forums).
 
-### Caddy 2 feels harder to use. How is this an improvement over Caddy 1?
 
-Caddy's ease of use is one of the main reasons it is special. We are not taking that away in Caddy 2, but first we had to be sure to tackle the fundamental design limitations with Caddy 1. Usability can then be layered on top. This approach has several advantages which we discuss in the next question.
 
-### What about the Caddyfile; are there easier ways to configure Caddy 2?
+## About
 
-Yes! Caddy's native JSON configuration via API is nice when you are automating config changes at scale, but if you just have a simple, static configuration in a file, you can do that too with the [Caddyfile](https://github.com/caddyserver/caddy/wiki/v2:-Documentation#caddyfile-adapter).
+Matthew Holt began developing Caddy in 2014 while studying computer science at Brigham Young University. (The name "Caddy" was chosen because this software helps with the tedious, mundane tasks of serving the Web, and is also a single place for multiple things to be organized together.) It soon became the first web server to use HTTPS automatically and by default, and now has hundreds of contributors and has served trillions of HTTPS requests.
 
-The v2 Caddyfile is very similar to the v1 Caddyfile, but they are not compatible. Several improvements have been made to request matching and directives in v2, giving you more power with less complexity and fewer inconsistencies.
+**The name "Caddy" is trademarked.** The name of the software is "Caddy", not "Caddy Server" or "CaddyServer". Please call it "Caddy" or, if you wish to clarify, "the Caddy web server". Caddy is a registered trademark of Stack Holdings GmbH.
 
-Caddy's default _config adapter_ is the Caddyfile adapter. This takes a Caddyfile as input and [outputs the JSON config](https://caddyserver.com/docs/command-line#caddy-adapt). You can even run Caddy directly without having to see or think about the underlying JSON config.
+- _Project on Twitter: [@caddyserver](https://twitter.com/caddyserver)_
+- _Author on Twitter: [@mholt6](https://twitter.com/mholt6)_
 
-The following _config adapters_ are already being built or plan to be built:
+Caddy is a project of [ZeroSSL](https://zerossl.com), a Stack Holdings company.
 
-- Caddyfile
+Debian package repository hosting is graciously provided by [Cloudsmith](https://cloudsmith.com). Cloudsmith is the only fully hosted, cloud-native, universal package management solution, that enables your organization to create, store and share packages in any format, to any place, with total confidence.
-- JSON 5
-- JSON-C
-- nginx
-- YAML
-- TOML
-- any others that the community would like to contribute
-
-Config adapters allow you to configure Caddy not just one way but _any_ of these ways. For example, you'll be able to bring your existing NGINX config to Caddy and it will spit out the Caddy config JSON you need (to the best of its ability). How cool is that! You can then easily tweak the resulting config by hand, if necessary.
-
-All config adapters vary in their theoretical expressiveness; that is, if you need more advanced configuration you'll have to drop down to the JSON config, because the Caddyfile or an nginx config may not be expressive enough.
-
-However, we expect that most users will be able to use the Caddyfile (or another easy config adapter) exclusively for their sites.
-
-### Why JSON for configuration? Why not _&lt;any other serialization format&gt;_?
-
-We know there might be strong opinions on this one. Regardless, for Caddy 2, we've decided to go with JSON. If that proves to be a fatal mistake, then Caddy 3 probably won't use JSON.
-
-JSON may not be the fastest, the most compact, the easiest to write, serialization format that exists. But those aren't our goals. It has withstood the test of time and checks all our boxes.
-
-- It is almost entirely ubiquitous. JSON works natively in web browsers and has mature libraries in pretty much every language.
-- It is human-readable (as opposed to a binary format).
-- It is easy to tweak by hand. Although composing raw JSON by hand is not awesome, this will not be mainstream once our config adapters are done.
-- It is generally easy to convert other serializations or config formats into JSON, as opposed to the other way around.
-- Even though JSON deserialization is not fast per-se, that kind of performance is not really a concern since config reloads are not the server's hottest path like HTTP request handling or TLS handshakes are. Even with JSON, Caddy 2 can handle dozens of config changes per second, which is probably plenty for now.
-- It maps almost 1:1 to the actual, in-memory values that power your HTTP handlers and other parts of the server (no need to parse a config file with some arbitrary DSL and do a bunch of extra pre-processing).
-
-Ultimately, we think all these properties are appropriate -- if not ideal -- for a web server configuration.
-
-If you're still not happy with the choice of JSON, feel free to contribute a config adapter of your own choice!
-
-Or just use YAML or TOML, which seamlessly translate to JSON.
-
-### JSON is declarative; what if I need more programmability (i.e. imperative syntax)?
-
-NGINX also realized the need for imperative logic in declarative configs, so they tried "if" statements, [but it was a bad idea](https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/).
-
-We have good news. Caddy 2 can give you the power of imperative logic without the perils of mixing declarative and imperative config such as befell NGINX. We do this by allowing embedded imperative syntax awithin the Caddy's declarative config.
-
-Caddy 2's configuration is declarative because configuration is very much declarative in nature. Configuration is a tricky medium, as it is read and written by both computers and humans. Computers use it, but humans constantly refer to it and update it. Declarative syntaxes are fairly straightforward to make sense of, whereas it is difficult to reason about imperative logic.
-
-However, sometimes computation is useful, and in some cases, the only way to express what you need. This can be illustrated really well in the simple case of trying to decide whether a particular HTTP middleware should be invoked as part of an HTTP request. A lot of the time, such logic is as simple as: "GET requests for any path starting with /foo/bar", which can be expressed declaratively in JSON:
-
-```json
-{
-	"method": "GET",
-	"path": "/foo/bar"
-}
-```
-
-But what if you need to match /foo/bar OR /topaz? How do you express that OR clause? Maybe an array:
-
-```json
-{
-	"method": ["GET"],
-	"path": ["/foo/bar", "/topaz"]
-}
-```
-
-Now what if you need add a NOT or AND clause? JSON quickly tires out. As you learn about Caddy 2's request matching, you will see how we handled this. Caddy 2's JSON gives you the ability to express moderately-complex logic such as:
-
-```js
-// this is not actual Caddy config, just logic pseudocode
-IF (Host = "example.com")
-	OR (Host = "sub.example.com" AND Path != "/foo/bar")
-```
-
-Already, this is more expressive power than most web servers offer with their native config, yet Caddy 2 offers this in JSON.
-
-But in most web servers, to make logic this complex feasible, you'll generally call out to Lua or some extra DSL. For example, in NGINX you could use a Lua module to express this logic. Traefik 2.0 has [yet another kind of clunky-looking custom DSL](https://blog.containo.us/back-to-traefik-2-0-2f9aa17be305#d22e) just for this.
-
-Caddy 2 solves this in a novel way with [Starlark expressions](https://godoc.org/go.starlark.net/starlark#Eval). Starlark is a familiar dialect of Python! So, no new DSLs to learn and no VMs to slow things down:
-
-```python
-req.host == 'example.com' ||
-	(req.host == 'sub.example.com' && req.path != '/foo/bar')
-```
-
-Starlark performs at least as well as NGINX+Lua (more performance tests ongoing, as well as optimizations to make it even faster!) and because it's basically Python, it's familiar and easy to use.
-
-In summary: Caddy 2 config is declarative, but can be imperative where that is useful.
-
-### What is Caddy 2 licensed as?
-
-Caddy 2 is licensed under the Apache 2.0 open source license. There are no official Caddy 2 distributions that are proprietary.
-
-### Does Caddy 2 have telemetry?
-
-No. There was not enough academic interest to continue supporting it. If telemetry does get added later, it will not be on by default or will be vastly reduced in its scope.
-
-## Does Caddy 2 use HTTPS by default?
-
-Yes. HTTPS is automatic and enabled by default when possible, just like in Caddy 1. Basically, if your HTTP routes specify a `host` matcher with qualifying domain names, those names will be enabled for automatic HTTPS. Automatic HTTPS is disabled for domains which match certificates that are manually loaded by your config.
-
-## How do I avoid Let's Encrypt rate limits with Caddy 2?
-
-As you are testing and developing with Caddy 2, you should use test ("staging") certificates from Let's Encrypt to avoid rate limits. By default, Caddy 2 uses Let's Encrypt's production endpoint to get real certificates for your domains, but their [rate limits](https://letsencrypt.org/docs/rate-limits/) forbid testing and development use of this endpoint for good reasons. You can switch to their [staging endpoint](https://letsencrypt.org/docs/staging-environment/) by adding the staging CA to your automation policy in the `tls` app:
-
-```json
-"tls": {
-	"automation": {
-		"policies": [
-			{
-				"management": {
-					"module": "acme",
-					"ca": "https://acme-staging-v02.api.letsencrypt.org/directory"
-				}
-			}
-		]
-	}
-}
-```
-
-Or with the Caddyfile, using a global options block at the top:
-
-```
-{
-	acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
-}
-```
-
-## Can we get some access controls on the admin endpoint?
-
-Yeah, that's coming. For now, you can use a permissioned unix socket for some basic security.

admin_test.go

@@ -16,10 +16,31 @@ package caddy
 
 import (
 	"encoding/json"
+	"fmt"
+	"net/http"
 	"reflect"
+	"sync"
 	"testing"
 )
 
+var testCfg = []byte(`{
+			"apps": {
+				"http": {
+					"servers": {
+						"myserver": {
+							"listen": ["tcp/localhost:8080-8084"],
+							"read_timeout": "30s"
+						},
+						"yourserver": {
+							"listen": ["127.0.0.1:5000"],
+							"read_header_timeout": "15s"
+						}
+					}
+				}
+			}
+		}
+		`)
+
 func TestUnsyncedConfigAccess(t *testing.T) {
 	// each test is performed in sequence, so
 	// each change builds on the previous ones;
@@ -54,6 +75,12 @@ func TestUnsyncedConfigAccess(t *testing.T) {
 			path:   "/bar/qq",
 			expect: `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c"]}`,
 		},
+		{
+			method:    "DELETE",
+			path:      "/bar/qq",
+			expect:    `{"foo": "jet", "bar": {"aa": "bb"}, "list": ["a", "b", "c"]}`,
+			shouldErr: true,
+		},
 		{
 			method:  "POST",
 			path:    "/list",
@@ -94,7 +121,7 @@ func TestUnsyncedConfigAccess(t *testing.T) {
 		}
 
 		// decode the expected config so we can do a convenient DeepEqual
-		var expectedDecoded interface{}
+		var expectedDecoded any
 		err = json.Unmarshal([]byte(tc.expect), &expectedDecoded)
 		if err != nil {
 			t.Fatalf("Test %d: Unmarshaling expected config: %v", i, err)
@@ -108,25 +135,71 @@ func TestUnsyncedConfigAccess(t *testing.T) {
 	}
 }
 
-func BenchmarkLoad(b *testing.B) {
+// TestLoadConcurrent exercises Load under concurrent conditions
-	for i := 0; i < b.N; i++ {
+// and is most useful under test with `-race` enabled.
-		cfg := []byte(`{
+func TestLoadConcurrent(t *testing.T) {
-			"apps": {
+	var wg sync.WaitGroup
-				"http": {
+
-					"servers": {
+	for i := 0; i < 100; i++ {
-						"myserver": {
+		wg.Add(1)
-							"listen": ["tcp/localhost:8080-8084"],
+		go func() {
-							"read_timeout": "30s"
+			_ = Load(testCfg, true)
-						},
+			wg.Done()
-						"yourserver": {
+		}()
-							"listen": ["127.0.0.1:5000"],
-							"read_header_timeout": "15s"
 	}
-					}
+	wg.Wait()
-				}
+}
-			}
+
-		}
+type fooModule struct {
-		`)
+	IntField int
-		Load(cfg, true)
+	StrField string
+}
+
+func (fooModule) CaddyModule() ModuleInfo {
+	return ModuleInfo{
+		ID:  "foo",
+		New: func() Module { return new(fooModule) },
+	}
+}
+func (fooModule) Start() error { return nil }
+func (fooModule) Stop() error  { return nil }
+
+func TestETags(t *testing.T) {
+	RegisterModule(fooModule{})
+
+	if err := Load([]byte(`{"admin": {"listen": "localhost:2999"}, "apps": {"foo": {"strField": "abc", "intField": 0}}}`), true); err != nil {
+		t.Fatalf("loading: %s", err)
+	}
+
+	const key = "/" + rawConfigKey + "/apps/foo"
+
+	// try update the config with the wrong etag
+	err := changeConfig(http.MethodPost, key, []byte(`{"strField": "abc", "intField": 1}}`), fmt.Sprintf(`"/%s not_an_etag"`, rawConfigKey), false)
+	if apiErr, ok := err.(APIError); !ok || apiErr.HTTPStatus != http.StatusPreconditionFailed {
+		t.Fatalf("expected precondition failed; got %v", err)
+	}
+
+	// get the etag
+	hash := etagHasher()
+	if err := readConfig(key, hash); err != nil {
+		t.Fatalf("reading: %s", err)
+	}
+
+	// do the same update with the correct key
+	err = changeConfig(http.MethodPost, key, []byte(`{"strField": "abc", "intField": 1}`), makeEtag(key, hash), false)
+	if err != nil {
+		t.Fatalf("expected update to work; got %v", err)
+	}
+
+	// now try another update. The hash should no longer match and we should get precondition failed
+	err = changeConfig(http.MethodPost, key, []byte(`{"strField": "abc", "intField": 2}`), makeEtag(key, hash), false)
+	if apiErr, ok := err.(APIError); !ok || apiErr.HTTPStatus != http.StatusPreconditionFailed {
+		t.Fatalf("expected precondition failed; got %v", err)
+	}
+}
+
+func BenchmarkLoad(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		Load(testCfg, true)
 	}
 }

azure-pipelines.yml

@@ -1,240 +0,0 @@
-# Mutilated beyond recognition from the example at:
-# https://docs.microsoft.com/azure/devops/pipelines/languages/go
-
-trigger:
-  - v2
-
-schedules:
-- cron: "0 0 * * *"
-  displayName: Daily midnight fuzzing
-  branches:
-    include:
-    - v2
-  always: true
-
-variables:
-  GOROOT: $(gorootDir)/go
-  GOPATH: $(system.defaultWorkingDirectory)/gopath
-  GOBIN:  $(GOPATH)/bin
-  modulePath: '$(GOPATH)/src/github.com/$(build.repository.name)'
-  # TODO: Remove once it's enabled by default
-  GO111MODULE: on
-
-jobs:
-- job: crossPlatformTest
-  displayName: "Cross-Platform Tests"
-  strategy:
-    matrix:
-      linux:
-        imageName: ubuntu-16.04
-        gorootDir: /usr/local
-      mac:
-        imageName: macos-10.13
-        gorootDir: /usr/local
-      windows:
-        imageName: windows-2019
-        gorootDir: C:\
-  pool:
-    vmImage: $(imageName)
-  
-  steps:
-  - bash: |
-      latestGo=$(curl "https://golang.org/VERSION?m=text")
-      echo "##vso[task.setvariable variable=LATEST_GO]$latestGo"
-      echo "Latest Go version: $latestGo"
-    displayName: "Get latest Go version"
-
-  - bash: |
-      sudo rm -f $(which go)
-      echo '##vso[task.prependpath]$(GOBIN)'
-      echo '##vso[task.prependpath]$(GOROOT)/bin'
-      mkdir -p '$(modulePath)'
-      shopt -s extglob
-      shopt -s dotglob
-      mv !(gopath) '$(modulePath)'
-    displayName: Remove old Go, set GOBIN/GOROOT, and move project into GOPATH
-
-  # Install Go (this varies by platform)
-  - bash: |
-      wget "https://dl.google.com/go/$(LATEST_GO).linux-amd64.tar.gz"
-      sudo tar -C $(gorootDir) -xzf "$(LATEST_GO).linux-amd64.tar.gz"
-    condition: eq( variables['Agent.OS'], 'Linux' )
-    displayName: Install Go on Linux
-
-  - bash: |
-      wget "https://dl.google.com/go/$(LATEST_GO).darwin-amd64.tar.gz"
-      sudo tar -C $(gorootDir) -xzf "$(LATEST_GO).darwin-amd64.tar.gz"
-    condition: eq( variables['Agent.OS'], 'Darwin' )
-    displayName: Install Go on macOS
-
-  # The low performance is partly due to PowerShell's attempt to update the progress bar. Disabling it speeds up the process.
-  # Reference: https://github.com/PowerShell/PowerShell/issues/2138
-  - powershell: |
-      $ProgressPreference = 'SilentlyContinue'
-      Write-Host "Downloading Go..."
-      (New-Object System.Net.WebClient).DownloadFile("https://dl.google.com/go/$(LATEST_GO).windows-amd64.zip", "$(LATEST_GO).windows-amd64.zip")
-      Write-Host "Extracting Go... (I'm slow too)"
-      7z x "$(LATEST_GO).windows-amd64.zip" -o"$(gorootDir)"
-    condition: eq( variables['Agent.OS'], 'Windows_NT' )
-    displayName: Install Go on Windows
-
-  - bash: curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.22.2
-    displayName: Install golangci-lint
-
-  - script: |
-      go get github.com/axw/gocov/gocov
-      go get github.com/AlekSi/gocov-xml
-      go get -u github.com/jstemmer/go-junit-report
-    displayName: Install test and coverage analysis tools
-
-  - bash: |
-      printf "Using go at: $(which go)\n"
-      printf "Go version: $(go version)\n"
-      printf "\n\nGo environment:\n\n"
-      go env
-      printf "\n\nSystem environment:\n\n"
-      env
-    displayName: Print Go version and environment
-
-  - script: |
-      go get -v -t -d ./...
-      mkdir test-results
-    workingDirectory: '$(modulePath)'
-    displayName: Get dependencies
-
-  # its behavior is governed by .golangci.yml
-  - script: |
-      (golangci-lint run --out-format junit-xml) > test-results/lint-result.xml
-      exit 0
-    workingDirectory: '$(modulePath)'
-    continueOnError: true
-    displayName: Run lint check
-
-  - script: |
-      (go test -v -coverprofile=cover-profile.out -race ./... 2>&1) > test-results/test-result.out
-    workingDirectory: '$(modulePath)'
-    continueOnError: true
-    displayName: Run tests
-
-  - script: |
-      mkdir coverage
-      gocov convert cover-profile.out > coverage/coverage.json
-      # Because Windows doesn't work with input redirection like *nix, but output redirection works.
-      (cat ./coverage/coverage.json | gocov-xml) > coverage/coverage.xml 
-    workingDirectory: '$(modulePath)'
-    displayName: Prepare coverage reports
-
-  - script: |
-      (cat ./test-results/test-result.out | go-junit-report) > test-results/test-result.xml
-    workingDirectory: '$(modulePath)'
-    displayName: Prepare test report
-
-  - task: PublishCodeCoverageResults@1
-    displayName: Publish test coverage report
-    inputs:
-      codeCoverageTool: Cobertura 
-      summaryFileLocation: $(modulePath)/coverage/coverage.xml
-
-  - task: PublishTestResults@2
-    displayName: Publish unit test
-    inputs:
-      testResultsFormat: 'JUnit'
-      testResultsFiles: $(modulePath)/test-results/test-result.xml
-      testRunTitle: $(agent.OS) Unit Test
-      mergeTestResults: false
-
-  - task: PublishTestResults@2
-    displayName: Publish lint results
-    inputs:
-      testResultsFormat: 'JUnit'
-      testResultsFiles: $(modulePath)/test-results/lint-result.xml
-      testRunTitle: $(agent.OS) Lint
-      mergeTestResults: false
-
-  - bash: |
-      exit 1
-    condition: eq(variables['Agent.JobStatus'], 'SucceededWithIssues')
-    displayName: Coerce correct build result
-
-- job: fuzzing
-  displayName: 'Fuzzing'
-  # Only run this job on schedules or PRs for non-forks.
-  condition: or(eq(variables['System.PullRequest.IsFork'], 'False'), eq(variables['Build.Reason'], 'Schedule') )
-  strategy:
-    matrix:
-      linux:
-        imageName: ubuntu-16.04
-        gorootDir: /usr/local
-  pool:
-    vmImage: $(imageName)
-
-  steps:
-  - bash: |
-      latestGo=$(curl "https://golang.org/VERSION?m=text")
-      echo "##vso[task.setvariable variable=LATEST_GO]$latestGo"
-      echo "Latest Go version: $latestGo"
-    displayName: "Get latest Go version"
-
-  - bash: |
-      sudo rm -f $(which go)
-      echo '##vso[task.prependpath]$(GOBIN)'
-      echo '##vso[task.prependpath]$(GOROOT)/bin'
-      mkdir -p '$(modulePath)'
-      shopt -s extglob
-      shopt -s dotglob
-      mv !(gopath) '$(modulePath)'
-    displayName: Remove old Go, set GOBIN/GOROOT, and move project into GOPATH
-
-  - bash: |
-      wget "https://dl.google.com/go/$(LATEST_GO).linux-amd64.tar.gz"
-      sudo tar -C $(gorootDir) -xzf "$(LATEST_GO).linux-amd64.tar.gz"
-    condition: eq( variables['Agent.OS'], 'Linux' )
-    displayName: Install Go on Linux
-
-  - bash: |
-      # Install Clang-7.0 because other versions seem to be missing the file libclang_rt.fuzzer-x86_64.a
-      sudo add-apt-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-7 main"
-      wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
-      sudo apt update && sudo apt install -y clang-7 lldb-7 lld-7
-
-      go get -v github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build
-      wget -q -O fuzzit https://github.com/fuzzitdev/fuzzit/releases/download/v2.4.77/fuzzit_Linux_x86_64
-      chmod a+x fuzzit
-      mv fuzzit $(GOBIN)
-    displayName: Download go-fuzz tools and the Fuzzit CLI, and move Fuzzit CLI to GOBIN
-    condition: and(eq(variables['System.PullRequest.IsFork'], 'False') , eq( variables['Agent.OS'], 'Linux' ))
-
-  - bash: |
-      declare -A fuzzers_funcs=(\
-        ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="FuzzParseAddress" \
-        ["./caddyconfig/caddyfile/parse_fuzz.go"]="FuzzParseCaddyfile" \
-        ["./listeners_fuzz.go"]="FuzzParseNetworkAddress" \
-        ["./replacer_fuzz.go"]="FuzzReplacer" \
-      )
-
-      declare -A fuzzers_targets=(\
-        ["./caddyconfig/httpcaddyfile/addresses_fuzz.go"]="parse-address" \
-        ["./caddyconfig/caddyfile/parse_fuzz.go"]="parse-caddyfile" \
-        ["./listeners_fuzz.go"]="parse-network-address" \
-        ["./replacer_fuzz.go"]="replacer" \
-      )
-
-      fuzz_type="local-regression"
-      if [[ $(Build.Reason) == "Schedule" ]]; then
-        fuzz_type="fuzzing"
-      fi
-      echo "Fuzzing type: $fuzz_type"
-
-      for f in $(find . -name \*_fuzz.go); do
-        FUZZER_DIRECTORY=$(dirname $f)
-        echo "go-fuzz-build func ${fuzzers_funcs[$f]} residing in $f"
-        go-fuzz-build -func "${fuzzers_funcs[$f]}" -libfuzzer -o "$FUZZER_DIRECTORY/${fuzzers_targets[$f]}.a" $FUZZER_DIRECTORY
-        echo "Generating fuzzer binary of func ${fuzzers_funcs[$f]} which resides in $f"
-        clang-7 -fsanitize=fuzzer "$FUZZER_DIRECTORY/${fuzzers_targets[$f]}.a" -o "$FUZZER_DIRECTORY/${fuzzers_targets[$f]}"
-        fuzzit create job caddyserver/${fuzzers_targets[$f]} $FUZZER_DIRECTORY/${fuzzers_targets[$f]} --api-key ${FUZZIT_API_KEY} --type "${fuzz_type}" --branch "${SYSTEM_PULLREQUEST_SOURCEBRANCH}" --revision "${BUILD_SOURCEVERSION}"
-        echo "Completed $f"
-      done
-    env:
-      FUZZIT_API_KEY: $(FUZZIT_API_KEY)
-    workingDirectory: '$(modulePath)'
-    displayName: Generate fuzzers & submit them to Fuzzit

caddy.go

@@ -17,10 +17,11 @@ package caddy
 import (
 	"bytes"
 	"context"
+	"encoding/hex"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"log"
 	"net/http"
 	"os"
@@ -30,12 +31,25 @@ import (
 	"strconv"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"
 
-	"github.com/mholt/certmagic"
+	"github.com/caddyserver/certmagic"
+	"github.com/google/uuid"
 	"go.uber.org/zap"
+
+	"github.com/caddyserver/caddy/v2/notify"
 )
 
+func init() {
+	RegisterNamespace("", []interface{}{
+		(*App)(nil),
+	})
+	RegisterNamespace("caddy.storage", []interface{}{
+		(*StorageConverter)(nil),
+	})
+}
+
 // Config is the top (or beginning) of the Caddy configuration structure.
 // Caddy config is expressed natively as a JSON document. If you prefer
 // not to work with JSON directly, there are [many config adapters](/docs/config-adapters)
@@ -67,11 +81,15 @@ type Config struct {
 	// module is `caddy.storage.file_system` (the local file system),
 	// and the default path
 	// [depends on the OS and environment](/docs/conventions#data-directory).
+	// A storage `module` should implement the following interfaces:
+	// - [StorageConverter](https://pkg.go.dev/github.com/caddyserver/caddy/v2#StorageConverter)
 	StorageRaw json.RawMessage `json:"storage,omitempty" caddy:"namespace=caddy.storage inline_key=module"`
 
 	// AppsRaw are the apps that Caddy will load and run. The
 	// app module name is the key, and the app's config is the
 	// associated value.
+	// An `app` should implement the following interfaces:
+	// - [caddy.App](https://pkg.go.dev/github.com/caddyserver/caddy/v2?tab=doc#App)
 	AppsRaw ModuleMap `json:"apps,omitempty" caddy:"namespace="`
 
 	apps    map[string]App
@@ -99,16 +117,50 @@ func Run(cfg *Config) error {
 // if it is different from the current config or
 // forceReload is true.
 func Load(cfgJSON []byte, forceReload bool) error {
-	return changeConfig(http.MethodPost, "/"+rawConfigKey, cfgJSON, forceReload)
+	if err := notify.Reloading(); err != nil {
+		Log().Error("unable to notify service manager of reloading state", zap.Error(err))
+	}
+
+	// after reload, notify system of success or, if
+	// failure, update with status (error message)
+	var err error
+	defer func() {
+		if err != nil {
+			if notifyErr := notify.Error(err, 0); notifyErr != nil {
+				Log().Error("unable to notify to service manager of reload error",
+					zap.Error(notifyErr),
+					zap.String("reload_err", err.Error()))
+			}
+			return
+		}
+		if err := notify.Ready(); err != nil {
+			Log().Error("unable to notify to service manager of ready state", zap.Error(err))
+		}
+	}()
+
+	err = changeConfig(http.MethodPost, "/"+rawConfigKey, cfgJSON, "", forceReload)
+	if errors.Is(err, errSameConfig) {
+		err = nil // not really an error
+	}
+
+	return err
 }
 
 // changeConfig changes the current config (rawCfg) according to the
 // method, traversed via the given path, and uses the given input as
 // the new value (if applicable; i.e. "DELETE" doesn't have an input).
 // If the resulting config is the same as the previous, no reload will
-// occur unless forceReload is true. This function is safe for
+// occur unless forceReload is true. If the config is unchanged and not
+// forcefully reloaded, then errConfigUnchanged This function is safe for
 // concurrent use.
-func changeConfig(method, path string, input []byte, forceReload bool) error {
+// The ifMatchHeader can optionally be given a string of the format:
+//
+//	"<path> <hash>"
+//
+// where <path> is the absolute path in the config and <hash> is the expected hash of
+// the config at that path. If the hash in the ifMatchHeader doesn't match
+// the hash of the config, then an APIError with status 412 will be returned.
+func changeConfig(method, path string, input []byte, ifMatchHeader string, forceReload bool) error {
 	switch method {
 	case http.MethodGet,
 		http.MethodHead,
@@ -118,8 +170,42 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 		return fmt.Errorf("method not allowed")
 	}
 
-	currentCfgMu.Lock()
+	rawCfgMu.Lock()
-	defer currentCfgMu.Unlock()
+	defer rawCfgMu.Unlock()
+
+	if ifMatchHeader != "" {
+		// expect the first and last character to be quotes
+		if len(ifMatchHeader) < 2 || ifMatchHeader[0] != '"' || ifMatchHeader[len(ifMatchHeader)-1] != '"' {
+			return APIError{
+				HTTPStatus: http.StatusBadRequest,
+				Err:        fmt.Errorf("malformed If-Match header; expect quoted string"),
+			}
+		}
+
+		// read out the parts
+		parts := strings.Fields(ifMatchHeader[1 : len(ifMatchHeader)-1])
+		if len(parts) != 2 {
+			return APIError{
+				HTTPStatus: http.StatusBadRequest,
+				Err:        fmt.Errorf("malformed If-Match header; expect format \"<path> <hash>\""),
+			}
+		}
+
+		// get the current hash of the config
+		// at the given path
+		hash := etagHasher()
+		err := unsyncedConfigAccess(http.MethodGet, parts[0], nil, hash)
+		if err != nil {
+			return err
+		}
+
+		if hex.EncodeToString(hash.Sum(nil)) != parts[1] {
+			return APIError{
+				HTTPStatus: http.StatusPreconditionFailed,
+				Err:        fmt.Errorf("If-Match header did not match current config hash"),
+			}
+		}
+	}
 
 	err := unsyncedConfigAccess(method, path, input, nil)
 	if err != nil {
@@ -130,15 +216,15 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 	newCfg, err := json.Marshal(rawCfg[rawConfigKey])
 	if err != nil {
 		return APIError{
-			Code: http.StatusBadRequest,
+			HTTPStatus: http.StatusBadRequest,
 			Err:        fmt.Errorf("encoding new config: %v", err),
 		}
 	}
 
 	// if nothing changed, no need to do a whole reload unless the client forces it
 	if !forceReload && bytes.Equal(rawCfgJSON, newCfg) {
-		Log().Named("admin.api").Info("config is unchanged")
+		Log().Info("config is unchanged")
-		return nil
+		return errSameConfig
 	}
 
 	// find any IDs in this config and index them
@@ -146,21 +232,21 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 	err = indexConfigObjects(rawCfg[rawConfigKey], "/"+rawConfigKey, idx)
 	if err != nil {
 		return APIError{
-			Code: http.StatusInternalServerError,
+			HTTPStatus: http.StatusInternalServerError,
 			Err:        fmt.Errorf("indexing config: %v", err),
 		}
 	}
 
 	// load this new config; if it fails, we need to revert to
 	// our old representation of caddy's actual config
-	err = unsyncedDecodeAndRun(newCfg)
+	err = unsyncedDecodeAndRun(newCfg, true)
 	if err != nil {
 		if len(rawCfgJSON) > 0 {
 			// restore old config state to keep it consistent
 			// with what caddy is still running; we need to
 			// unmarshal it again because it's likely that
 			// pointers deep in our rawCfg map were modified
-			var oldCfg interface{}
+			var oldCfg any
 			err2 := json.Unmarshal(rawCfgJSON, &oldCfg)
 			if err2 != nil {
 				err = fmt.Errorf("%v; additionally, restoring old config: %v", err, err2)
@@ -185,18 +271,18 @@ func changeConfig(method, path string, input []byte, forceReload bool) error {
 // readConfig traverses the current config to path
 // and writes its JSON encoding to out.
 func readConfig(path string, out io.Writer) error {
-	currentCfgMu.RLock()
+	rawCfgMu.RLock()
-	defer currentCfgMu.RUnlock()
+	defer rawCfgMu.RUnlock()
 	return unsyncedConfigAccess(http.MethodGet, path, nil, out)
 }
 
-// indexConfigObjects recurisvely searches ptr for object fields named
+// indexConfigObjects recursively searches ptr for object fields named
 // "@id" and maps that ID value to the full configPath in the index.
 // This function is NOT safe for concurrent access; obtain a write lock
-// on currentCfgMu.
+// on currentCtxMu.
-func indexConfigObjects(ptr interface{}, configPath string, index map[string]string) error {
+func indexConfigObjects(ptr any, configPath string, index map[string]string) error {
 	switch val := ptr.(type) {
-	case map[string]interface{}:
+	case map[string]any:
 		for k, v := range val {
 			if k == idKey {
 				switch idVal := v.(type) {
@@ -215,7 +301,7 @@ func indexConfigObjects(ptr interface{}, configPath string, index map[string]str
 				return err
 			}
 		}
-	case []interface{}:
+	case []any:
 		// traverse each element of the array recursively
 		for i := range val {
 			err := indexConfigObjects(val[i], path.Join(configPath, strconv.Itoa(i)), index)
@@ -233,47 +319,66 @@ func indexConfigObjects(ptr interface{}, configPath string, index map[string]str
 // it as the new config, replacing any other current config.
 // It does NOT update the raw config state, as this is a
 // lower-level function; most callers will want to use Load
-// instead. A write lock on currentCfgMu is required!
+// instead. A write lock on rawCfgMu is required! If
-func unsyncedDecodeAndRun(cfgJSON []byte) error {
+// allowPersist is false, it will not be persisted to disk,
+// even if it is configured to.
+func unsyncedDecodeAndRun(cfgJSON []byte, allowPersist bool) error {
 	// remove any @id fields from the JSON, which would cause
 	// loading to break since the field wouldn't be recognized
 	strippedCfgJSON := RemoveMetaFields(cfgJSON)
 
 	var newCfg *Config
-	err := strictUnmarshalJSON(strippedCfgJSON, &newCfg)
+	err := StrictUnmarshalJSON(strippedCfgJSON, &newCfg)
 	if err != nil {
 		return err
 	}
 
+	// prevent recursive config loads; that is a user error, and
+	// although frequent config loads should be safe, we cannot
+	// guarantee that in the presence of third party plugins, nor
+	// do we want this error to go unnoticed (we assume it was a
+	// pulled config if we're not allowed to persist it)
+	if !allowPersist &&
+		newCfg != nil &&
+		newCfg.Admin != nil &&
+		newCfg.Admin.Config != nil &&
+		newCfg.Admin.Config.LoadRaw != nil &&
+		newCfg.Admin.Config.LoadDelay <= 0 {
+		return fmt.Errorf("recursive config loading detected: pulled configs cannot pull other configs without positive load_delay")
+	}
+
 	// run the new config and start all its apps
-	err = run(newCfg, true)
+	ctx, err := run(newCfg, true)
 	if err != nil {
 		return err
 	}
 
-	// swap old config with the new one
+	// swap old context (including its config) with the new one
-	oldCfg := currentCfg
+	currentCtxMu.Lock()
-	currentCfg = newCfg
+	oldCtx := currentCtx
+	currentCtx = ctx
+	currentCtxMu.Unlock()
 
 	// Stop, Cleanup each old app
-	unsyncedStop(oldCfg)
+	unsyncedStop(oldCtx)
 
 	// autosave a non-nil config, if not disabled
-	if newCfg != nil &&
+	if allowPersist &&
+		newCfg != nil &&
 		(newCfg.Admin == nil ||
 			newCfg.Admin.Config == nil ||
 			newCfg.Admin.Config.Persist == nil ||
 			*newCfg.Admin.Config.Persist) {
 		dir := filepath.Dir(ConfigAutosavePath)
-		err := os.MkdirAll(dir, 0700)
+		err := os.MkdirAll(dir, 0o700)
 		if err != nil {
 			Log().Error("unable to create folder for config autosave",
 				zap.String("dir", dir),
 				zap.Error(err))
 		} else {
-			err := ioutil.WriteFile(ConfigAutosavePath, cfgJSON, 0600)
+			err := os.WriteFile(ConfigAutosavePath, cfgJSON, 0o600)
 			if err == nil {
-				Log().Info("autosaved config", zap.String("file", ConfigAutosavePath))
+				Log().Info("autosaved config (load with --resume flag)", zap.String("file", ConfigAutosavePath))
 			} else {
 				Log().Error("unable to autosave config",
 					zap.String("file", ConfigAutosavePath),
@@ -299,7 +404,7 @@ func unsyncedDecodeAndRun(cfgJSON []byte) error {
 // This is a low-level function; most callers
 // will want to use Run instead, which also
 // updates the config's raw state.
-func run(newCfg *Config, start bool) error {
+func run(newCfg *Config, start bool) (Context, error) {
 	// because we will need to roll back any state
 	// modifications if this function errors, we
 	// keep a single error value and scope all
@@ -309,21 +414,10 @@ func run(newCfg *Config, start bool) error {
 	// been set by a short assignment
 	var err error
 
-	// start the admin endpoint (and stop any prior one)
-	if start {
-		err = replaceAdmin(newCfg)
-		if err != nil {
-			return fmt.Errorf("starting caddy administration endpoint: %v", err)
-		}
-	}
-
 	if newCfg == nil {
-		return nil
+		newCfg = new(Config)
 	}
 
-	// prepare the new config for use
-	newCfg.apps = make(map[string]App)
-
 	// create a context within which to load
 	// modules - essentially our new config's
 	// execution environment; be sure that
@@ -341,8 +435,8 @@ func run(newCfg *Config, start bool) error {
 			cancel()
 
 			// also undo any other state changes we made
-			if currentCfg != nil {
+			if currentCtx.cfg != nil {
-				certmagic.Default.Storage = currentCfg.storage
+				certmagic.Default.Storage = currentCtx.cfg.storage
 			}
 		}
 	}()
@@ -354,9 +448,20 @@ func run(newCfg *Config, start bool) error {
 	}
 	err = newCfg.Logging.openLogs(ctx)
 	if err != nil {
-		return err
+		return ctx, err
 	}
 
+	// start the admin endpoint (and stop any prior one)
+	if start {
+		err = replaceLocalAdminServer(newCfg)
+		if err != nil {
+			return ctx, fmt.Errorf("starting caddy administration endpoint: %v", err)
+		}
+	}
+
+	// prepare the new config for use
+	newCfg.apps = make(map[string]App)
+
 	// set up global storage and make it CertMagic's default storage, too
 	err = func() error {
 		if newCfg.StorageRaw != nil {
@@ -372,38 +477,43 @@ func run(newCfg *Config, start bool) error {
 		}
 
 		if newCfg.storage == nil {
-			newCfg.storage = &certmagic.FileStorage{Path: AppDataDir()}
+			newCfg.storage = DefaultStorage
 		}
 		certmagic.Default.Storage = newCfg.storage
 
 		return nil
 	}()
 	if err != nil {
-		return err
+		return ctx, err
 	}
 
-	// Load, Provision, Validate each app and their submodules
+	// Load and Provision each app and their submodules
 	err = func() error {
-		appsIface, err := ctx.LoadModule(newCfg, "AppsRaw")
+		for appName := range newCfg.AppsRaw {
-		if err != nil {
+			if _, err := ctx.App(appName); err != nil {
-			return fmt.Errorf("loading app modules: %v", err)
+				return err
 			}
-		for appName, appIface := range appsIface.(map[string]interface{}) {
-			newCfg.apps[appName] = appIface.(App)
 		}
 		return nil
 	}()
 	if err != nil {
-		return err
+		return ctx, err
 	}
 
 	if !start {
-		return nil
+		return ctx, nil
+	}
+
+	// Provision any admin routers which may need to access
+	// some of the other apps at runtime
+	err = newCfg.Admin.provisionAdminRouters(ctx)
+	if err != nil {
+		return ctx, err
 	}
 
 	// Start
-	return func() error {
+	err = func() error {
-		var started []string
+		started := make([]string, 0, len(newCfg.apps))
 		for name, a := range newCfg.apps {
 			err := a.Start()
 			if err != nil {
@@ -422,6 +532,108 @@ func run(newCfg *Config, start bool) error {
 		}
 		return nil
 	}()
+	if err != nil {
+		return ctx, err
+	}
+
+	// now that the user's config is running, finish setting up anything else,
+	// such as remote admin endpoint, config loader, etc.
+	return ctx, finishSettingUp(ctx, newCfg)
+}
+
+// finishSettingUp should be run after all apps have successfully started.
+func finishSettingUp(ctx Context, cfg *Config) error {
+	// establish this server's identity (only after apps are loaded
+	// so that cert management of this endpoint doesn't prevent user's
+	// servers from starting which likely also use HTTP/HTTPS ports;
+	// but before remote management which may depend on these creds)
+	err := manageIdentity(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("provisioning remote admin endpoint: %v", err)
+	}
+
+	// replace any remote admin endpoint
+	err = replaceRemoteAdminServer(ctx, cfg)
+	if err != nil {
+		return fmt.Errorf("provisioning remote admin endpoint: %v", err)
+	}
+
+	// if dynamic config is requested, set that up and run it
+	if cfg != nil && cfg.Admin != nil && cfg.Admin.Config != nil && cfg.Admin.Config.LoadRaw != nil {
+		val, err := ctx.LoadModule(cfg.Admin.Config, "LoadRaw")
+		if err != nil {
+			return fmt.Errorf("loading config loader module: %s", err)
+		}
+
+		logger := Log().Named("config_loader").With(
+			zap.String("module", val.(Module).CaddyModule().ID.Name()),
+			zap.Int("load_delay", int(cfg.Admin.Config.LoadDelay)))
+
+		runLoadedConfig := func(config []byte) error {
+			logger.Info("applying dynamically-loaded config")
+			err := changeConfig(http.MethodPost, "/"+rawConfigKey, config, "", false)
+			if errors.Is(err, errSameConfig) {
+				return err
+			}
+			if err != nil {
+				logger.Error("failed to run dynamically-loaded config", zap.Error(err))
+				return err
+			}
+			logger.Info("successfully applied dynamically-loaded config")
+			return nil
+		}
+
+		if cfg.Admin.Config.LoadDelay > 0 {
+			go func() {
+				// the loop is here to iterate ONLY if there is an error, a no-op config load,
+				// or an unchanged config; in which case we simply wait the delay and try again
+				for {
+					timer := time.NewTimer(time.Duration(cfg.Admin.Config.LoadDelay))
+					select {
+					case <-timer.C:
+						loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
+						if err != nil {
+							logger.Error("failed loading dynamic config; will retry", zap.Error(err))
+							continue
+						}
+						if loadedConfig == nil {
+							logger.Info("dynamically-loaded config was nil; will retry")
+							continue
+						}
+						err = runLoadedConfig(loadedConfig)
+						if errors.Is(err, errSameConfig) {
+							logger.Info("dynamically-loaded config was unchanged; will retry")
+							continue
+						}
+					case <-ctx.Done():
+						if !timer.Stop() {
+							<-timer.C
+						}
+						logger.Info("stopping dynamic config loading")
+					}
+					break
+				}
+			}()
+		} else {
+			// if no LoadDelay is provided, will load config synchronously
+			loadedConfig, err := val.(ConfigLoader).LoadConfig(ctx)
+			if err != nil {
+				return fmt.Errorf("loading dynamic config from %T: %v", val, err)
+			}
+			// do this in a goroutine so current config can finish being loaded; otherwise deadlock
+			go func() { _ = runLoadedConfig(loadedConfig) }()
+		}
+	}
+
+	return nil
+}
+
+// ConfigLoader is a type that can load a Caddy config. If
+// the return value is non-nil, it must be valid Caddy JSON;
+// if nil or with non-nil error, it is considered to be a
+// no-op load and may be retried later.
+type ConfigLoader interface {
+	LoadConfig(Context) ([]byte, error)
 }
 
 // Stop stops running the current configuration.
@@ -431,29 +643,42 @@ func run(newCfg *Config, start bool) error {
 // stop the others. Stop should only be called
 // if not replacing with a new config.
 func Stop() error {
-	currentCfgMu.Lock()
+	currentCtxMu.RLock()
-	defer currentCfgMu.Unlock()
+	ctx := currentCtx
-	unsyncedStop(currentCfg)
+	currentCtxMu.RUnlock()
-	currentCfg = nil
+
+	rawCfgMu.Lock()
+	unsyncedStop(ctx)
+
+	currentCtxMu.Lock()
+	currentCtx = Context{}
+	currentCtxMu.Unlock()
+
 	rawCfgJSON = nil
 	rawCfgIndex = nil
 	rawCfg[rawConfigKey] = nil
+	rawCfgMu.Unlock()
+
 	return nil
 }
 
-// unsyncedStop stops cfg from running, but has
+// unsyncedStop stops ctx from running, but has
-// no locking around cfg. It is a no-op if cfg is
+// no locking around ctx. It is a no-op if ctx has a
-// nil. If any app returns an error when stopping,
+// nil cfg. If any app returns an error when stopping,
 // it is logged and the function continues stopping
 // the next app. This function assumes all apps in
-// cfg were successfully started first.
+// ctx were successfully started first.
-func unsyncedStop(cfg *Config) {
+//
-	if cfg == nil {
+// A lock on rawCfgMu is required, even though this
+// function does not access rawCfg, that lock
+// synchronizes the stop/start of apps.
+func unsyncedStop(ctx Context) {
+	if ctx.cfg == nil {
 		return
 	}
 
 	// stop each app
-	for name, a := range cfg.apps {
+	for name, a := range ctx.cfg.apps {
 		err := a.Stop()
 		if err != nil {
 			log.Printf("[ERROR] stop %s: %v", name, err)
@@ -461,34 +686,103 @@ func unsyncedStop(cfg *Config) {
 	}
 
 	// clean up all modules
-	cfg.cancelFunc()
+	ctx.cfg.cancelFunc()
-}
-
-// stopAndCleanup calls stop and cleans up anything
-// else that is expedient. This should only be used
-// when stopping and not replacing with a new config.
-func stopAndCleanup() error {
-	if err := Stop(); err != nil {
-		return err
-	}
-	certmagic.CleanUpOwnLocks()
-	return nil
 }
 
 // Validate loads, provisions, and validates
 // cfg, but does not start running it.
 func Validate(cfg *Config) error {
-	err := run(cfg, false)
+	_, err := run(cfg, false)
 	if err == nil {
 		cfg.cancelFunc() // call Cleanup on all modules
 	}
 	return err
 }
 
+// exitProcess exits the process as gracefully as possible,
+// but it always exits, even if there are errors doing so.
+// It stops all apps, cleans up external locks, removes any
+// PID file, and shuts down admin endpoint(s) in a goroutine.
+// Errors are logged along the way, and an appropriate exit
+// code is emitted.
+func exitProcess(ctx context.Context, logger *zap.Logger) {
+	// let the rest of the program know we're quitting
+	atomic.StoreInt32(exiting, 1)
+
+	// give the OS or service/process manager our 2 weeks' notice: we quit
+	if err := notify.Stopping(); err != nil {
+		Log().Error("unable to notify service manager of stopping state", zap.Error(err))
+	}
+
+	if logger == nil {
+		logger = Log()
+	}
+	logger.Warn("exiting; byeee!! 👋")
+
+	exitCode := ExitCodeSuccess
+
+	// stop all apps
+	if err := Stop(); err != nil {
+		logger.Error("failed to stop apps", zap.Error(err))
+		exitCode = ExitCodeFailedQuit
+	}
+
+	// clean up certmagic locks
+	certmagic.CleanUpOwnLocks(ctx, logger)
+
+	// remove pidfile
+	if pidfile != "" {
+		err := os.Remove(pidfile)
+		if err != nil {
+			logger.Error("cleaning up PID file:",
+				zap.String("pidfile", pidfile),
+				zap.Error(err))
+			exitCode = ExitCodeFailedQuit
+		}
+	}
+
+	// shut down admin endpoint(s) in goroutines so that
+	// if this function was called from an admin handler,
+	// it has a chance to return gracefully
+	// use goroutine so that we can finish responding to API request
+	go func() {
+		defer func() {
+			logger = logger.With(zap.Int("exit_code", exitCode))
+			if exitCode == ExitCodeSuccess {
+				logger.Info("shutdown complete")
+			} else {
+				logger.Error("unclean shutdown")
+			}
+			os.Exit(exitCode)
+		}()
+
+		if remoteAdminServer != nil {
+			err := stopAdminServer(remoteAdminServer)
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				logger.Error("failed to stop remote admin server gracefully", zap.Error(err))
+			}
+		}
+		if localAdminServer != nil {
+			err := stopAdminServer(localAdminServer)
+			if err != nil {
+				exitCode = ExitCodeFailedQuit
+				logger.Error("failed to stop local admin server gracefully", zap.Error(err))
+			}
+		}
+	}()
+}
+
+var exiting = new(int32) // accessed atomically
+
+// Exiting returns true if the process is exiting.
+// EXPERIMENTAL API: subject to change or removal.
+func Exiting() bool { return atomic.LoadInt32(exiting) == 1 }
+
 // Duration can be an integer or a string. An integer is
 // interpreted as nanoseconds. If a string, it is a Go
 // time.Duration value such as `300ms`, `1.5h`, or `2h45m`;
-// valid units are `ns`, `us`/`µs`, `ms`, `s`, `m`, and `h`.
+// valid units are `ns`, `us`/`µs`, `ms`, `s`, `m`, `h`, and `d`.
 type Duration time.Duration
 
 // UnmarshalJSON satisfies json.Unmarshaler.
@@ -499,7 +793,7 @@ func (d *Duration) UnmarshalJSON(b []byte) error {
 	var dur time.Duration
 	var err error
 	if b[0] == byte('"') && b[len(b)-1] == byte('"') {
-		dur, err = time.ParseDuration(strings.Trim(string(b), `"`))
+		dur, err = ParseDuration(strings.Trim(string(b), `"`))
 	} else {
 		err = json.Unmarshal(b, &dur)
 	}
@@ -507,36 +801,201 @@ func (d *Duration) UnmarshalJSON(b []byte) error {
 	return err
 }
 
-// GoModule returns the build info of this Caddy
+// ParseDuration parses a duration string, adding
-// build from debug.BuildInfo (requires Go modules).
+// support for the "d" unit meaning number of days,
-// If no version information is available, a non-nil
+// where a day is assumed to be 24h. The maximum
-// value will still be returned, but with an
+// input string length is 1024.
-// unknown version.
+func ParseDuration(s string) (time.Duration, error) {
-func GoModule() *debug.Module {
+	if len(s) > 1024 {
-	var mod debug.Module
+		return 0, fmt.Errorf("parsing duration: input string too long")
-	return goModule(&mod)
+	}
+	var inNumber bool
+	var numStart int
+	for i := 0; i < len(s); i++ {
+		ch := s[i]
+		if ch == 'd' {
+			daysStr := s[numStart:i]
+			days, err := strconv.ParseFloat(daysStr, 64)
+			if err != nil {
+				return 0, err
+			}
+			hours := days * 24.0
+			hoursStr := strconv.FormatFloat(hours, 'f', -1, 64)
+			s = s[:numStart] + hoursStr + "h" + s[i+1:]
+			i--
+			continue
+		}
+		if !inNumber {
+			numStart = i
+		}
+		inNumber = (ch >= '0' && ch <= '9') || ch == '.' || ch == '-' || ch == '+'
+	}
+	return time.ParseDuration(s)
 }
 
-// goModule holds the actual implementation of GoModule.
+// InstanceID returns the UUID for this instance, and generates one if it
-// Allocating debug.Module in GoModule() and passing a
+// does not already exist. The UUID is stored in the local data directory,
-// reference to goModule enables mid-stack inlining.
+// regardless of storage configuration, since each instance is intended to
-func goModule(mod *debug.Module) *debug.Module {
+// have its own unique ID.
-	mod.Version = "unknown"
+func InstanceID() (uuid.UUID, error) {
+	appDataDir := AppDataDir()
+	uuidFilePath := filepath.Join(appDataDir, "instance.uuid")
+	uuidFileBytes, err := os.ReadFile(uuidFilePath)
+	if os.IsNotExist(err) {
+		uuid, err := uuid.NewRandom()
+		if err != nil {
+			return uuid, err
+		}
+		err = os.MkdirAll(appDataDir, 0o600)
+		if err != nil {
+			return uuid, err
+		}
+		err = os.WriteFile(uuidFilePath, []byte(uuid.String()), 0o600)
+		return uuid, err
+	} else if err != nil {
+		return [16]byte{}, err
+	}
+	return uuid.ParseBytes(uuidFileBytes)
+}
+
+// CustomVersion is an optional string that overrides Caddy's
+// reported version. It can be helpful when downstream packagers
+// need to manually set Caddy's version. If no other version
+// information is available, the short form version (see
+// Version()) will be set to CustomVersion, and the full version
+// will include CustomVersion at the beginning.
+//
+// Set this variable during `go build` with `-ldflags`:
+//
+//	-ldflags '-X github.com/caddyserver/caddy/v2.CustomVersion=v2.6.2'
+//
+// for example.
+var CustomVersion string
+
+// Version returns the Caddy version in a simple/short form, and
+// a full version string. The short form will not have spaces and
+// is intended for User-Agent strings and similar, but may be
+// omitting valuable information. Note that Caddy must be compiled
+// in a special way to properly embed complete version information.
+// First this function tries to get the version from the embedded
+// build info provided by go.mod dependencies; then it tries to
+// get info from embedded VCS information, which requires having
+// built Caddy from a git repository. If no version is available,
+// this function returns "(devel)" because Go uses that, but for
+// the simple form we change it to "unknown". If still no version
+// is available (e.g. no VCS repo), then it will use CustomVersion;
+// CustomVersion is always prepended to the full version string.
+//
+// See relevant Go issues: https://github.com/golang/go/issues/29228
+// and https://github.com/golang/go/issues/50603.
+//
+// This function is experimental and subject to change or removal.
+func Version() (simple, full string) {
+	// the currently-recommended way to build Caddy involves
+	// building it as a dependency so we can extract version
+	// information from go.mod tooling; once the upstream
+	// Go issues are fixed, we should just be able to use
+	// bi.Main... hopefully.
+	var module *debug.Module
 	bi, ok := debug.ReadBuildInfo()
-	if ok {
+	if !ok {
-		mod.Path = bi.Main.Path
+		if CustomVersion != "" {
-		// The recommended way to build Caddy involves
+			full = CustomVersion
-		// creating a separate main module, which
+			simple = CustomVersion
-		// TODO: track related Go issue: https://github.com/golang/go/issues/29228
+			return
-		// once that issue is fixed, we should just be able to use bi.Main... hopefully.
+		}
+		full = "unknown"
+		simple = "unknown"
+		return
+	}
+	// find the Caddy module in the dependency list
 	for _, dep := range bi.Deps {
 		if dep.Path == ImportPath {
-				return dep
+			module = dep
+			break
 		}
 	}
-		return &bi.Main
+	if module != nil {
+		simple, full = module.Version, module.Version
+		if module.Sum != "" {
+			full += " " + module.Sum
 		}
-	return mod
+		if module.Replace != nil {
+			full += " => " + module.Replace.Path
+			if module.Replace.Version != "" {
+				simple = module.Replace.Version + "_custom"
+				full += "@" + module.Replace.Version
+			}
+			if module.Replace.Sum != "" {
+				full += " " + module.Replace.Sum
+			}
+		}
+	}
+
+	if full == "" {
+		var vcsRevision string
+		var vcsTime time.Time
+		var vcsModified bool
+		for _, setting := range bi.Settings {
+			switch setting.Key {
+			case "vcs.revision":
+				vcsRevision = setting.Value
+			case "vcs.time":
+				vcsTime, _ = time.Parse(time.RFC3339, setting.Value)
+			case "vcs.modified":
+				vcsModified, _ = strconv.ParseBool(setting.Value)
+			}
+		}
+
+		if vcsRevision != "" {
+			var modified string
+			if vcsModified {
+				modified = "+modified"
+			}
+			full = fmt.Sprintf("%s%s (%s)", vcsRevision, modified, vcsTime.Format(time.RFC822))
+			simple = vcsRevision
+
+			// use short checksum for simple, if hex-only
+			if _, err := hex.DecodeString(simple); err == nil {
+				simple = simple[:8]
+			}
+
+			// append date to simple since it can be convenient
+			// to know the commit date as part of the version
+			if !vcsTime.IsZero() {
+				simple += "-" + vcsTime.Format("20060102")
+			}
+		}
+	}
+
+	if full == "" {
+		if CustomVersion != "" {
+			full = CustomVersion
+		} else {
+			full = "unknown"
+		}
+	} else if CustomVersion != "" {
+		full = CustomVersion + " " + full
+	}
+
+	if simple == "" || simple == "(devel)" {
+		if CustomVersion != "" {
+			simple = CustomVersion
+		} else {
+			simple = "unknown"
+		}
+	}
+
+	return
+}
+
+// ActiveContext returns the currently-active context.
+// This function is experimental and might be changed
+// or removed in the future.
+func ActiveContext() Context {
+	currentCtxMu.RLock()
+	defer currentCtxMu.RUnlock()
+	return currentCtx
 }
 
 // CtxKey is a value type for use with context.WithValue.
@@ -544,18 +1003,19 @@ type CtxKey string
 
 // This group of variables pertains to the current configuration.
 var (
-	// currentCfgMu protects everything in this var block.
+	// currentCtx is the root context for the currently-running
-	currentCfgMu sync.RWMutex
+	// configuration, which can be accessed through this value.
-
+	// If the Config contained in this value is not nil, then
-	// currentCfg is the currently-running configuration.
+	// a config is currently active/running.
-	currentCfg *Config
+	currentCtx   Context
+	currentCtxMu sync.RWMutex
 
 	// rawCfg is the current, generic-decoded configuration;
 	// we initialize it as a map with one field ("config")
 	// to maintain parity with the API endpoint and to avoid
 	// the special case of having to access/mutate the variable
 	// directly without traversing into it.
-	rawCfg = map[string]interface{}{
+	rawCfg = map[string]any{
 		rawConfigKey: nil,
 	}
 
@@ -566,7 +1026,17 @@ var (
 	// rawCfgIndex is the map of user-assigned ID to expanded
 	// path, for converting /id/ paths to /config/ paths.
 	rawCfgIndex map[string]string
+
+	// rawCfgMu protects all the rawCfg fields and also
+	// essentially synchronizes config changes/reloads.
+	rawCfgMu sync.RWMutex
 )
 
+// errSameConfig is returned if the new config is the same
+// as the old one. This isn't usually an actual, actionable
+// error; it's mostly a sentinel value.
+var errSameConfig = errors.New("config is unchanged")
+
 // ImportPath is the package import path for Caddy core.
+// This identifier may be removed in the future.
 const ImportPath = "github.com/caddyserver/caddy/v2"

caddy_test.go

@@ -0,0 +1,74 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddy
+
+import (
+	"testing"
+	"time"
+)
+
+func TestParseDuration(t *testing.T) {
+	const day = 24 * time.Hour
+	for i, tc := range []struct {
+		input  string
+		expect time.Duration
+	}{
+		{
+			input:  "3h",
+			expect: 3 * time.Hour,
+		},
+		{
+			input:  "1d",
+			expect: day,
+		},
+		{
+			input:  "1d30m",
+			expect: day + 30*time.Minute,
+		},
+		{
+			input:  "1m2d",
+			expect: time.Minute + day*2,
+		},
+		{
+			input:  "1m2d30s",
+			expect: time.Minute + day*2 + 30*time.Second,
+		},
+		{
+			input:  "1d2d",
+			expect: 3 * day,
+		},
+		{
+			input:  "1.5d",
+			expect: time.Duration(1.5 * float64(day)),
+		},
+		{
+			input:  "4m1.25d",
+			expect: 4*time.Minute + time.Duration(1.25*float64(day)),
+		},
+		{
+			input:  "-1.25d12h",
+			expect: time.Duration(-1.25*float64(day)) - 12*time.Hour,
+		},
+	} {
+		actual, err := ParseDuration(tc.input)
+		if err != nil {
+			t.Errorf("Test %d ('%s'): Got error: %v", i, tc.input, err)
+			continue
+		}
+		if actual != tc.expect {
+			t.Errorf("Test %d ('%s'): Expected=%s Actual=%s", i, tc.input, tc.expect, actual)
+		}
+	}
+}

caddyconfig/caddyfile/adapter.go

@@ -15,6 +15,7 @@
 package caddyfile
 
 import (
+	"bytes"
 	"encoding/json"
 	"fmt"
 
@@ -28,12 +29,12 @@ type Adapter struct {
 }
 
 // Adapt converts the Caddyfile config in body to Caddy JSON.
-func (a Adapter) Adapt(body []byte, options map[string]interface{}) ([]byte, []caddyconfig.Warning, error) {
+func (a Adapter) Adapt(body []byte, options map[string]any) ([]byte, []caddyconfig.Warning, error) {
 	if a.ServerType == nil {
 		return nil, nil, fmt.Errorf("no server type")
 	}
 	if options == nil {
-		options = make(map[string]interface{})
+		options = make(map[string]any)
 	}
 
 	filename, _ := options["filename"].(string)
@@ -51,15 +52,46 @@ func (a Adapter) Adapt(body []byte, options map[string]interface{}) ([]byte, []c
 		return nil, warnings, err
 	}
 
-	marshalFunc := json.Marshal
+	// lint check: see if input was properly formatted; sometimes messy files files parse
-	if options["pretty"] == "true" {
+	// successfully but result in logical errors (the Caddyfile is a bad format, I'm sorry)
-		marshalFunc = caddyconfig.JSONIndent
+	if warning, different := FormattingDifference(filename, body); different {
+		warnings = append(warnings, warning)
 	}
-	result, err := marshalFunc(cfg)
+
+	result, err := json.Marshal(cfg)
 
 	return result, warnings, err
 }
 
+// FormattingDifference returns a warning and true if the formatted version
+// is any different from the input; empty warning and false otherwise.
+// TODO: also perform this check on imported files
+func FormattingDifference(filename string, body []byte) (caddyconfig.Warning, bool) {
+	// replace windows-style newlines to normalize comparison
+	normalizedBody := bytes.Replace(body, []byte("\r\n"), []byte("\n"), -1)
+
+	formatted := Format(normalizedBody)
+	if bytes.Equal(formatted, normalizedBody) {
+		return caddyconfig.Warning{}, false
+	}
+
+	// find where the difference is
+	line := 1
+	for i, ch := range normalizedBody {
+		if i >= len(formatted) || ch != formatted[i] {
+			break
+		}
+		if ch == '\n' {
+			line++
+		}
+	}
+	return caddyconfig.Warning{
+		File:    filename,
+		Line:    line,
+		Message: "Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies",
+	}, true
+}
+
 // Unmarshaler is a type that can unmarshal
 // Caddyfile tokens to set itself up for a
 // JSON encoding. The goal of an unmarshaler
@@ -68,6 +100,11 @@ func (a Adapter) Adapt(body []byte, options map[string]interface{}) ([]byte, []c
 // into JSON. Caddyfile-unmarshaled values
 // will not be used directly; they will be
 // encoded as JSON and then used from that.
+// Implementations must be able to support
+// multiple segments (instances of their
+// directive or batch of tokens); typically
+// this means wrapping all token logic in
+// a loop: `for d.Next() { ... }`.
 type Unmarshaler interface {
 	UnmarshalCaddyfile(d *Dispenser) error
 }
@@ -79,7 +116,33 @@ type ServerType interface {
 	// (e.g. CLI flags) and creates a Caddy
 	// config, along with any warnings or
 	// an error.
-	Setup([]ServerBlock, map[string]interface{}) (*caddy.Config, []caddyconfig.Warning, error)
+	Setup([]ServerBlock, map[string]any) (*caddy.Config, []caddyconfig.Warning, error)
+}
+
+// UnmarshalModule instantiates a module with the given ID and invokes
+// UnmarshalCaddyfile on the new value using the immediate next segment
+// of d as input. In other words, d's next token should be the first
+// token of the module's Caddyfile input.
+//
+// This function is used when the next segment of Caddyfile tokens
+// belongs to another Caddy module. The returned value is often
+// type-asserted to the module's associated type for practical use
+// when setting up a config.
+func UnmarshalModule(d *Dispenser, moduleID string) (Unmarshaler, error) {
+	mod, err := caddy.GetModule(moduleID)
+	if err != nil {
+		return nil, d.Errf("getting module named '%s': %v", moduleID, err)
+	}
+	inst := mod.New()
+	unm, ok := inst.(Unmarshaler)
+	if !ok {
+		return nil, d.Errf("module %s is not a Caddyfile unmarshaler; is %T", mod.ID, inst)
+	}
+	err = unm.UnmarshalCaddyfile(d.NewFromNextSegment())
+	if err != nil {
+		return nil, err
+	}
+	return unm, nil
 }
 
 // Interface guard

caddyconfig/caddyfile/dispenser.go

@@ -17,6 +17,9 @@ package caddyfile
 import (
 	"errors"
 	"fmt"
+	"io"
+	"log"
+	"strconv"
 	"strings"
 )
 
@@ -37,6 +40,16 @@ func NewDispenser(tokens []Token) *Dispenser {
 	}
 }
 
+// NewTestDispenser parses input into tokens and creates a new
+// Dispenser for test purposes only; any errors are fatal.
+func NewTestDispenser(input string) *Dispenser {
+	tokens, err := allTokens("Testfile", []byte(input))
+	if err != nil && err != io.EOF {
+		log.Fatalf("getting all tokens from input: %v", err)
+	}
+	return NewDispenser(tokens)
+}
+
 // Next loads the next token. Returns true if a token
 // was loaded; false otherwise. If false, all tokens
 // have been consumed.
@@ -88,12 +101,12 @@ func (d *Dispenser) nextOnSameLine() bool {
 		d.cursor++
 		return true
 	}
-	if d.cursor >= len(d.tokens) {
+	if d.cursor >= len(d.tokens)-1 {
 		return false
 	}
-	if d.cursor < len(d.tokens)-1 &&
+	curr := d.tokens[d.cursor]
-		d.tokens[d.cursor].File == d.tokens[d.cursor+1].File &&
+	next := d.tokens[d.cursor+1]
-		d.tokens[d.cursor].Line+d.numLineBreaks(d.cursor) == d.tokens[d.cursor+1].Line {
+	if !isNextOnNewLine(curr, next) {
 		d.cursor++
 		return true
 	}
@@ -109,12 +122,12 @@ func (d *Dispenser) NextLine() bool {
 		d.cursor++
 		return true
 	}
-	if d.cursor >= len(d.tokens) {
+	if d.cursor >= len(d.tokens)-1 {
 		return false
 	}
-	if d.cursor < len(d.tokens)-1 &&
+	curr := d.tokens[d.cursor]
-		(d.tokens[d.cursor].File != d.tokens[d.cursor+1].File ||
+	next := d.tokens[d.cursor+1]
-			d.tokens[d.cursor].Line+d.numLineBreaks(d.cursor) < d.tokens[d.cursor+1].Line) {
+	if isNextOnNewLine(curr, next) {
 		d.cursor++
 		return true
 	}
@@ -189,6 +202,46 @@ func (d *Dispenser) Val() string {
 	return d.tokens[d.cursor].Text
 }
 
+// ValRaw gets the raw text of the current token (including quotes).
+// If the token was a heredoc, then the delimiter is not included,
+// because that is not relevant to any unmarshaling logic at this time.
+// If there is no token loaded, it returns empty string.
+func (d *Dispenser) ValRaw() string {
+	if d.cursor < 0 || d.cursor >= len(d.tokens) {
+		return ""
+	}
+	quote := d.tokens[d.cursor].wasQuoted
+	if quote > 0 && quote != '<' {
+		// string literal
+		return string(quote) + d.tokens[d.cursor].Text + string(quote)
+	}
+	return d.tokens[d.cursor].Text
+}
+
+// ScalarVal gets value of the current token, converted to the closest
+// scalar type. If there is no token loaded, it returns nil.
+func (d *Dispenser) ScalarVal() any {
+	if d.cursor < 0 || d.cursor >= len(d.tokens) {
+		return nil
+	}
+	quote := d.tokens[d.cursor].wasQuoted
+	text := d.tokens[d.cursor].Text
+
+	if quote > 0 {
+		return text // string literal
+	}
+	if num, err := strconv.Atoi(text); err == nil {
+		return num
+	}
+	if num, err := strconv.ParseFloat(text, 64); err == nil {
+		return num
+	}
+	if bool, err := strconv.ParseBool(text); err == nil {
+		return bool
+	}
+	return text
+}
+
 // Line gets the line number of the current token.
 // If there is no token loaded, it returns 0.
 func (d *Dispenser) Line() int {
@@ -237,6 +290,19 @@ func (d *Dispenser) AllArgs(targets ...*string) bool {
 	return true
 }
 
+// CountRemainingArgs counts the amount of remaining arguments
+// (tokens on the same line) without consuming the tokens.
+func (d *Dispenser) CountRemainingArgs() int {
+	count := 0
+	for d.NextArg() {
+		count++
+	}
+	for i := 0; i < count; i++ {
+		d.Prev()
+	}
+	return count
+}
+
 // RemainingArgs loads any more arguments (tokens on the same line)
 // into a slice and returns them. Open curly brace tokens also indicate
 // the end of arguments, and the curly brace is not included in
@@ -249,13 +315,32 @@ func (d *Dispenser) RemainingArgs() []string {
 	return args
 }
 
-// NewFromNextTokens returns a new dispenser with a copy of
+// RemainingArgsRaw loads any more arguments (tokens on the same line,
+// retaining quotes) into a slice and returns them. Open curly brace
+// tokens also indicate the end of arguments, and the curly brace is
+// not included in the return value nor is it loaded.
+func (d *Dispenser) RemainingArgsRaw() []string {
+	var args []string
+	for d.NextArg() {
+		args = append(args, d.ValRaw())
+	}
+	return args
+}
+
+// NewFromNextSegment returns a new dispenser with a copy of
 // the tokens from the current token until the end of the
 // "directive" whether that be to the end of the line or
 // the end of a block that starts at the end of the line;
 // in other words, until the end of the segment.
-func (d *Dispenser) NewFromNextTokens() *Dispenser {
+func (d *Dispenser) NewFromNextSegment() *Dispenser {
-	tkns := []Token{d.Token()}
+	return NewDispenser(d.NextSegment())
+}
+
+// NextSegment returns a copy of the tokens from the current
+// token until the end of the line or block that starts at
+// the end of the line.
+func (d *Dispenser) NextSegment() Segment {
+	tkns := Segment{d.Token()}
 	for d.NextArg() {
 		tkns = append(tkns, d.Token())
 	}
@@ -282,7 +367,7 @@ func (d *Dispenser) NewFromNextTokens() *Dispenser {
 		// next iteration of the enclosing loop will
 		// call Next() and consume it
 	}
-	return NewDispenser(tkns)
+	return tkns
 }
 
 // Token returns the current token.
@@ -306,33 +391,40 @@ func (d *Dispenser) Reset() {
 // an argument.
 func (d *Dispenser) ArgErr() error {
 	if d.Val() == "{" {
-		return d.Err("Unexpected token '{', expecting argument")
+		return d.Err("unexpected token '{', expecting argument")
 	}
-	return d.Errf("Wrong argument count or unexpected line ending after '%s'", d.Val())
+	return d.Errf("wrong argument count or unexpected line ending after '%s'", d.Val())
 }
 
 // SyntaxErr creates a generic syntax error which explains what was
 // found and what was expected.
 func (d *Dispenser) SyntaxErr(expected string) error {
-	msg := fmt.Sprintf("%s:%d - Syntax error: Unexpected token '%s', expecting '%s'", d.File(), d.Line(), d.Val(), expected)
+	msg := fmt.Sprintf("syntax error: unexpected token '%s', expecting '%s', at %s:%d import chain: ['%s']", d.Val(), expected, d.File(), d.Line(), strings.Join(d.Token().imports, "','"))
 	return errors.New(msg)
 }
 
 // EOFErr returns an error indicating that the dispenser reached
 // the end of the input when searching for the next token.
 func (d *Dispenser) EOFErr() error {
-	return d.Errf("Unexpected EOF")
+	return d.Errf("unexpected EOF")
 }
 
 // Err generates a custom parse-time error with a message of msg.
 func (d *Dispenser) Err(msg string) error {
-	msg = fmt.Sprintf("%s:%d - Error during parsing: %s", d.File(), d.Line(), msg)
+	return d.Errf(msg)
-	return errors.New(msg)
 }
 
 // Errf is like Err, but for formatted error messages
-func (d *Dispenser) Errf(format string, args ...interface{}) error {
+func (d *Dispenser) Errf(format string, args ...any) error {
-	return d.Err(fmt.Sprintf(format, args...))
+	return d.WrapErr(fmt.Errorf(format, args...))
+}
+
+// WrapErr takes an existing error and adds the Caddyfile file and line number.
+func (d *Dispenser) WrapErr(err error) error {
+	if len(d.Token().imports) > 0 {
+		return fmt.Errorf("%w, at %s:%d import chain ['%s']", err, d.File(), d.Line(), strings.Join(d.Token().imports, "','"))
+	}
+	return fmt.Errorf("%w, at %s:%d", err, d.File(), d.Line())
 }
 
 // Delete deletes the current token and returns the updated slice
@@ -352,14 +444,14 @@ func (d *Dispenser) Delete() []Token {
 	return d.tokens
 }
 
-// numLineBreaks counts how many line breaks are in the token
+// DeleteN is the same as Delete, but can delete many tokens at once.
-// value given by the token index tknIdx. It returns 0 if the
+// If there aren't N tokens available to delete, none are deleted.
-// token does not exist or there are no line breaks.
+func (d *Dispenser) DeleteN(amount int) []Token {
-func (d *Dispenser) numLineBreaks(tknIdx int) int {
+	if amount > 0 && d.cursor >= (amount-1) && d.cursor <= len(d.tokens)-1 {
-	if tknIdx < 0 || tknIdx >= len(d.tokens) {
+		d.tokens = append(d.tokens[:d.cursor-(amount-1)], d.tokens[d.cursor+1:]...)
-		return 0
+		d.cursor -= amount
 	}
-	return strings.Count(d.tokens[tknIdx].Text, "\n")
+	return d.tokens
 }
 
 // isNewLine determines whether the current token is on a different
@@ -372,6 +464,24 @@ func (d *Dispenser) isNewLine() bool {
 	if d.cursor > len(d.tokens)-1 {
 		return false
 	}
-	return d.tokens[d.cursor-1].File != d.tokens[d.cursor].File ||
+
-		d.tokens[d.cursor-1].Line+d.numLineBreaks(d.cursor-1) < d.tokens[d.cursor].Line
+	prev := d.tokens[d.cursor-1]
+	curr := d.tokens[d.cursor]
+	return isNextOnNewLine(prev, curr)
+}
+
+// isNextOnNewLine determines whether the current token is on a different
+// line (higher line number) than the next token. It handles imported
+// tokens correctly. If there isn't a next token, it returns true.
+func (d *Dispenser) isNextOnNewLine() bool {
+	if d.cursor < 0 {
+		return false
+	}
+	if d.cursor >= len(d.tokens)-1 {
+		return true
+	}
+
+	curr := d.tokens[d.cursor]
+	next := d.tokens[d.cursor+1]
+	return isNextOnNewLine(curr, next)
 }

caddyconfig/caddyfile/dispenser_test.go

@@ -15,8 +15,7 @@
 package caddyfile
 
 import (
-	"io"
+	"errors"
-	"log"
 	"reflect"
 	"strings"
 	"testing"
@@ -27,7 +26,7 @@ func TestDispenser_Val_Next(t *testing.T) {
 			  dir1 arg1
 			  dir2 arg2 arg3
 			  dir3`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	if val := d.Val(); val != "" {
 		t.Fatalf("Val(): Should return empty string when no token loaded; got '%s'", val)
@@ -65,7 +64,7 @@ func TestDispenser_NextArg(t *testing.T) {
 	input := `dir1 arg1
 			  dir2 arg2 arg3
 			  dir3`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	assertNext := func(shouldLoad bool, expectedVal string, expectedCursor int) {
 		if d.Next() != shouldLoad {
@@ -112,7 +111,7 @@ func TestDispenser_NextLine(t *testing.T) {
 	input := `host:port
 			  dir1 arg1
 			  dir2 arg2 arg3`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	assertNextLine := func(shouldLoad bool, expectedVal string, expectedCursor int) {
 		if d.NextLine() != shouldLoad {
@@ -145,7 +144,7 @@ func TestDispenser_NextBlock(t *testing.T) {
 			  }
 			  foobar2 {
 			  }`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	assertNextBlock := func(shouldLoad bool, expectedCursor, expectedNesting int) {
 		if loaded := d.NextBlock(0); loaded != shouldLoad {
@@ -175,7 +174,7 @@ func TestDispenser_Args(t *testing.T) {
 			  dir2 arg4 arg5
 			  dir3 arg6 arg7
 			  dir4`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	d.Next() // dir1
 
@@ -242,7 +241,7 @@ func TestDispenser_RemainingArgs(t *testing.T) {
 			  dir2 arg4 arg5
 			  dir3 arg6 { arg7
 			  dir4`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	d.Next() // dir1
 
@@ -279,7 +278,7 @@ func TestDispenser_ArgErr_Err(t *testing.T) {
 	input := `dir1 {
 			  }
 			  dir2 arg1 arg2`
-	d := newTestDispenser(input)
+	d := NewTestDispenser(input)
 
 	d.cursor = 1 // {
 
@@ -305,12 +304,10 @@ func TestDispenser_ArgErr_Err(t *testing.T) {
 	if !strings.Contains(err.Error(), "foobar") {
 		t.Errorf("Expected error message with custom message in it ('foobar'); got '%v'", err)
 	}
-}
 
-func newTestDispenser(input string) *Dispenser {
+	var ErrBarIsFull = errors.New("bar is full")
-	tokens, err := allTokens("Testfile", []byte(input))
+	bookingError := d.Errf("unable to reserve: %w", ErrBarIsFull)
-	if err != nil && err != io.EOF {
+	if !errors.Is(bookingError, ErrBarIsFull) {
-		log.Fatalf("getting all tokens from input: %v", err)
+		t.Errorf("Errf(): should be able to unwrap the error chain")
 	}
-	return NewDispenser(tokens)
 }

caddyconfig/caddyfile/formatter.go

@@ -0,0 +1,219 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"bytes"
+	"io"
+	"unicode"
+)
+
+// Format formats the input Caddyfile to a standard, nice-looking
+// appearance. It works by reading each rune of the input and taking
+// control over all the bracing and whitespace that is written; otherwise,
+// words, comments, placeholders, and escaped characters are all treated
+// literally and written as they appear in the input.
+func Format(input []byte) []byte {
+	input = bytes.TrimSpace(input)
+
+	out := new(bytes.Buffer)
+	rdr := bytes.NewReader(input)
+
+	var (
+		last rune // the last character that was written to the result
+
+		space           = true // whether current/previous character was whitespace (beginning of input counts as space)
+		beginningOfLine = true // whether we are at beginning of line
+
+		openBrace        bool // whether current word/token is or started with open curly brace
+		openBraceWritten bool // if openBrace, whether that brace was written or not
+		openBraceSpace   bool // whether there was a non-newline space before open brace
+
+		newLines int // count of newlines consumed
+
+		comment bool // whether we're in a comment
+		quoted  bool // whether we're in a quoted segment
+		escaped bool // whether current char is escaped
+
+		nesting int // indentation level
+	)
+
+	write := func(ch rune) {
+		out.WriteRune(ch)
+		last = ch
+	}
+
+	indent := func() {
+		for tabs := nesting; tabs > 0; tabs-- {
+			write('\t')
+		}
+	}
+
+	nextLine := func() {
+		write('\n')
+		beginningOfLine = true
+	}
+
+	for {
+		ch, _, err := rdr.ReadRune()
+		if err != nil {
+			if err == io.EOF {
+				break
+			}
+			panic(err)
+		}
+
+		if comment {
+			if ch == '\n' {
+				comment = false
+				space = true
+				nextLine()
+				continue
+			} else {
+				write(ch)
+				continue
+			}
+		}
+
+		if !escaped && ch == '\\' {
+			if space {
+				write(' ')
+				space = false
+			}
+			write(ch)
+			escaped = true
+			continue
+		}
+
+		if escaped {
+			write(ch)
+			escaped = false
+			continue
+		}
+
+		if quoted {
+			if ch == '"' {
+				quoted = false
+			}
+			write(ch)
+			continue
+		}
+
+		if space && ch == '"' {
+			quoted = true
+		}
+
+		if unicode.IsSpace(ch) {
+			space = true
+			if ch == '\n' {
+				newLines++
+			}
+			continue
+		}
+		spacePrior := space
+		space = false
+
+		//////////////////////////////////////////////////////////
+		// I find it helpful to think of the formatting loop in two
+		// main sections; by the time we reach this point, we
+		// know we are in a "regular" part of the file: we know
+		// the character is not a space, not in a literal segment
+		// like a comment or quoted, it's not escaped, etc.
+		//////////////////////////////////////////////////////////
+
+		if ch == '#' {
+			comment = true
+		}
+
+		if openBrace && spacePrior && !openBraceWritten {
+			if nesting == 0 && last == '}' {
+				nextLine()
+				nextLine()
+			}
+
+			openBrace = false
+			if beginningOfLine {
+				indent()
+			} else if !openBraceSpace {
+				write(' ')
+			}
+			write('{')
+			openBraceWritten = true
+			nextLine()
+			newLines = 0
+			// prevent infinite nesting from ridiculous inputs (issue #4169)
+			if nesting < 10 {
+				nesting++
+			}
+		}
+
+		switch {
+		case ch == '{':
+			openBrace = true
+			openBraceWritten = false
+			openBraceSpace = spacePrior && !beginningOfLine
+			if openBraceSpace {
+				write(' ')
+			}
+			continue
+
+		case ch == '}' && (spacePrior || !openBrace):
+			if last != '\n' {
+				nextLine()
+			}
+			if nesting > 0 {
+				nesting--
+			}
+			indent()
+			write('}')
+			newLines = 0
+			continue
+		}
+
+		if newLines > 2 {
+			newLines = 2
+		}
+		for i := 0; i < newLines; i++ {
+			nextLine()
+		}
+		newLines = 0
+		if beginningOfLine {
+			indent()
+		}
+		if nesting == 0 && last == '}' && beginningOfLine {
+			nextLine()
+			nextLine()
+		}
+
+		if !beginningOfLine && spacePrior {
+			write(' ')
+		}
+
+		if openBrace && !openBraceWritten {
+			write('{')
+			openBraceWritten = true
+		}
+		write(ch)
+
+		beginningOfLine = false
+	}
+
+	// the Caddyfile does not need any leading or trailing spaces, but...
+	trimmedResult := bytes.TrimSpace(out.Bytes())
+
+	// ...Caddyfiles should, however, end with a newline because
+	// newlines are significant to the syntax of the file
+	return append(trimmedResult, '\n')
+}

caddyconfig/caddyfile/formatter_fuzz.go

@@ -0,0 +1,27 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build gofuzz
+
+package caddyfile
+
+import "bytes"
+
+func FuzzFormat(input []byte) int {
+	formatted := Format(input)
+	if bytes.Equal(formatted, Format(formatted)) {
+		return 1
+	}
+	return 0
+}

caddyconfig/caddyfile/formatter_test.go

@@ -0,0 +1,381 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestFormatter(t *testing.T) {
+	for i, tc := range []struct {
+		description string
+		input       string
+		expect      string
+	}{
+		{
+			description: "very simple",
+			input: `abc   def
+	g hi jkl
+mn`,
+			expect: `abc def
+g hi jkl
+mn`,
+		},
+		{
+			description: "basic indentation, line breaks, and nesting",
+			input: `  a
+b
+
+	c {
+		d
+}
+
+e { f
+}
+
+
+
+g {
+h {
+i
+}
+}
+
+j { k {
+l
+}
+}
+
+m {
+	n { o
+	}
+	p { q r
+s }
+}
+
+	{
+{ t
+		u
+
+	v
+
+w
+}
+}`,
+			expect: `a
+b
+
+c {
+	d
+}
+
+e {
+	f
+}
+
+g {
+	h {
+		i
+	}
+}
+
+j {
+	k {
+		l
+	}
+}
+
+m {
+	n {
+		o
+	}
+	p {
+		q r
+		s
+	}
+}
+
+{
+	{
+		t
+		u
+
+		v
+
+		w
+	}
+}`,
+		},
+		{
+			description: "block spacing",
+			input: `a{
+	b
+}
+
+c{ d
+}`,
+			expect: `a {
+	b
+}
+
+c {
+	d
+}`,
+		},
+		{
+			description: "advanced spacing",
+			input: `abc {
+	def
+}ghi{
+	jkl mno
+pqr}`,
+			expect: `abc {
+	def
+}
+
+ghi {
+	jkl mno
+	pqr
+}`,
+		},
+		{
+			description: "env var placeholders",
+			input: `{$A}
+
+b {
+{$C}
+}
+
+d { {$E}
+}
+
+{ {$F}
+}
+`,
+			expect: `{$A}
+
+b {
+	{$C}
+}
+
+d {
+	{$E}
+}
+
+{
+	{$F}
+}`,
+		},
+		{
+			description: "env var placeholders with port",
+			input:       `:{$PORT}`,
+			expect:      `:{$PORT}`,
+		},
+		{
+			description: "comments",
+			input: `#a "\n"
+
+ #b {
+	c
+}
+
+d {
+e#f
+# g
+}
+
+h { # i
+}`,
+			expect: `#a "\n"
+
+#b {
+c
+}
+
+d {
+	e#f
+	# g
+}
+
+h {
+	# i
+}`,
+		},
+		{
+			description: "quotes and escaping",
+			input: `"a \"b\" "#c
+	d
+
+e {
+"f"
+}
+
+g { "h"
+}
+
+i {
+	"foo
+bar"
+}
+
+j {
+"\"k\" l m"
+}`,
+			expect: `"a \"b\" "#c
+d
+
+e {
+	"f"
+}
+
+g {
+	"h"
+}
+
+i {
+	"foo
+bar"
+}
+
+j {
+	"\"k\" l m"
+}`,
+		},
+		{
+			description: "bad nesting (too many open)",
+			input: `a
+{
+	{
+}`,
+			expect: `a {
+	{
+	}
+`,
+		},
+		{
+			description: "bad nesting (too many close)",
+			input: `a
+{
+	{
+}}}`,
+			expect: `a {
+	{
+	}
+}
+}
+`,
+		},
+		{
+			description: "json",
+			input: `foo
+bar      "{\"key\":34}"
+`,
+			expect: `foo
+bar "{\"key\":34}"`,
+		},
+		{
+			description: "escaping after spaces",
+			input:       `foo \"literal\"`,
+			expect:      `foo \"literal\"`,
+		},
+		{
+			description: "simple placeholders as standalone tokens",
+			input:       `foo {bar}`,
+			expect:      `foo {bar}`,
+		},
+		{
+			description: "simple placeholders within tokens",
+			input:       `foo{bar} foo{bar}baz`,
+			expect:      `foo{bar} foo{bar}baz`,
+		},
+		{
+			description: "placeholders and malformed braces",
+			input:       `foo{bar} foo{ bar}baz`,
+			expect: `foo{bar} foo {
+	bar
+}
+
+baz`,
+		},
+		{
+			description: "hash within string is not a comment",
+			input:       `redir / /some/#/path`,
+			expect:      `redir / /some/#/path`,
+		},
+		{
+			description: "brace does not fold into comment above",
+			input: `# comment
+{
+	foo
+}`,
+			expect: `# comment
+{
+	foo
+}`,
+		},
+		{
+			description: "matthewpi/vscode-caddyfile-support#13",
+			input: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+			expect: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+		},
+		{
+			description: "matthewpi/vscode-caddyfile-support#13 - bad formatting",
+			input: `{
+	email {$ACMEEMAIL}
+	#debug
+	}
+
+	block {
+	}
+`,
+			expect: `{
+	email {$ACMEEMAIL}
+	#debug
+}
+
+block {
+}
+`,
+		},
+	} {
+		// the formatter should output a trailing newline,
+		// even if the tests aren't written to expect that
+		if !strings.HasSuffix(tc.expect, "\n") {
+			tc.expect += "\n"
+		}
+
+		actual := Format([]byte(tc.input))
+
+		if string(actual) != tc.expect {
+			t.Errorf("\n[TEST %d: %s]\n====== EXPECTED ======\n%s\n====== ACTUAL ======\n%s^^^^^^^^^^^^^^^^^^^^^",
+				i, tc.description, string(tc.expect), string(actual))
+		}
+	}
+}

caddyconfig/caddyfile/importargs.go

@@ -0,0 +1,160 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"regexp"
+	"strconv"
+	"strings"
+
+	"go.uber.org/zap"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+// parseVariadic determines if the token is a variadic placeholder,
+// and if so, determines the index range (start/end) of args to use.
+// Returns a boolean signaling whether a variadic placeholder was found,
+// and the start and end indices.
+func parseVariadic(token Token, argCount int) (bool, int, int) {
+	if !strings.HasPrefix(token.Text, "{args[") {
+		return false, 0, 0
+	}
+	if !strings.HasSuffix(token.Text, "]}") {
+		return false, 0, 0
+	}
+
+	argRange := strings.TrimSuffix(strings.TrimPrefix(token.Text, "{args["), "]}")
+	if argRange == "" {
+		caddy.Log().Named("caddyfile").Warn(
+			"Placeholder "+token.Text+" cannot have an empty index",
+			zap.String("file", token.File+":"+strconv.Itoa(token.Line)), zap.Strings("import_chain", token.imports))
+		return false, 0, 0
+	}
+
+	start, end, found := strings.Cut(argRange, ":")
+
+	// If no ":" delimiter is found, this is not a variadic.
+	// The replacer will pick this up.
+	if !found {
+		return false, 0, 0
+	}
+
+	// A valid token may contain several placeholders, and
+	// they may be separated by ":". It's not variadic.
+	// https://github.com/caddyserver/caddy/issues/5716
+	if strings.Contains(start, "}") || strings.Contains(end, "{") {
+		return false, 0, 0
+	}
+
+	var (
+		startIndex = 0
+		endIndex   = argCount
+		err        error
+	)
+	if start != "" {
+		startIndex, err = strconv.Atoi(start)
+		if err != nil {
+			caddy.Log().Named("caddyfile").Warn(
+				"Variadic placeholder "+token.Text+" has an invalid start index",
+				zap.String("file", token.File+":"+strconv.Itoa(token.Line)), zap.Strings("import_chain", token.imports))
+			return false, 0, 0
+		}
+	}
+	if end != "" {
+		endIndex, err = strconv.Atoi(end)
+		if err != nil {
+			caddy.Log().Named("caddyfile").Warn(
+				"Variadic placeholder "+token.Text+" has an invalid end index",
+				zap.String("file", token.File+":"+strconv.Itoa(token.Line)), zap.Strings("import_chain", token.imports))
+			return false, 0, 0
+		}
+	}
+
+	// bound check
+	if startIndex < 0 || startIndex > endIndex || endIndex > argCount {
+		caddy.Log().Named("caddyfile").Warn(
+			"Variadic placeholder "+token.Text+" indices are out of bounds, only "+strconv.Itoa(argCount)+" argument(s) exist",
+			zap.String("file", token.File+":"+strconv.Itoa(token.Line)), zap.Strings("import_chain", token.imports))
+		return false, 0, 0
+	}
+	return true, startIndex, endIndex
+}
+
+// makeArgsReplacer prepares a Replacer which can replace
+// non-variadic args placeholders in imported tokens.
+func makeArgsReplacer(args []string) *caddy.Replacer {
+	repl := caddy.NewEmptyReplacer()
+	repl.Map(func(key string) (any, bool) {
+		// TODO: Remove the deprecated {args.*} placeholder
+		// support at some point in the future
+		if matches := argsRegexpIndexDeprecated.FindStringSubmatch(key); len(matches) > 0 {
+			// What's matched may be a substring of the key
+			if matches[0] != key {
+				return nil, false
+			}
+
+			value, err := strconv.Atoi(matches[1])
+			if err != nil {
+				caddy.Log().Named("caddyfile").Warn(
+					"Placeholder {args." + matches[1] + "} has an invalid index")
+				return nil, false
+			}
+			if value >= len(args) {
+				caddy.Log().Named("caddyfile").Warn(
+					"Placeholder {args." + matches[1] + "} index is out of bounds, only " + strconv.Itoa(len(args)) + " argument(s) exist")
+				return nil, false
+			}
+			caddy.Log().Named("caddyfile").Warn(
+				"Placeholder {args." + matches[1] + "} deprecated, use {args[" + matches[1] + "]} instead")
+			return args[value], true
+		}
+
+		// Handle args[*] form
+		if matches := argsRegexpIndex.FindStringSubmatch(key); len(matches) > 0 {
+			// What's matched may be a substring of the key
+			if matches[0] != key {
+				return nil, false
+			}
+
+			if strings.Contains(matches[1], ":") {
+				caddy.Log().Named("caddyfile").Warn(
+					"Variadic placeholder {args[" + matches[1] + "]} must be a token on its own")
+				return nil, false
+			}
+			value, err := strconv.Atoi(matches[1])
+			if err != nil {
+				caddy.Log().Named("caddyfile").Warn(
+					"Placeholder {args[" + matches[1] + "]} has an invalid index")
+				return nil, false
+			}
+			if value >= len(args) {
+				caddy.Log().Named("caddyfile").Warn(
+					"Placeholder {args[" + matches[1] + "]} index is out of bounds, only " + strconv.Itoa(len(args)) + " argument(s) exist")
+				return nil, false
+			}
+			return args[value], true
+		}
+
+		// Not an args placeholder, ignore
+		return nil, false
+	})
+	return repl
+}
+
+var (
+	argsRegexpIndexDeprecated = regexp.MustCompile(`args\.(.+)`)
+	argsRegexpIndex           = regexp.MustCompile(`args\[(.+)]`)
+)

caddyconfig/caddyfile/importgraph.go

@@ -0,0 +1,130 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyfile
+
+import (
+	"fmt"
+)
+
+type adjacency map[string][]string
+
+type importGraph struct {
+	nodes map[string]bool
+	edges adjacency
+}
+
+func (i *importGraph) addNode(name string) {
+	if i.nodes == nil {
+		i.nodes = make(map[string]bool)
+	}
+	if _, exists := i.nodes[name]; exists {
+		return
+	}
+	i.nodes[name] = true
+}
+
+func (i *importGraph) addNodes(names []string) {
+	for _, name := range names {
+		i.addNode(name)
+	}
+}
+
+func (i *importGraph) removeNode(name string) {
+	delete(i.nodes, name)
+}
+
+func (i *importGraph) removeNodes(names []string) {
+	for _, name := range names {
+		i.removeNode(name)
+	}
+}
+
+func (i *importGraph) addEdge(from, to string) error {
+	if !i.exists(from) || !i.exists(to) {
+		return fmt.Errorf("one of the nodes does not exist")
+	}
+
+	if i.willCycle(to, from) {
+		return fmt.Errorf("a cycle of imports exists between %s and %s", from, to)
+	}
+
+	if i.areConnected(from, to) {
+		// if connected, there's nothing to do
+		return nil
+	}
+
+	if i.nodes == nil {
+		i.nodes = make(map[string]bool)
+	}
+	if i.edges == nil {
+		i.edges = make(adjacency)
+	}
+
+	i.edges[from] = append(i.edges[from], to)
+	return nil
+}
+
+func (i *importGraph) addEdges(from string, tos []string) error {
+	for _, to := range tos {
+		err := i.addEdge(from, to)
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func (i *importGraph) areConnected(from, to string) bool {
+	al, ok := i.edges[from]
+	if !ok {
+		return false
+	}
+	for _, v := range al {
+		if v == to {
+			return true
+		}
+	}
+	return false
+}
+
+func (i *importGraph) willCycle(from, to string) bool {
+	collector := make(map[string]bool)
+
+	var visit func(string)
+	visit = func(start string) {
+		if !collector[start] {
+			collector[start] = true
+			for _, v := range i.edges[start] {
+				visit(v)
+			}
+		}
+	}
+
+	for _, v := range i.edges[from] {
+		visit(v)
+	}
+	for k := range collector {
+		if to == k {
+			return true
+		}
+	}
+
+	return false
+}
+
+func (i *importGraph) exists(key string) bool {
+	_, exists := i.nodes[key]
+	return exists
+}

caddyconfig/caddyfile/lexer.go

@@ -1,4 +1,4 @@
-// Copyright 2015 Light Code Labs, LLC
+// Copyright 2015 Matthew Holt and The Caddy Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -16,7 +16,11 @@ package caddyfile
 
 import (
 	"bufio"
+	"bytes"
+	"fmt"
 	"io"
+	"regexp"
+	"strings"
 	"unicode"
 )
 
@@ -35,11 +39,40 @@ type (
 	// Token represents a single parsable unit.
 	Token struct {
 		File          string
+		imports       []string
 		Line          int
 		Text          string
+		wasQuoted     rune // enclosing quote character, if any
+		heredocMarker string
+		snippetName   string
 	}
 )
 
+// Tokenize takes bytes as input and lexes it into
+// a list of tokens that can be parsed as a Caddyfile.
+// Also takes a filename to fill the token's File as
+// the source of the tokens, which is important to
+// determine relative paths for `import` directives.
+func Tokenize(input []byte, filename string) ([]Token, error) {
+	l := lexer{}
+	if err := l.load(bytes.NewReader(input)); err != nil {
+		return nil, err
+	}
+	var tokens []Token
+	for {
+		found, err := l.next()
+		if err != nil {
+			return nil, err
+		}
+		if !found {
+			break
+		}
+		l.token.File = filename
+		tokens = append(tokens, l.token)
+	}
+	return tokens, nil
+}
+
 // load prepares the lexer to scan an input for tokens.
 // It discards any leading byte order mark.
 func (l *lexer) load(input io.Reader) error {
@@ -71,34 +104,114 @@ func (l *lexer) load(input io.Reader) error {
 // may be escaped. The rest of the line is skipped
 // if a "#" character is read in. Returns true if
 // a token was loaded; false otherwise.
-func (l *lexer) next() bool {
+func (l *lexer) next() (bool, error) {
 	var val []rune
-	var comment, quoted, escaped bool
+	var comment, quoted, btQuoted, inHeredoc, heredocEscaped, escaped bool
+	var heredocMarker string
 
-	makeToken := func() bool {
+	makeToken := func(quoted rune) bool {
 		l.token.Text = string(val)
+		l.token.wasQuoted = quoted
+		l.token.heredocMarker = heredocMarker
 		return true
 	}
 
 	for {
+		// Read a character in; if err then if we had
+		// read some characters, make a token. If we
+		// reached EOF, then no more tokens to read.
+		// If no EOF, then we had a problem.
 		ch, _, err := l.reader.ReadRune()
 		if err != nil {
 			if len(val) > 0 {
-				return makeToken()
+				if inHeredoc {
-			}
+					return false, fmt.Errorf("incomplete heredoc <<%s on line #%d, expected ending marker %s", heredocMarker, l.line+l.skippedLines, heredocMarker)
-			if err == io.EOF {
-				return false
-			}
-			panic(err)
 				}
 
-		if !escaped && ch == '\\' {
+				return makeToken(0), nil
+			}
+			if err == io.EOF {
+				return false, nil
+			}
+			return false, err
+		}
+
+		// detect whether we have the start of a heredoc
+		if !(quoted || btQuoted) && !(inHeredoc || heredocEscaped) &&
+			len(val) > 1 && string(val[:2]) == "<<" {
+			// a space means it's just a regular token and not a heredoc
+			if ch == ' ' {
+				return makeToken(0), nil
+			}
+
+			// skip CR, we only care about LF
+			if ch == '\r' {
+				continue
+			}
+
+			// after hitting a newline, we know that the heredoc marker
+			// is the characters after the two << and the newline.
+			// we reset the val because the heredoc is syntax we don't
+			// want to keep.
+			if ch == '\n' {
+				if len(val) == 2 {
+					return false, fmt.Errorf("missing opening heredoc marker on line #%d; must contain only alpha-numeric characters, dashes and underscores; got empty string", l.line)
+				}
+
+				// check if there's too many <
+				if string(val[:3]) == "<<<" {
+					return false, fmt.Errorf("too many '<' for heredoc on line #%d; only use two, for example <<END", l.line)
+				}
+
+				heredocMarker = string(val[2:])
+				if !heredocMarkerRegexp.Match([]byte(heredocMarker)) {
+					return false, fmt.Errorf("heredoc marker on line #%d must contain only alpha-numeric characters, dashes and underscores; got '%s'", l.line, heredocMarker)
+				}
+
+				inHeredoc = true
+				l.skippedLines++
+				val = nil
+				continue
+			}
+			val = append(val, ch)
+			continue
+		}
+
+		// if we're in a heredoc, all characters are read as-is
+		if inHeredoc {
+			val = append(val, ch)
+
+			if ch == '\n' {
+				l.skippedLines++
+			}
+
+			// check if we're done, i.e. that the last few characters are the marker
+			if len(val) > len(heredocMarker) && heredocMarker == string(val[len(val)-len(heredocMarker):]) {
+				// set the final value
+				val, err = l.finalizeHeredoc(val, heredocMarker)
+				if err != nil {
+					return false, err
+				}
+
+				// set the line counter, and make the token
+				l.line += l.skippedLines
+				l.skippedLines = 0
+				return makeToken('<'), nil
+			}
+
+			// stay in the heredoc until we find the ending marker
+			continue
+		}
+
+		// track whether we found an escape '\' for the next
+		// iteration to be contextually aware
+		if !escaped && !btQuoted && ch == '\\' {
 			escaped = true
 			continue
 		}
 
-		if quoted {
+		if quoted || btQuoted {
-			if escaped {
+			if quoted && escaped {
 				// all is literal in quoted area,
 				// so only escape quotes
 				if ch != '"' {
@@ -106,23 +219,29 @@ func (l *lexer) next() bool {
 				}
 				escaped = false
 			} else {
-				if ch == '"' {
+				if (quoted && ch == '"') || (btQuoted && ch == '`') {
-					return makeToken()
+					return makeToken(ch), nil
 				}
 			}
+			// allow quoted text to wrap continue on multiple lines
 			if ch == '\n' {
 				l.line += 1 + l.skippedLines
 				l.skippedLines = 0
 			}
+			// collect this character as part of the quoted token
 			val = append(val, ch)
 			continue
 		}
 
 		if unicode.IsSpace(ch) {
+			// ignore CR altogether, we only actually care about LF (\n)
 			if ch == '\r' {
 				continue
 			}
+			// end of the line
 			if ch == '\n' {
+				// newlines can be escaped to chain arguments
+				// onto multiple lines; else, increment the line count
 				if escaped {
 					l.skippedLines++
 					escaped = false
@@ -130,15 +249,19 @@ func (l *lexer) next() bool {
 					l.line += 1 + l.skippedLines
 					l.skippedLines = 0
 				}
+				// comments (#) are single-line only
 				comment = false
 			}
+			// any kind of space means we're at the end of this token
 			if len(val) > 0 {
-				return makeToken()
+				return makeToken(0), nil
 			}
 			continue
 		}
 
-		if ch == '#' {
+		// comments must be at the start of a token,
+		// in other words, preceded by space or newline
+		if ch == '#' && len(val) == 0 {
 			comment = true
 		}
 		if comment {
@@ -151,13 +274,106 @@ func (l *lexer) next() bool {
 				quoted = true
 				continue
 			}
+			if ch == '`' {
+				btQuoted = true
+				continue
+			}
 		}
 
 		if escaped {
+			// allow escaping the first < to skip the heredoc syntax
+			if ch == '<' {
+				heredocEscaped = true
+			} else {
 				val = append(val, '\\')
+			}
 			escaped = false
 		}
 
 		val = append(val, ch)
 	}
 }
+
+// finalizeHeredoc takes the runes read as the heredoc text and the marker,
+// and processes the text to strip leading whitespace, returning the final
+// value without the leading whitespace.
+func (l *lexer) finalizeHeredoc(val []rune, marker string) ([]rune, error) {
+	stringVal := string(val)
+
+	// find the last newline of the heredoc, which is where the contents end
+	lastNewline := strings.LastIndex(stringVal, "\n")
+
+	// collapse the content, then split into separate lines
+	lines := strings.Split(stringVal[:lastNewline+1], "\n")
+
+	// figure out how much whitespace we need to strip from the front of every line
+	// by getting the string that precedes the marker, on the last line
+	paddingToStrip := stringVal[lastNewline+1 : len(stringVal)-len(marker)]
+
+	// iterate over each line and strip the whitespace from the front
+	var out string
+	for lineNum, lineText := range lines[:len(lines)-1] {
+		// find an exact match for the padding
+		index := strings.Index(lineText, paddingToStrip)
+
+		// if the padding doesn't match exactly at the start then we can't safely strip
+		if index != 0 {
+			return nil, fmt.Errorf("mismatched leading whitespace in heredoc <<%s on line #%d [%s], expected whitespace [%s] to match the closing marker", marker, l.line+lineNum+1, lineText, paddingToStrip)
+		}
+
+		// strip, then append the line, with the newline, to the output.
+		// also removes all "\r" because Windows.
+		out += strings.ReplaceAll(lineText[len(paddingToStrip):]+"\n", "\r", "")
+	}
+
+	// Remove the trailing newline from the loop
+	if len(out) > 0 && out[len(out)-1] == '\n' {
+		out = out[:len(out)-1]
+	}
+
+	// return the final value
+	return []rune(out), nil
+}
+
+func (t Token) Quoted() bool {
+	return t.wasQuoted > 0
+}
+
+// NumLineBreaks counts how many line breaks are in the token text.
+func (t Token) NumLineBreaks() int {
+	lineBreaks := strings.Count(t.Text, "\n")
+	if t.wasQuoted == '<' {
+		// heredocs have an extra linebreak because the opening
+		// delimiter is on its own line and is not included in the
+		// token Text itself, and the trailing newline is removed.
+		lineBreaks += 2
+	}
+	return lineBreaks
+}
+
+var heredocMarkerRegexp = regexp.MustCompile("^[A-Za-z0-9_-]+$")
+
+// isNextOnNewLine tests whether t2 is on a different line from t1
+func isNextOnNewLine(t1, t2 Token) bool {
+	// If the second token is from a different file,
+	// we can assume it's from a different line
+	if t1.File != t2.File {
+		return true
+	}
+
+	// If the second token is from a different import chain,
+	// we can assume it's from a different line
+	if len(t1.imports) != len(t2.imports) {
+		return true
+	}
+	for i, im := range t1.imports {
+		if im != t2.imports[i] {
+			return true
+		}
+	}
+
+	// If the first token (incl line breaks) ends
+	// on a line earlier than the next token,
+	// then the second token is on a new line
+	return t1.Line+t1.NumLineBreaks() < t2.Line
+}

caddyconfig/caddyfile/lexer_fuzz.go

@@ -12,25 +12,17 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// +build gofuzz
+//go:build gofuzz
-// +build gofuzz_libfuzzer
 
 package caddyfile
 
-import (
+func FuzzTokenize(input []byte) int {
-	"bytes"
+	tokens, err := Tokenize(input, "Caddyfile")
-)
-
-func FuzzParseCaddyfile(data []byte) (score int) {
-	sb, err := Parse("Caddyfile", bytes.NewReader(data))
 	if err != nil {
-		// if both an error is received and some ServerBlocks,
-		// then the parse was able to parse partially. Mark this
-		// result as interesting to push the fuzzer further through the parser.
-		if sb != nil && len(sb) > 0 {
-			return 1
-		}
 		return 0
 	}
+	if len(tokens) == 0 {
+		return -1
+	}
 	return 1
 }

caddyconfig/caddyfile/lexer_test.go

@@ -1,4 +1,4 @@
-// Copyright 2015 Light Code Labs, LLC
+// Copyright 2015 Matthew Holt and The Caddy Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -15,37 +15,35 @@
 package caddyfile
 
 import (
-	"log"
-	"strings"
 	"testing"
 )
 
-type lexerTestCase struct {
-	input    string
-	expected []Token
-}
-
 func TestLexer(t *testing.T) {
-	testCases := []lexerTestCase{
+	testCases := []struct {
+		input        []byte
+		expected     []Token
+		expectErr    bool
+		errorMessage string
+	}{
 		{
-			input: `host:123`,
+			input: []byte(`host:123`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 			},
 		},
 		{
-			input: `host:123
+			input: []byte(`host:123
 
-					directive`,
+					directive`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 3, Text: "directive"},
 			},
 		},
 		{
-			input: `host:123 {
+			input: []byte(`host:123 {
 						directive
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -54,7 +52,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `host:123 { directive }`,
+			input: []byte(`host:123 { directive }`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -63,12 +61,12 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `host:123 {
+			input: []byte(`host:123 {
 						#comment
 						directive
 						# comment
 						foobar # another comment
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "host:123"},
 				{Line: 1, Text: "{"},
@@ -78,8 +76,28 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `a "quoted value" b
+			input: []byte(`host:123 {
-					foobar`,
+						# hash inside string is not a comment
+						redir / /some/#/path
+					}`),
+			expected: []Token{
+				{Line: 1, Text: "host:123"},
+				{Line: 1, Text: "{"},
+				{Line: 3, Text: "redir"},
+				{Line: 3, Text: "/"},
+				{Line: 3, Text: "/some/#/path"},
+				{Line: 4, Text: "}"},
+			},
+		},
+		{
+			input: []byte("# comment at beginning of file\n# comment at beginning of line\nhost:123"),
+			expected: []Token{
+				{Line: 3, Text: "host:123"},
+			},
+		},
+		{
+			input: []byte(`a "quoted value" b
+					foobar`),
 			expected: []Token{
 				{Line: 1, Text: "a"},
 				{Line: 1, Text: "quoted value"},
@@ -88,7 +106,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `A "quoted \"value\" inside" B`,
+			input: []byte(`A "quoted \"value\" inside" B`),
 			expected: []Token{
 				{Line: 1, Text: "A"},
 				{Line: 1, Text: `quoted "value" inside`},
@@ -96,7 +114,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "An escaped \"newline\\\ninside\" quotes",
+			input: []byte("An escaped \"newline\\\ninside\" quotes"),
 			expected: []Token{
 				{Line: 1, Text: "An"},
 				{Line: 1, Text: "escaped"},
@@ -105,7 +123,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "An escaped newline\\\noutside quotes",
+			input: []byte("An escaped newline\\\noutside quotes"),
 			expected: []Token{
 				{Line: 1, Text: "An"},
 				{Line: 1, Text: "escaped"},
@@ -115,7 +133,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "line1\\\nescaped\nline2\nline3",
+			input: []byte("line1\\\nescaped\nline2\nline3"),
 			expected: []Token{
 				{Line: 1, Text: "line1"},
 				{Line: 1, Text: "escaped"},
@@ -124,7 +142,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "line1\\\nescaped1\\\nescaped2\nline4\nline5",
+			input: []byte("line1\\\nescaped1\\\nescaped2\nline4\nline5"),
 			expected: []Token{
 				{Line: 1, Text: "line1"},
 				{Line: 1, Text: "escaped1"},
@@ -134,34 +152,34 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `"unescapable\ in quotes"`,
+			input: []byte(`"unescapable\ in quotes"`),
 			expected: []Token{
 				{Line: 1, Text: `unescapable\ in quotes`},
 			},
 		},
 		{
-			input: `"don't\escape"`,
+			input: []byte(`"don't\escape"`),
 			expected: []Token{
 				{Line: 1, Text: `don't\escape`},
 			},
 		},
 		{
-			input: `"don't\\escape"`,
+			input: []byte(`"don't\\escape"`),
 			expected: []Token{
 				{Line: 1, Text: `don't\\escape`},
 			},
 		},
 		{
-			input: `un\escapable`,
+			input: []byte(`un\escapable`),
 			expected: []Token{
 				{Line: 1, Text: `un\escapable`},
 			},
 		},
 		{
-			input: `A "quoted value with line
+			input: []byte(`A "quoted value with line
 					break inside" {
 						foobar
-					}`,
+					}`),
 			expected: []Token{
 				{Line: 1, Text: "A"},
 				{Line: 1, Text: "quoted value with line\n\t\t\t\t\tbreak inside"},
@@ -171,13 +189,13 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: `"C:\php\php-cgi.exe"`,
+			input: []byte(`"C:\php\php-cgi.exe"`),
 			expected: []Token{
 				{Line: 1, Text: `C:\php\php-cgi.exe`},
 			},
 		},
 		{
-			input: `empty "" string`,
+			input: []byte(`empty "" string`),
 			expected: []Token{
 				{Line: 1, Text: `empty`},
 				{Line: 1, Text: ``},
@@ -185,7 +203,7 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "skip those\r\nCR characters",
+			input: []byte("skip those\r\nCR characters"),
 			expected: []Token{
 				{Line: 1, Text: "skip"},
 				{Line: 1, Text: "those"},
@@ -194,43 +212,274 @@ func TestLexer(t *testing.T) {
 			},
 		},
 		{
-			input: "\xEF\xBB\xBF:8080", // test with leading byte order mark
+			input: []byte("\xEF\xBB\xBF:8080"), // test with leading byte order mark
 			expected: []Token{
 				{Line: 1, Text: ":8080"},
 			},
 		},
+		{
+			input: []byte("simple `backtick quoted` string"),
+			expected: []Token{
+				{Line: 1, Text: `simple`},
+				{Line: 1, Text: `backtick quoted`},
+				{Line: 1, Text: `string`},
+			},
+		},
+		{
+			input: []byte("multiline `backtick\nquoted\n` string"),
+			expected: []Token{
+				{Line: 1, Text: `multiline`},
+				{Line: 1, Text: "backtick\nquoted\n"},
+				{Line: 3, Text: `string`},
+			},
+		},
+		{
+			input: []byte("nested `\"quotes inside\" backticks` string"),
+			expected: []Token{
+				{Line: 1, Text: `nested`},
+				{Line: 1, Text: `"quotes inside" backticks`},
+				{Line: 1, Text: `string`},
+			},
+		},
+		{
+			input: []byte("reverse-nested \"`backticks` inside\" quotes"),
+			expected: []Token{
+				{Line: 1, Text: `reverse-nested`},
+				{Line: 1, Text: "`backticks` inside"},
+				{Line: 1, Text: `quotes`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<EOF
+content
+EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "content"},
+				{Line: 3, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<VERY-LONG-MARKER
+content
+VERY-LONG-MARKER same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "content"},
+				{Line: 3, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<EOF
+extra-newline
+
+EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "extra-newline\n"},
+				{Line: 4, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<EOF
+		EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: ""},
+				{Line: 2, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<EOF
+	content
+	EOF same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "content"},
+				{Line: 3, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`prev-line
+	heredoc <<EOF
+		multi
+		line
+		content
+	EOF same-line-arg
+	next-line
+	`),
+			expected: []Token{
+				{Line: 1, Text: `prev-line`},
+				{Line: 2, Text: `heredoc`},
+				{Line: 2, Text: "\tmulti\n\tline\n\tcontent"},
+				{Line: 6, Text: `same-line-arg`},
+				{Line: 7, Text: `next-line`},
+			},
+		},
+		{
+			input: []byte(`escaped-heredoc \<< >>`),
+			expected: []Token{
+				{Line: 1, Text: `escaped-heredoc`},
+				{Line: 1, Text: `<<`},
+				{Line: 1, Text: `>>`},
+			},
+		},
+		{
+			input: []byte(`not-a-heredoc <EOF
+	content
+	`),
+			expected: []Token{
+				{Line: 1, Text: `not-a-heredoc`},
+				{Line: 1, Text: `<EOF`},
+				{Line: 2, Text: `content`},
+			},
+		},
+		{
+			input: []byte(`not-a-heredoc <<<EOF content`),
+			expected: []Token{
+				{Line: 1, Text: `not-a-heredoc`},
+				{Line: 1, Text: `<<<EOF`},
+				{Line: 1, Text: `content`},
+			},
+		},
+		{
+			input: []byte(`not-a-heredoc "<<" ">>"`),
+			expected: []Token{
+				{Line: 1, Text: `not-a-heredoc`},
+				{Line: 1, Text: `<<`},
+				{Line: 1, Text: `>>`},
+			},
+		},
+		{
+			input: []byte(`not-a-heredoc << >>`),
+			expected: []Token{
+				{Line: 1, Text: `not-a-heredoc`},
+				{Line: 1, Text: `<<`},
+				{Line: 1, Text: `>>`},
+			},
+		},
+		{
+			input: []byte(`not-a-heredoc <<HERE SAME LINE
+	content
+	HERE same-line-arg
+	`),
+			expected: []Token{
+				{Line: 1, Text: `not-a-heredoc`},
+				{Line: 1, Text: `<<HERE`},
+				{Line: 1, Text: `SAME`},
+				{Line: 1, Text: `LINE`},
+				{Line: 2, Text: `content`},
+				{Line: 3, Text: `HERE`},
+				{Line: 3, Text: `same-line-arg`},
+			},
+		},
+		{
+			input: []byte(`heredoc <<s
+			�
+			s
+	`),
+			expected: []Token{
+				{Line: 1, Text: `heredoc`},
+				{Line: 1, Text: "�"},
+			},
+		},
+		{
+			input: []byte("\u000Aheredoc \u003C\u003C\u0073\u0073\u000A\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F\u000A\u0073\u0073\u000A\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F\u000A\u00BF\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F"),
+			expected: []Token{
+				{
+					Line: 2,
+					Text: "heredoc",
+				},
+				{
+					Line: 2,
+					Text: "\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F",
+				},
+				{
+					Line: 5,
+					Text: "\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F",
+				},
+				{
+					Line: 6,
+					Text: "\u00BF\u00BF\u0057\u0001\u0000\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u00FF\u003D\u001F",
+				},
+			},
+		},
+		{
+			input:        []byte("not-a-heredoc <<\n"),
+			expectErr:    true,
+			errorMessage: "missing opening heredoc marker on line #1; must contain only alpha-numeric characters, dashes and underscores; got empty string",
+		},
+		{
+			input: []byte(`heredoc <<<EOF
+	content
+	EOF same-line-arg
+	`),
+			expectErr:    true,
+			errorMessage: "too many '<' for heredoc on line #1; only use two, for example <<END",
+		},
+		{
+			input: []byte(`heredoc <<EOF
+	content
+	`),
+			expectErr:    true,
+			errorMessage: "incomplete heredoc <<EOF on line #3, expected ending marker EOF",
+		},
+		{
+			input: []byte(`heredoc <<EOF
+	content
+		EOF
+	`),
+			expectErr:    true,
+			errorMessage: "mismatched leading whitespace in heredoc <<EOF on line #2 [\tcontent], expected whitespace [\t\t] to match the closing marker",
+		},
+		{
+			input: []byte(`heredoc <<EOF
+        content
+		EOF
+	`),
+			expectErr:    true,
+			errorMessage: "mismatched leading whitespace in heredoc <<EOF on line #2 [        content], expected whitespace [\t\t] to match the closing marker",
+		},
 	}
 
 	for i, testCase := range testCases {
-		actual := tokenize(testCase.input)
+		actual, err := Tokenize(testCase.input, "")
+		if testCase.expectErr {
+			if err == nil {
+				t.Fatalf("expected error, got actual: %v", actual)
+				continue
+			}
+			if err.Error() != testCase.errorMessage {
+				t.Fatalf("expected error '%v', got: %v", testCase.errorMessage, err)
+			}
+			continue
+		}
+
+		if err != nil {
+			t.Fatalf("%v", err)
+		}
 		lexerCompare(t, i, testCase.expected, actual)
 	}
 }
 
-func tokenize(input string) (tokens []Token) {
-	l := lexer{}
-	if err := l.load(strings.NewReader(input)); err != nil {
-		log.Printf("[ERROR] load failed: %v", err)
-	}
-	for l.next() {
-		tokens = append(tokens, l.token)
-	}
-	return
-}
-
 func lexerCompare(t *testing.T, n int, expected, actual []Token) {
 	if len(expected) != len(actual) {
-		t.Errorf("Test case %d: expected %d token(s) but got %d", n, len(expected), len(actual))
+		t.Fatalf("Test case %d: expected %d token(s) but got %d", n, len(expected), len(actual))
 	}
 
 	for i := 0; i < len(actual) && i < len(expected); i++ {
 		if actual[i].Line != expected[i].Line {
-			t.Errorf("Test case %d token %d ('%s'): expected line %d but was line %d",
+			t.Fatalf("Test case %d token %d ('%s'): expected line %d but was line %d",
 				n, i, expected[i].Text, expected[i].Line, actual[i].Line)
 			break
 		}
 		if actual[i].Text != expected[i].Text {
-			t.Errorf("Test case %d token %d: expected text '%s' but was '%s'",
+			t.Fatalf("Test case %d token %d: expected text '%s' but was '%s'",
 				n, i, expected[i].Text, actual[i].Text)
 			break
 		}

caddyconfig/caddyfile/parse.go

@@ -1,4 +1,4 @@
-// Copyright 2015 Light Code Labs, LLC
+// Copyright 2015 Matthew Holt and The Caddy Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -16,11 +16,15 @@ package caddyfile
 
 import (
 	"bytes"
-	"io/ioutil"
+	"fmt"
-	"log"
+	"io"
 	"os"
 	"path/filepath"
 	"strings"
+
+	"go.uber.org/zap"
+
+	"github.com/caddyserver/caddy/v2"
 )
 
 // Parse parses the input just enough to group tokens, in
@@ -33,16 +37,36 @@ import (
 // Environment variables in {$ENVIRONMENT_VARIABLE} notation
 // will be replaced before parsing begins.
 func Parse(filename string, input []byte) ([]ServerBlock, error) {
-	tokens, err := allTokens(filename, input)
+	// unfortunately, we must copy the input because parsing must
+	// remain a read-only operation, but we have to expand environment
+	// variables before we parse, which changes the underlying array (#4422)
+	inputCopy := make([]byte, len(input))
+	copy(inputCopy, input)
+
+	tokens, err := allTokens(filename, inputCopy)
 	if err != nil {
 		return nil, err
 	}
-	p := parser{Dispenser: NewDispenser(tokens)}
+	p := parser{
+		Dispenser: NewDispenser(tokens),
+		importGraph: importGraph{
+			nodes: make(map[string]bool),
+			edges: make(adjacency),
+		},
+	}
 	return p.parseAll()
 }
 
+// allTokens lexes the entire input, but does not parse it.
+// It returns all the tokens from the input, unstructured
+// and in order. It may mutate input as it expands env vars.
+func allTokens(filename string, input []byte) ([]Token, error) {
+	return Tokenize(replaceEnvVars(input), filename)
+}
+
 // replaceEnvVars replaces all occurrences of environment variables.
-func replaceEnvVars(input []byte) ([]byte, error) {
+// It mutates the underlying array and returns the updated slice.
+func replaceEnvVars(input []byte) []byte {
 	var offset int
 	for {
 		begin := bytes.Index(input[offset:], spanOpen)
@@ -57,44 +81,33 @@ func replaceEnvVars(input []byte) ([]byte, error) {
 		end += begin + len(spanOpen) // make end relative to input, not begin
 
 		// get the name; if there is no name, skip it
-		envVarName := input[begin+len(spanOpen) : end]
+		envString := input[begin+len(spanOpen) : end]
-		if len(envVarName) == 0 {
+		if len(envString) == 0 {
 			offset = end + len(spanClose)
 			continue
 		}
 
+		// split the string into a key and an optional default
+		envParts := strings.SplitN(string(envString), envVarDefaultDelimiter, 2)
+
+		// do a lookup for the env var, replace with the default if not found
+		envVarValue, found := os.LookupEnv(envParts[0])
+		if !found && len(envParts) == 2 {
+			envVarValue = envParts[1]
+		}
+
 		// get the value of the environment variable
-		envVarValue := []byte(os.Getenv(string(envVarName)))
+		// note that this causes one-level deep chaining
+		envVarBytes := []byte(envVarValue)
 
 		// splice in the value
 		input = append(input[:begin],
-			append(envVarValue, input[end+len(spanClose):]...)...)
+			append(envVarBytes, input[end+len(spanClose):]...)...)
 
 		// continue at the end of the replacement
-		offset = begin + len(envVarValue)
+		offset = begin + len(envVarBytes)
 	}
-	return input, nil
+	return input
-}
-
-// allTokens lexes the entire input, but does not parse it.
-// It returns all the tokens from the input, unstructured
-// and in order.
-func allTokens(filename string, input []byte) ([]Token, error) {
-	input, err := replaceEnvVars(input)
-	if err != nil {
-		return nil, err
-	}
-	l := new(lexer)
-	err = l.load(bytes.NewReader(input))
-	if err != nil {
-		return nil, err
-	}
-	var tokens []Token
-	for l.next() {
-		l.token.File = filename
-		tokens = append(tokens, l.token)
-	}
-	return tokens, nil
 }
 
 type parser struct {
@@ -103,6 +116,7 @@ type parser struct {
 	eof             bool        // if we encounter a valid EOF in a hard place
 	definedSnippets map[string][]Token
 	nesting         int
+	importGraph     importGraph
 }
 
 func (p *parser) parseAll() ([]ServerBlock, error) {
@@ -135,7 +149,6 @@ func (p *parser) begin() error {
 	}
 
 	err := p.addresses()
-
 	if err != nil {
 		return err
 	}
@@ -146,6 +159,25 @@ func (p *parser) begin() error {
 		return nil
 	}
 
+	if ok, name := p.isNamedRoute(); ok {
+		// named routes only have one key, the route name
+		p.block.Keys = []string{name}
+		p.block.IsNamedRoute = true
+
+		// we just need a dummy leading token to ease parsing later
+		nameToken := p.Token()
+		nameToken.Text = name
+
+		// get all the tokens from the block, including the braces
+		tokens, err := p.blockTokens(true)
+		if err != nil {
+			return err
+		}
+		tokens = append([]Token{nameToken}, tokens...)
+		p.block.Segments = []Segment{tokens}
+		return nil
+	}
+
 	if ok, name := p.isSnippet(); ok {
 		if p.definedSnippets == nil {
 			p.definedSnippets = map[string][]Token{}
@@ -154,10 +186,18 @@ func (p *parser) begin() error {
 			return p.Errf("redeclaration of previously declared snippet %s", name)
 		}
 		// consume all tokens til matched close brace
-		tokens, err := p.snippetTokens()
+		tokens, err := p.blockTokens(false)
 		if err != nil {
 			return err
 		}
+		// Just as we need to track which file the token comes from, we need to
+		// keep track of which snippet the token comes from. This is helpful
+		// in tracking import cycles across files/snippets by namespacing them.
+		// Without this, we end up with false-positives in cycle-detection.
+		for k, v := range tokens {
+			v.snippetName = name
+			tokens[k] = v
+		}
 		p.definedSnippets[name] = tokens
 		// empty block keys so we don't save this block as a real server.
 		p.block.Keys = nil
@@ -175,7 +215,7 @@ func (p *parser) addresses() error {
 
 		// special case: import directive replaces tokens during parse-time
 		if tkn == "import" && p.isNewLine() {
-			err := p.doImport()
+			err := p.doImport(0)
 			if err != nil {
 				return err
 			}
@@ -187,9 +227,20 @@ func (p *parser) addresses() error {
 			if expectingAnother {
 				return p.Errf("Expected another address but had '%s' - check for extra comma", tkn)
 			}
+			// Mark this server block as being defined with braces.
+			// This is used to provide a better error message when
+			// the user may have tried to define two server blocks
+			// without having used braces, which are required in
+			// that case.
+			p.block.HasBraces = true
 			break
 		}
 
+		// Users commonly forget to place a space between the address and the '{'
+		if strings.HasSuffix(tkn, "{") {
+			return p.Errf("Site addresses cannot end with a curly brace: '%s' - put a space between the token and the brace", tkn)
+		}
+
 		if tkn != "" { // empty token possible if user typed ""
 			// Trailing comma indicates another address will follow, which
 			// may possibly be on the next line
@@ -200,6 +251,13 @@ func (p *parser) addresses() error {
 				expectingAnother = false // but we may still see another one on this line
 			}
 
+			// If there's a comma here, it's probably because they didn't use a space
+			// between their two domains, e.g. "foo.com,bar.com", which would not be
+			// parsed as two separate site addresses.
+			if strings.Contains(tkn, ",") {
+				return p.Errf("Site addresses cannot contain a comma ',': '%s' - put a space after the comma to separate site addresses", tkn)
+			}
+
 			p.block.Keys = append(p.block.Keys, tkn)
 		}
 
@@ -257,7 +315,7 @@ func (p *parser) directives() error {
 
 		// special case: import directive replaces tokens during parse-time
 		if p.Val() == "import" {
-			err := p.doImport()
+			err := p.doImport(1)
 			if err != nil {
 				return err
 			}
@@ -283,7 +341,7 @@ func (p *parser) directives() error {
 // is on the token before where the import directive was. In
 // other words, call Next() to access the first token that was
 // imported.
-func (p *parser) doImport() error {
+func (p *parser) doImport(nesting int) error {
 	// syntax checks
 	if !p.NextArg() {
 		return p.ArgErr()
@@ -292,17 +350,27 @@ func (p *parser) doImport() error {
 	if importPattern == "" {
 		return p.Err("Import requires a non-empty filepath")
 	}
-	if p.NextArg() {
+
-		return p.Err("Import takes only one argument (glob pattern or file)")
+	// grab remaining args as placeholder replacements
-	}
+	args := p.RemainingArgs()
-	// splice out the import directive and its argument (2 tokens total)
+
-	tokensBefore := p.tokens[:p.cursor-1]
+	// set up a replacer for non-variadic args replacement
+	repl := makeArgsReplacer(args)
+
+	// splice out the import directive and its arguments
+	// (2 tokens, plus the length of args)
+	tokensBefore := p.tokens[:p.cursor-1-len(args)]
 	tokensAfter := p.tokens[p.cursor+1:]
 	var importedTokens []Token
+	var nodes []string
 
 	// first check snippets. That is a simple, non-recursive replacement
 	if p.definedSnippets != nil && p.definedSnippets[importPattern] != nil {
 		importedTokens = p.definedSnippets[importPattern]
+		if len(importedTokens) > 0 {
+			// just grab the first one
+			nodes = append(nodes, fmt.Sprintf("%s:%s", importedTokens[0].File, importedTokens[0].snippetName))
+		}
 	} else {
 		// make path relative to the file of the _token_ being processed rather
 		// than current working directory (issue #867) and then use glob to get
@@ -331,14 +399,27 @@ func (p *parser) doImport() error {
 		}
 		if len(matches) == 0 {
 			if strings.ContainsAny(globPattern, "*?[]") {
-				log.Printf("[WARNING] No files matching import glob pattern: %s", importPattern)
+				caddy.Log().Warn("No files matching import glob pattern", zap.String("pattern", importPattern))
 			} else {
 				return p.Errf("File to import not found: %s", importPattern)
 			}
+		} else {
+			// See issue #5295 - should skip any files that start with a . when iterating over them.
+			sep := string(filepath.Separator)
+			segGlobPattern := strings.Split(globPattern, sep)
+			if strings.HasPrefix(segGlobPattern[len(segGlobPattern)-1], "*") {
+				var tmpMatches []string
+				for _, m := range matches {
+					seg := strings.Split(m, sep)
+					if !strings.HasPrefix(seg[len(seg)-1], ".") {
+						tmpMatches = append(tmpMatches, m)
+					}
+				}
+				matches = tmpMatches
+			}
 		}
 
 		// collect all the imported tokens
-
 		for _, importFile := range matches {
 			newTokens, err := p.doSingleImport(importFile)
 			if err != nil {
@@ -346,12 +427,90 @@ func (p *parser) doImport() error {
 			}
 			importedTokens = append(importedTokens, newTokens...)
 		}
+		nodes = matches
+	}
+
+	nodeName := p.File()
+	if p.Token().snippetName != "" {
+		nodeName += fmt.Sprintf(":%s", p.Token().snippetName)
+	}
+	p.importGraph.addNode(nodeName)
+	p.importGraph.addNodes(nodes)
+	if err := p.importGraph.addEdges(nodeName, nodes); err != nil {
+		p.importGraph.removeNodes(nodes)
+		return err
+	}
+
+	// copy the tokens so we don't overwrite p.definedSnippets
+	tokensCopy := make([]Token, 0, len(importedTokens))
+
+	var (
+		maybeSnippet   bool
+		maybeSnippetId bool
+		index          int
+	)
+
+	// run the argument replacer on the tokens
+	// golang for range slice return a copy of value
+	// similarly, append also copy value
+	for i, token := range importedTokens {
+		// update the token's imports to refer to import directive filename, line number and snippet name if there is one
+		if token.snippetName != "" {
+			token.imports = append(token.imports, fmt.Sprintf("%s:%d (import %s)", p.File(), p.Line(), token.snippetName))
+		} else {
+			token.imports = append(token.imports, fmt.Sprintf("%s:%d (import)", p.File(), p.Line()))
+		}
+
+		// naive way of determine snippets, as snippets definition can only follow name + block
+		// format, won't check for nesting correctness or any other error, that's what parser does.
+		if !maybeSnippet && nesting == 0 {
+			// first of the line
+			if i == 0 || isNextOnNewLine(tokensCopy[i-1], token) {
+				index = 0
+			} else {
+				index++
+			}
+
+			if index == 0 && len(token.Text) >= 3 && strings.HasPrefix(token.Text, "(") && strings.HasSuffix(token.Text, ")") {
+				maybeSnippetId = true
+			}
+		}
+
+		switch token.Text {
+		case "{":
+			nesting++
+			if index == 1 && maybeSnippetId && nesting == 1 {
+				maybeSnippet = true
+				maybeSnippetId = false
+			}
+		case "}":
+			nesting--
+			if nesting == 0 && maybeSnippet {
+				maybeSnippet = false
+			}
+		}
+
+		if maybeSnippet {
+			tokensCopy = append(tokensCopy, token)
+			continue
+		}
+
+		foundVariadic, startIndex, endIndex := parseVariadic(token, len(args))
+		if foundVariadic {
+			for _, arg := range args[startIndex:endIndex] {
+				token.Text = arg
+				tokensCopy = append(tokensCopy, token)
+			}
+		} else {
+			token.Text = repl.ReplaceKnown(token.Text, "")
+			tokensCopy = append(tokensCopy, token)
+		}
 	}
 
 	// splice the imported tokens in the place of the import statement
 	// and rewind cursor so Next() will land on first imported token
-	p.tokens = append(tokensBefore, append(importedTokens, tokensAfter...)...)
+	p.tokens = append(tokensBefore, append(tokensCopy, tokensAfter...)...)
-	p.cursor--
+	p.cursor -= len(args) + 1
 
 	return nil
 }
@@ -371,11 +530,17 @@ func (p *parser) doSingleImport(importFile string) ([]Token, error) {
 		return nil, p.Errf("Could not import %s: is a directory", importFile)
 	}
 
-	input, err := ioutil.ReadAll(file)
+	input, err := io.ReadAll(file)
 	if err != nil {
 		return nil, p.Errf("Could not read imported file %s: %v", importFile, err)
 	}
 
+	// only warning in case of empty files
+	if len(input) == 0 || len(strings.TrimSpace(string(input))) == 0 {
+		caddy.Log().Warn("Import file is empty", zap.String("file", importFile))
+		return []Token{}, nil
+	}
+
 	importedTokens, err := allTokens(importFile, input)
 	if err != nil {
 		return nil, p.Errf("Could not read tokens while importing %s: %v", importFile, err)
@@ -401,7 +566,6 @@ func (p *parser) doSingleImport(importFile string) ([]Token, error) {
 // are loaded into the current server block for later use
 // by directive setup functions.
 func (p *parser) directive() error {
-
 	// a segment is a list of tokens associated with this directive
 	var segment Segment
 
@@ -411,6 +575,16 @@ func (p *parser) directive() error {
 	for p.Next() {
 		if p.Val() == "{" {
 			p.nesting++
+			if !p.isNextOnNewLine() && p.Token().wasQuoted == 0 {
+				return p.Err("Unexpected next token after '{' on same line")
+			}
+			if p.isNewLine() {
+				return p.Err("Unexpected '{' on a new line; did you mean to place the '{' on the previous line?")
+			}
+		} else if p.Val() == "{}" {
+			if p.isNextOnNewLine() && p.Token().wasQuoted == 0 {
+				return p.Err("Unexpected '{}' at end of line")
+			}
 		} else if p.isNewLine() && p.nesting == 0 {
 			p.cursor-- // read too far
 			break
@@ -419,7 +593,7 @@ func (p *parser) directive() error {
 		} else if p.Val() == "}" && p.nesting == 0 {
 			return p.Err("Unexpected '}' because no matching opening brace")
 		} else if p.Val() == "import" && p.isNewLine() {
-			if err := p.doImport(); err != nil {
+			if err := p.doImport(1); err != nil {
 				return err
 			}
 			p.cursor-- // cursor is advanced when we continue, so roll back one more
@@ -460,6 +634,15 @@ func (p *parser) closeCurlyBrace() error {
 	return nil
 }
 
+func (p *parser) isNamedRoute() (bool, string) {
+	keys := p.block.Keys
+	// A named route block is a single key with parens, prefixed with &.
+	if len(keys) == 1 && strings.HasPrefix(keys[0], "&(") && strings.HasSuffix(keys[0], ")") {
+		return true, strings.TrimSuffix(keys[0][2:], ")")
+	}
+	return false, ""
+}
+
 func (p *parser) isSnippet() (bool, string) {
 	keys := p.block.Keys
 	// A snippet block is a single key with parens. Nothing else qualifies.
@@ -470,18 +653,24 @@ func (p *parser) isSnippet() (bool, string) {
 }
 
 // read and store everything in a block for later replay.
-func (p *parser) snippetTokens() ([]Token, error) {
+func (p *parser) blockTokens(retainCurlies bool) ([]Token, error) {
-	// snippet must have curlies.
+	// block must have curlies.
 	err := p.openCurlyBrace()
 	if err != nil {
 		return nil, err
 	}
-	nesting := 1 // count our own nesting in snippets
+	nesting := 1 // count our own nesting
 	tokens := []Token{}
+	if retainCurlies {
+		tokens = append(tokens, p.Token())
+	}
 	for p.Next() {
 		if p.Val() == "}" {
 			nesting--
 			if nesting == 0 {
+				if retainCurlies {
+					tokens = append(tokens, p.Token())
+				}
 				break
 			}
 		}
@@ -501,8 +690,10 @@ func (p *parser) snippetTokens() ([]Token, error) {
 // head of the server block with tokens, which are
 // grouped by segments.
 type ServerBlock struct {
+	HasBraces    bool
 	Keys         []string
 	Segments     []Segment
+	IsNamedRoute bool
 }
 
 // DispenseDirective returns a dispenser that contains
@@ -533,4 +724,7 @@ func (s Segment) Directive() string {
 
 // spanOpen and spanClose are used to bound spans that
 // contain the name of an environment variable.
-var spanOpen, spanClose = []byte{'{', '$'}, []byte{'}'}
+var (
+	spanOpen, spanClose    = []byte{'{', '$'}, []byte{'}'}
+	envVarDefaultDelimiter = ":"
+)

caddyconfig/caddyfile/parse_test.go

@@ -1,4 +1,4 @@
-// Copyright 2015 Light Code Labs, LLC
+// Copyright 2015 Matthew Holt and The Caddy Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -16,12 +16,97 @@ package caddyfile
 
 import (
 	"bytes"
-	"io/ioutil"
 	"os"
 	"path/filepath"
 	"testing"
 )
 
+func TestParseVariadic(t *testing.T) {
+	var args = make([]string, 10)
+	for i, tc := range []struct {
+		input  string
+		result bool
+	}{
+		{
+			input:  "",
+			result: false,
+		},
+		{
+			input:  "{args[1",
+			result: false,
+		},
+		{
+			input:  "1]}",
+			result: false,
+		},
+		{
+			input:  "{args[:]}aaaaa",
+			result: false,
+		},
+		{
+			input:  "aaaaa{args[:]}",
+			result: false,
+		},
+		{
+			input:  "{args.}",
+			result: false,
+		},
+		{
+			input:  "{args.1}",
+			result: false,
+		},
+		{
+			input:  "{args[]}",
+			result: false,
+		},
+		{
+			input:  "{args[:]}",
+			result: true,
+		},
+		{
+			input:  "{args[:]}",
+			result: true,
+		},
+		{
+			input:  "{args[0:]}",
+			result: true,
+		},
+		{
+			input:  "{args[:0]}",
+			result: true,
+		},
+		{
+			input:  "{args[-1:]}",
+			result: false,
+		},
+		{
+			input:  "{args[:11]}",
+			result: false,
+		},
+		{
+			input:  "{args[10:0]}",
+			result: false,
+		},
+		{
+			input:  "{args[0:10]}",
+			result: true,
+		},
+		{
+			input:  "{args[0]}:{args[1]}:{args[2]}",
+			result: false,
+		},
+	} {
+		token := Token{
+			File: "test",
+			Line: 1,
+			Text: tc.input,
+		}
+		if v, _, _ := parseVariadic(token, len(args)); v != tc.result {
+			t.Errorf("Test %d error expectation failed Expected: %t, got %t", i, tc.result, v)
+		}
+	}
+}
+
 func TestAllTokens(t *testing.T) {
 	input := []byte("a b c\nd e")
 	expected := []string{"a", "b", "c", "d", "e"}
@@ -160,6 +245,10 @@ func TestParseOneAndImport(t *testing.T) {
 			"localhost",
 		}, []int{}},
 
+		{`localhost{
+		    dir1
+		  }`, true, []string{}, []int{}},
+
 		{`localhost
 		  dir1 {
 		    nested {
@@ -182,14 +271,56 @@ func TestParseOneAndImport(t *testing.T) {
 			"host1",
 		}, []int{1, 2}},
 
-		{`import testdata/import_test1.txt testdata/import_test2.txt`, true, []string{}, []int{}},
-
 		{`import testdata/not_found.txt`, true, []string{}, []int{}},
 
+		// empty file should just log a warning, and result in no tokens
+		{`import testdata/empty.txt`, false, []string{}, []int{}},
+
+		{`import testdata/only_white_space.txt`, false, []string{}, []int{}},
+
+		// import path/to/dir/* should skip any files that start with a . when iterating over them.
+		{`localhost
+		  dir1 arg1
+		  import testdata/glob/*`, false, []string{
+			"localhost",
+		}, []int{2, 3, 1}},
+
+		// import path/to/dir/.* should continue to read all dotfiles in a dir.
+		{`import testdata/glob/.*`, false, []string{
+			"host1",
+		}, []int{1, 2}},
+
 		{`""`, false, []string{}, []int{}},
 
 		{``, false, []string{}, []int{}},
 
+		// Unexpected next token after '{' on same line
+		{`localhost
+		  dir1 { a b }`, true, []string{"localhost"}, []int{}},
+
+		// Unexpected '{' on a new line
+		{`localhost
+		dir1
+		{
+			a b
+		}`, true, []string{"localhost"}, []int{}},
+
+		// Workaround with quotes
+		{`localhost
+		  dir1 "{" a b "}"`, false, []string{"localhost"}, []int{5}},
+
+		// Unexpected '{}' at end of line
+		{`localhost
+		  dir1 {}`, true, []string{"localhost"}, []int{}},
+		// Workaround with quotes
+		{`localhost
+		  dir1 "{}"`, false, []string{"localhost"}, []int{2}},
+
+		// import with args
+		{`import testdata/import_args0.txt a`, false, []string{"a"}, []int{}},
+		{`import testdata/import_args1.txt a b`, false, []string{"a", "b"}, []int{}},
+		{`import testdata/import_args*.txt a b`, false, []string{"a"}, []int{2}},
+
 		// test cases found by fuzzing!
 		{`import }{$"`, true, []string{}, []int{}},
 		{`import /*/*.txt`, true, []string{}, []int{}},
@@ -210,6 +341,7 @@ func TestParseOneAndImport(t *testing.T) {
 			t.Errorf("Test %d: Expected no error, but got: %v", i, err)
 		}
 
+		// t.Logf("%+v\n", result)
 		if len(result.Keys) != len(test.keys) {
 			t.Errorf("Test %d: Expected %d keys, got %d",
 				i, len(test.keys), len(result.Keys))
@@ -256,7 +388,7 @@ func TestRecursiveImport(t *testing.T) {
 			return false
 		}
 		if len(got.Segments[0]) != 1 || len(got.Segments[1]) != 2 {
-			t.Errorf("got unexpect tokens: %v", got.Segments)
+			t.Errorf("got unexpected tokens: %v", got.Segments)
 			return false
 		}
 		return true
@@ -272,7 +404,7 @@ func TestRecursiveImport(t *testing.T) {
 	}
 
 	// test relative recursive import
-	err = ioutil.WriteFile(recursiveFile1, []byte(
+	err = os.WriteFile(recursiveFile1, []byte(
 		`localhost
 		dir1
 		import recursive_import_test2`), 0644)
@@ -281,7 +413,7 @@ func TestRecursiveImport(t *testing.T) {
 	}
 	defer os.Remove(recursiveFile1)
 
-	err = ioutil.WriteFile(recursiveFile2, []byte("dir2 1"), 0644)
+	err = os.WriteFile(recursiveFile2, []byte("dir2 1"), 0644)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -306,7 +438,7 @@ func TestRecursiveImport(t *testing.T) {
 	}
 
 	// test absolute recursive import
-	err = ioutil.WriteFile(recursiveFile1, []byte(
+	err = os.WriteFile(recursiveFile1, []byte(
 		`localhost
 		dir1
 		import `+recursiveFile2), 0644)
@@ -351,7 +483,7 @@ func TestDirectiveImport(t *testing.T) {
 			return false
 		}
 		if len(got.Segments[0]) != 1 || len(got.Segments[1]) != 8 {
-			t.Errorf("got unexpect tokens: %v", got.Segments)
+			t.Errorf("got unexpected tokens: %v", got.Segments)
 			return false
 		}
 		return true
@@ -362,7 +494,7 @@ func TestDirectiveImport(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	err = ioutil.WriteFile(directiveFile, []byte(`prop1 1
+	err = os.WriteFile(directiveFile, []byte(`prop1 1
 	prop2 2`), 0644)
 	if err != nil {
 		t.Fatal(err)
@@ -440,6 +572,28 @@ func TestParseAll(t *testing.T) {
 
 		{`import notfound/*`, false, [][]string{}},        // glob needn't error with no matches
 		{`import notfound/file.conf`, true, [][]string{}}, // but a specific file should
+
+		// recursive self-import
+		{`import testdata/import_recursive0.txt`, true, [][]string{}},
+		{`import testdata/import_recursive3.txt
+		import testdata/import_recursive1.txt`, true, [][]string{}},
+
+		// cyclic imports
+		{`(A) {
+			import A
+		}
+		:80
+		import A
+		`, true, [][]string{}},
+		{`(A) {
+			import B
+		}
+		(B) {
+			import A
+		}
+		:80
+		import A
+		`, true, [][]string{}},
 	} {
 		p := testParser(test.input)
 		blocks, err := p.parseAll()
@@ -474,6 +628,7 @@ func TestParseAll(t *testing.T) {
 
 func TestEnvironmentReplacement(t *testing.T) {
 	os.Setenv("FOOBAR", "foobar")
+	os.Setenv("CHAINED", "$FOOBAR")
 
 	for i, test := range []struct {
 		input  string
@@ -519,6 +674,22 @@ func TestEnvironmentReplacement(t *testing.T) {
 			input:  "{$FOOBAR}{$FOOBAR}",
 			expect: "foobarfoobar",
 		},
+		{
+			input:  "{$CHAINED}",
+			expect: "$FOOBAR", // should not chain env expands
+		},
+		{
+			input:  "{$FOO:default}",
+			expect: "default",
+		},
+		{
+			input:  "foo{$BAR:bar}baz",
+			expect: "foobarbaz",
+		},
+		{
+			input:  "foo{$BAR:$FOOBAR}baz",
+			expect: "foo$FOOBARbaz", // should not chain env expands
+		},
 		{
 			input:  "{$FOOBAR",
 			expect: "{$FOOBAR",
@@ -544,16 +715,43 @@ func TestEnvironmentReplacement(t *testing.T) {
 			expect: "}{$",
 		},
 	} {
-		actual, err := replaceEnvVars([]byte(test.input))
+		actual := replaceEnvVars([]byte(test.input))
-		if err != nil {
-			t.Fatal(err)
-		}
 		if !bytes.Equal(actual, []byte(test.expect)) {
 			t.Errorf("Test %d: Expected: '%s' but got '%s'", i, test.expect, actual)
 		}
 	}
 }
 
+func TestImportReplacementInJSONWithBrace(t *testing.T) {
+	for i, test := range []struct {
+		args   []string
+		input  string
+		expect string
+	}{
+		{
+			args:   []string{"123"},
+			input:  "{args[0]}",
+			expect: "123",
+		},
+		{
+			args:   []string{"123"},
+			input:  `{"key":"{args[0]}"}`,
+			expect: `{"key":"123"}`,
+		},
+		{
+			args:   []string{"123", "123"},
+			input:  `{"key":[{args[0]},{args[1]}]}`,
+			expect: `{"key":[123,123]}`,
+		},
+	} {
+		repl := makeArgsReplacer(test.args)
+		actual := repl.ReplaceKnown(test.input, "")
+		if actual != test.expect {
+			t.Errorf("Test %d: Expected: '%s' but got '%s'", i, test.expect, actual)
+		}
+	}
+}
+
 func TestSnippets(t *testing.T) {
 	p := testParser(`
 		(common) {
@@ -568,10 +766,6 @@ func TestSnippets(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	if len(blocks) != 1 {
 		t.Fatalf("Expect exactly one server block. Got %d.", len(blocks))
 	}
@@ -590,7 +784,7 @@ func TestSnippets(t *testing.T) {
 }
 
 func writeStringToTempFileOrDie(t *testing.T, str string) (pathToFile string) {
-	file, err := ioutil.TempFile("", t.Name())
+	file, err := os.CreateTemp("", t.Name())
 	if err != nil {
 		panic(err) // get a stack trace so we know where this was called from.
 	}
@@ -616,10 +810,6 @@ func TestImportedFilesIgnoreNonDirectiveImportTokens(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	auth := blocks[0].Segments[0]
 	line := auth[0].Text + " " + auth[1].Text + " " + auth[2].Text + " " + auth[3].Text
 	if line != "basicauth / import password" {
@@ -651,10 +841,6 @@ func TestSnippetAcrossMultipleFiles(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	for _, b := range blocks {
-		t.Log(b.Keys)
-		t.Log(b.Segments)
-	}
 	if len(blocks) != 1 {
 		t.Fatalf("Expect exactly one server block. Got %d.", len(blocks))
 	}
@@ -670,5 +856,5 @@ func TestSnippetAcrossMultipleFiles(t *testing.T) {
 }
 
 func testParser(input string) parser {
-	return parser{Dispenser: newTestDispenser(input)}
+	return parser{Dispenser: NewTestDispenser(input)}
 }

caddyconfig/caddyfile/testdata/glob/.dotfile.txt

@@ -0,0 +1,4 @@
+host1 {
+	dir1
+	dir2 arg1
+}

caddyconfig/caddyfile/testdata/glob/import_test1.txt

@@ -0,0 +1,2 @@
+dir2 arg1 arg2
+dir3

caddyconfig/caddyfile/testdata/import_args0.txt

@@ -0,0 +1 @@
+{args[0]}

caddyconfig/caddyfile/testdata/import_args1.txt

@@ -0,0 +1 @@
+{args[0]} {args[1]}

caddyconfig/caddyfile/testdata/import_recursive0.txt

@@ -0,0 +1 @@
+import import_recursive0.txt

caddyconfig/caddyfile/testdata/import_recursive1.txt

@@ -0,0 +1 @@
+import import_recursive2.txt

caddyconfig/caddyfile/testdata/import_recursive2.txt

@@ -0,0 +1 @@
+import import_recursive3.txt

caddyconfig/caddyfile/testdata/import_recursive3.txt

@@ -0,0 +1 @@
+import import_recursive1.txt

caddyconfig/caddyfile/testdata/only_white_space.txt

@@ -0,0 +1,7 @@
+
+
+ 
+  
+　
+
+

caddyconfig/configadapters.go

@@ -17,12 +17,14 @@ package caddyconfig
 import (
 	"encoding/json"
 	"fmt"
+
+	"github.com/caddyserver/caddy/v2"
 )
 
 // Adapter is a type which can adapt a configuration to Caddy JSON.
 // It returns the results and any warnings, or an error.
 type Adapter interface {
-	Adapt(body []byte, options map[string]interface{}) ([]byte, []Warning, error)
+	Adapt(body []byte, options map[string]any) ([]byte, []Warning, error)
 }
 
 // Warning represents a warning or notice related to conversion.
@@ -33,12 +35,20 @@ type Warning struct {
 	Message   string `json:"message,omitempty"`
 }
 
+func (w Warning) String() string {
+	var directive string
+	if w.Directive != "" {
+		directive = fmt.Sprintf(" (%s)", w.Directive)
+	}
+	return fmt.Sprintf("%s:%d%s: %s", w.File, w.Line, directive, w.Message)
+}
+
 // JSON encodes val as JSON, returning it as a json.RawMessage. Any
 // marshaling errors (which are highly unlikely with correct code)
 // are converted to warnings. This is convenient when filling config
 // structs that require a json.RawMessage, without having to worry
 // about errors.
-func JSON(val interface{}, warnings *[]Warning) json.RawMessage {
+func JSON(val any, warnings *[]Warning) json.RawMessage {
 	b, err := json.Marshal(val)
 	if err != nil {
 		if warnings != nil {
@@ -49,15 +59,14 @@ func JSON(val interface{}, warnings *[]Warning) json.RawMessage {
 	return b
 }
 
-// JSONModuleObject is like JSON, except it marshals val into a JSON object
+// JSONModuleObject is like JSON(), except it marshals val into a JSON object
-// and then adds a key to that object named fieldName with the value fieldVal.
+// with an added key named fieldName with the value fieldVal. This is useful
-// This is useful for JSON-encoding module values where the module name has to
+// for encoding module values where the module name has to be described within
-// be described within the object by a certain key; for example,
+// the object by a certain key; for example, `"handler": "file_server"` for a
-// "responder": "file_server" for a file server HTTP responder. The val must
+// file server HTTP handler (fieldName="handler" and fieldVal="file_server").
-// encode into a map[string]interface{} (i.e. it must be a struct or map),
+// The val parameter must encode into a map[string]any (i.e. it must be
-// and any errors are converted into warnings, so this can be conveniently
+// a struct or map). Any errors are converted into warnings.
-// used when filling a struct. For correct code, there should be no errors.
+func JSONModuleObject(val any, fieldName, fieldVal string, warnings *[]Warning) json.RawMessage {
-func JSONModuleObject(val interface{}, fieldName, fieldVal string, warnings *[]Warning) json.RawMessage {
 	// encode to a JSON object first
 	enc, err := json.Marshal(val)
 	if err != nil {
@@ -68,7 +77,7 @@ func JSONModuleObject(val interface{}, fieldName, fieldVal string, warnings *[]W
 	}
 
 	// then decode the object
-	var tmp map[string]interface{}
+	var tmp map[string]any
 	err = json.Unmarshal(enc, &tmp)
 	if err != nil {
 		if warnings != nil {
@@ -92,20 +101,15 @@ func JSONModuleObject(val interface{}, fieldName, fieldVal string, warnings *[]W
 	return result
 }
 
-// JSONIndent is used to JSON-marshal the final resulting Caddy
-// configuration in a consistent, human-readable way.
-func JSONIndent(val interface{}) ([]byte, error) {
-	return json.MarshalIndent(val, "", "\t")
-}
-
 // RegisterAdapter registers a config adapter with the given name.
-// This should usually be done at init-time.
+// This should usually be done at init-time. It panics if the
-func RegisterAdapter(name string, adapter Adapter) error {
+// adapter cannot be registered successfully.
+func RegisterAdapter(name string, adapter Adapter) {
 	if _, ok := configAdapters[name]; ok {
-		return fmt.Errorf("%s: already registered", name)
+		panic(fmt.Errorf("%s: already registered", name))
 	}
 	configAdapters[name] = adapter
-	return nil
+	caddy.RegisterModule(adapterModule{name, adapter})
 }
 
 // GetAdapter returns the adapter with the given name,
@@ -114,4 +118,21 @@ func GetAdapter(name string) Adapter {
 	return configAdapters[name]
 }
 
+// adapterModule is a wrapper type that can turn any config
+// adapter into a Caddy module, which has the benefit of being
+// counted with other modules, even though they do not
+// technically extend the Caddy configuration structure.
+// See caddyserver/caddy#3132.
+type adapterModule struct {
+	name string
+	Adapter
+}
+
+func (am adapterModule) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  caddy.ModuleID("caddy.adapters." + am.name),
+		New: func() caddy.Module { return am },
+	}
+}
+
 var configAdapters = make(map[string]Adapter)

caddyconfig/httpcaddyfile/addresses.go

@@ -17,13 +17,18 @@ package httpcaddyfile
 import (
 	"fmt"
 	"net"
+	"net/netip"
 	"reflect"
+	"sort"
 	"strconv"
 	"strings"
+	"unicode"
 
+	"github.com/caddyserver/certmagic"
+
+	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
-	"github.com/mholt/certmagic"
 )
 
 // mapAddressToServerBlocks returns a map of listener address to list of server
@@ -45,7 +50,7 @@ import (
 // key of its server block (specifying the host part), and each key may have
 // a different port. And we definitely need to be sure that a site which is
 // bound to be served on a specific interface is not served on others just
-// beceause that is more convenient: it would be a potential security risk
+// because that is more convenient: it would be a potential security risk
 // if the difference between interfaces means private vs. public.
 //
 // So what this function does for the example above is iterate each server
@@ -73,7 +78,8 @@ import (
 // multiple addresses to the same lists of server blocks (a many:many mapping).
 // (Doing this is essentially a map-reduce technique.)
 func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBlock,
-	options map[string]interface{}) (map[string][]serverBlock, error) {
+	options map[string]any,
+) (map[string][]serverBlock, error) {
 	sbmap := make(map[string][]serverBlock)
 
 	for i, sblock := range originalServerBlocks {
@@ -99,18 +105,36 @@ func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBloc
 			}
 		}
 
+		// make a slice of the map keys so we can iterate in sorted order
+		addrs := make([]string, 0, len(addrToKeys))
+		for k := range addrToKeys {
+			addrs = append(addrs, k)
+		}
+		sort.Strings(addrs)
+
 		// now that we know which addresses serve which keys of this
 		// server block, we iterate that mapping and create a list of
 		// new server blocks for each address where the keys of the
 		// server block are only the ones which use the address; but
 		// the contents (tokens) are of course the same
-		for addr, keys := range addrToKeys {
+		for _, addr := range addrs {
+			keys := addrToKeys[addr]
+			// parse keys so that we only have to do it once
+			parsedKeys := make([]Address, 0, len(keys))
+			for _, key := range keys {
+				addr, err := ParseAddress(key)
+				if err != nil {
+					return nil, fmt.Errorf("parsing key '%s': %v", key, err)
+				}
+				parsedKeys = append(parsedKeys, addr.Normalize())
+			}
 			sbmap[addr] = append(sbmap[addr], serverBlock{
 				block: caddyfile.ServerBlock{
 					Keys:     keys,
 					Segments: sblock.block.Segments,
 				},
 				pile: sblock.pile,
+				keys: parsedKeys,
 			})
 		}
 	}
@@ -127,7 +151,7 @@ func (st *ServerType) mapAddressToServerBlocks(originalServerBlocks []serverBloc
 // association from multiple addresses to multiple server blocks; i.e. each element of
 // the returned slice) becomes a server definition in the output JSON.
 func (st *ServerType) consolidateAddrMappings(addrToServerBlocks map[string][]serverBlock) []sbAddrAssociation {
-	var sbaddrs []sbAddrAssociation
+	sbaddrs := make([]sbAddrAssociation, 0, len(addrToServerBlocks))
 	for addr, sblocks := range addrToServerBlocks {
 		// we start with knowing that at least this address
 		// maps to these server blocks
@@ -148,23 +172,45 @@ func (st *ServerType) consolidateAddrMappings(addrToServerBlocks map[string][]se
 				delete(addrToServerBlocks, otherAddr)
 			}
 		}
+		sort.Strings(a.addresses)
 
 		sbaddrs = append(sbaddrs, a)
 	}
+
+	// sort them by their first address (we know there will always be at least one)
+	// to avoid problems with non-deterministic ordering (makes tests flaky)
+	sort.Slice(sbaddrs, func(i, j int) bool {
+		return sbaddrs[i].addresses[0] < sbaddrs[j].addresses[0]
+	})
+
 	return sbaddrs
 }
 
+// listenerAddrsForServerBlockKey essentially converts the Caddyfile
+// site addresses to Caddy listener addresses for each server block.
 func (st *ServerType) listenerAddrsForServerBlockKey(sblock serverBlock, key string,
-	options map[string]interface{}) ([]string, error) {
+	options map[string]any,
+) ([]string, error) {
 	addr, err := ParseAddress(key)
 	if err != nil {
 		return nil, fmt.Errorf("parsing key: %v", err)
 	}
 	addr = addr.Normalize()
 
+	switch addr.Scheme {
+	case "wss":
+		return nil, fmt.Errorf("the scheme wss:// is only supported in browsers; use https:// instead")
+	case "ws":
+		return nil, fmt.Errorf("the scheme ws:// is only supported in browsers; use http:// instead")
+	case "https", "http", "":
+		// Do nothing or handle the valid schemes
+	default:
+		return nil, fmt.Errorf("unsupported URL scheme %s://", addr.Scheme)
+	}
+
 	// figure out the HTTP and HTTPS ports; either
 	// use defaults, or override with user config
-	httpPort, httpsPort := strconv.Itoa(certmagic.HTTPPort), strconv.Itoa(certmagic.HTTPSPort)
+	httpPort, httpsPort := strconv.Itoa(caddyhttp.DefaultHTTPPort), strconv.Itoa(caddyhttp.DefaultHTTPSPort)
 	if hport, ok := options["http_port"]; ok {
 		httpPort = strconv.Itoa(hport.(int))
 	}
@@ -172,20 +218,14 @@ func (st *ServerType) listenerAddrsForServerBlockKey(sblock serverBlock, key str
 		httpsPort = strconv.Itoa(hsport.(int))
 	}
 
-	lnPort := DefaultPort
+	// default port is the HTTPS port
+	lnPort := httpsPort
 	if addr.Port != "" {
 		// port explicitly defined
 		lnPort = addr.Port
-	} else if addr.Scheme != "" {
+	} else if addr.Scheme == "http" {
 		// port inferred from scheme
-		if addr.Scheme == "http" {
 		lnPort = httpPort
-		} else if addr.Scheme == "https" {
-			lnPort = httpsPort
-		}
-	} else if certmagic.HostQualifies(addr.Host) {
-		// automatic HTTPS
-		lnPort = httpsPort
 	}
 
 	// error if scheme and port combination violate convention
@@ -193,27 +233,50 @@ func (st *ServerType) listenerAddrsForServerBlockKey(sblock serverBlock, key str
 		return nil, fmt.Errorf("[%s] scheme and port violate convention", key)
 	}
 
-	// the bind directive specifies hosts, but is optional
+	// the bind directive specifies hosts (and potentially network), but is optional
-	var lnHosts []string
+	lnHosts := make([]string, 0, len(sblock.pile["bind"]))
 	for _, cfgVal := range sblock.pile["bind"] {
 		lnHosts = append(lnHosts, cfgVal.Value.([]string)...)
 	}
 	if len(lnHosts) == 0 {
+		if defaultBind, ok := options["default_bind"].([]string); ok {
+			lnHosts = defaultBind
+		} else {
 			lnHosts = []string{""}
 		}
+	}
 
 	// use a map to prevent duplication
 	listeners := make(map[string]struct{})
-	for _, host := range lnHosts {
+	for _, lnHost := range lnHosts {
-		listeners[net.JoinHostPort(host, lnPort)] = struct{}{}
+		// normally we would simply append the port,
+		// but if lnHost is IPv6, we need to ensure it
+		// is enclosed in [ ]; net.JoinHostPort does
+		// this for us, but lnHost might also have a
+		// network type in front (e.g. "tcp/") leading
+		// to "[tcp/::1]" which causes parsing failures
+		// later; what we need is "tcp/[::1]", so we have
+		// to split the network and host, then re-combine
+		network, host, ok := strings.Cut(lnHost, "/")
+		if !ok {
+			host = network
+			network = ""
+		}
+		host = strings.Trim(host, "[]") // IPv6
+		networkAddr := caddy.JoinNetworkAddress(network, host, lnPort)
+		addr, err := caddy.ParseNetworkAddress(networkAddr)
+		if err != nil {
+			return nil, fmt.Errorf("parsing network address: %v", err)
+		}
+		listeners[addr.String()] = struct{}{}
 	}
 
 	// now turn map into list
-	var listenersList []string
+	listenersList := make([]string, 0, len(listeners))
 	for lnStr := range listeners {
 		listenersList = append(listenersList, lnStr)
 	}
-	// sort.Strings(listenersList) // TODO: is sorting necessary?
+	sort.Strings(listenersList)
 
 	return listenersList, nil
 }
@@ -281,8 +344,6 @@ func ParseAddress(str string) (Address, error) {
 	return a, nil
 }
 
-// TODO: which of the methods on Address are even used?
-
 // String returns a human-readable form of a. It will
 // be a cleaned-up and filled-out URL string.
 func (a Address) String() string {
@@ -317,50 +378,48 @@ func (a Address) String() string {
 // Normalize returns a normalized version of a.
 func (a Address) Normalize() Address {
 	path := a.Path
-	if !caseSensitivePath {
-		path = strings.ToLower(path)
-	}
 
 	// ensure host is normalized if it's an IP address
-	host := a.Host
+	host := strings.TrimSpace(a.Host)
-	if ip := net.ParseIP(host); ip != nil {
+	if ip, err := netip.ParseAddr(host); err == nil {
+		if ip.Is6() && !ip.Is4() && !ip.Is4In6() {
 			host = ip.String()
 		}
+	}
 
 	return Address{
 		Original: a.Original,
-		Scheme:   strings.ToLower(a.Scheme),
+		Scheme:   lowerExceptPlaceholders(a.Scheme),
-		Host:     strings.ToLower(host),
+		Host:     lowerExceptPlaceholders(host),
 		Port:     a.Port,
 		Path:     path,
 	}
 }
 
-// Key returns a string form of a, much like String() does, but this
+// lowerExceptPlaceholders lowercases s except within
-// method doesn't add anything default that wasn't in the original.
+// placeholders (substrings in non-escaped '{ }' spans).
-func (a Address) Key() string {
+// See https://github.com/caddyserver/caddy/issues/3264
-	res := ""
+func lowerExceptPlaceholders(s string) string {
-	if a.Scheme != "" {
+	var sb strings.Builder
-		res += a.Scheme + "://"
+	var escaped, inPlaceholder bool
+	for _, ch := range s {
+		if ch == '\\' && !escaped {
+			escaped = true
+			sb.WriteRune(ch)
+			continue
 		}
-	if a.Host != "" {
+		if ch == '{' && !escaped {
-		res += a.Host
+			inPlaceholder = true
 		}
-	// insert port only if the original has its own explicit port
+		if ch == '}' && inPlaceholder && !escaped {
-	if a.Port != "" &&
+			inPlaceholder = false
-		len(a.Original) >= len(res) &&
-		strings.HasPrefix(a.Original[len(res):], ":"+a.Port) {
-		res += ":" + a.Port
 		}
-	if a.Path != "" {
+		if inPlaceholder {
-		res += a.Path
+			sb.WriteRune(ch)
+		} else {
+			sb.WriteRune(unicode.ToLower(ch))
 		}
-	return res
+		escaped = false
+	}
+	return sb.String()
 }
-
-const (
-	// DefaultPort is the default port to use.
-	DefaultPort = "2015"
-
-	caseSensitivePath = false // TODO: Used?
-)

caddyconfig/httpcaddyfile/addresses_fuzz.go

@@ -12,8 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// +build gofuzz
+//go:build gofuzz
-// +build gofuzz_libfuzzer
 
 package httpcaddyfile
 

caddyconfig/httpcaddyfile/addresses_test.go

@@ -1,7 +1,6 @@
 package httpcaddyfile
 
 import (
-	"strings"
 	"testing"
 )
 
@@ -107,47 +106,128 @@ func TestAddressString(t *testing.T) {
 func TestKeyNormalization(t *testing.T) {
 	testCases := []struct {
 		input  string
-		expect string
+		expect Address
 	}{
+		{
+			input: "example.com",
+			expect: Address{
+				Host: "example.com",
+			},
+		},
 		{
 			input: "http://host:1234/path",
-			expect: "http://host:1234/path",
+			expect: Address{
+				Scheme: "http",
+				Host:   "host",
+				Port:   "1234",
+				Path:   "/path",
+			},
 		},
 		{
 			input: "HTTP://A/ABCDEF",
-			expect: "http://a/ABCDEF",
+			expect: Address{
+				Scheme: "http",
+				Host:   "a",
+				Path:   "/ABCDEF",
+			},
 		},
 		{
 			input: "A/ABCDEF",
-			expect: "a/ABCDEF",
+			expect: Address{
+				Host: "a",
+				Path: "/ABCDEF",
+			},
 		},
 		{
 			input: "A:2015/Path",
-			expect: "a:2015/Path",
+			expect: Address{
+				Host: "a",
+				Port: "2015",
+				Path: "/Path",
+			},
+		},
+		{
+			input: "sub.{env.MY_DOMAIN}",
+			expect: Address{
+				Host: "sub.{env.MY_DOMAIN}",
+			},
+		},
+		{
+			input: "sub.ExAmPle",
+			expect: Address{
+				Host: "sub.example",
+			},
+		},
+		{
+			input: "sub.\\{env.MY_DOMAIN\\}",
+			expect: Address{
+				Host: "sub.\\{env.my_domain\\}",
+			},
+		},
+		{
+			input: "sub.{env.MY_DOMAIN}.com",
+			expect: Address{
+				Host: "sub.{env.MY_DOMAIN}.com",
+			},
 		},
 		{
 			input: ":80",
-			expect: ":80",
+			expect: Address{
+				Port: "80",
+			},
 		},
 		{
 			input: ":443",
-			expect: ":443",
+			expect: Address{
+				Port: "443",
+			},
 		},
 		{
 			input: ":1234",
-			expect: ":1234",
+			expect: Address{
+				Port: "1234",
+			},
 		},
 		{
 			input:  "",
-			expect: "",
+			expect: Address{},
 		},
 		{
 			input:  ":",
-			expect: "",
+			expect: Address{},
 		},
 		{
 			input: "[::]",
-			expect: "::",
+			expect: Address{
+				Host: "::",
+			},
+		},
+		{
+			input: "127.0.0.1",
+			expect: Address{
+				Host: "127.0.0.1",
+			},
+		},
+		{
+			input: "[2001:db8:85a3:8d3:1319:8a2e:370:7348]:1234",
+			expect: Address{
+				Host: "2001:db8:85a3:8d3:1319:8a2e:370:7348",
+				Port: "1234",
+			},
+		},
+		{
+			// IPv4 address in IPv6 form (#4381)
+			input: "[::ffff:cff4:e77d]:1234",
+			expect: Address{
+				Host: "::ffff:cff4:e77d",
+				Port: "1234",
+			},
+		},
+		{
+			input: "::ffff:cff4:e77d",
+			expect: Address{
+				Host: "::ffff:cff4:e77d",
+			},
 		},
 	}
 	for i, tc := range testCases {
@@ -156,16 +236,18 @@ func TestKeyNormalization(t *testing.T) {
 			t.Errorf("Test %d: Parsing address '%s': %v", i, tc.input, err)
 			continue
 		}
-		expect := tc.expect
+		actual := addr.Normalize()
-		if !caseSensitivePath {
+		if actual.Scheme != tc.expect.Scheme {
-			// every other part of the address should be lowercased when normalized,
+			t.Errorf("Test %d: Input '%s': Expected Scheme='%s' but got Scheme='%s'", i, tc.input, tc.expect.Scheme, actual.Scheme)
-			// so simply lower-case the whole thing to do case-insensitive comparison
-			// of the path as well
-			expect = strings.ToLower(expect)
 		}
-		if actual := addr.Normalize().Key(); actual != expect {
+		if actual.Host != tc.expect.Host {
-			t.Errorf("Test %d: Normalized key for address '%s' was '%s' but expected '%s'", i, tc.input, actual, expect)
+			t.Errorf("Test %d: Input '%s': Expected Host='%s' but got Host='%s'", i, tc.input, tc.expect.Host, actual.Host)
+		}
+		if actual.Port != tc.expect.Port {
+			t.Errorf("Test %d: Input '%s': Expected Port='%s' but got Port='%s'", i, tc.input, tc.expect.Port, actual.Port)
+		}
+		if actual.Path != tc.expect.Path {
+			t.Errorf("Test %d: Input '%s': Expected Path='%s' but got Path='%s'", i, tc.input, tc.expect.Path, actual.Path)
 		}
-
 	}
 }

caddyconfig/httpcaddyfile/builtins_test.go

@@ -0,0 +1,353 @@
+package httpcaddyfile
+
+import (
+	"strings"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	_ "github.com/caddyserver/caddy/v2/modules/logging"
+)
+
+func TestLogDirectiveSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		output      string
+		expectError bool
+	}{
+		{
+			input: `:8080 {
+				log
+			}
+			`,
+			output:      `{"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{}}}}}}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log {
+					output file foo.log
+				}
+			}
+			`,
+			output:      `{"logging":{"logs":{"default":{"exclude":["http.log.access.log0"]},"log0":{"writer":{"filename":"foo.log","output":"file"},"include":["http.log.access.log0"]}}},"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{"default_logger_name":"log0"}}}}}}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log {
+					format filter {
+						wrap console
+						fields {
+							request>remote_ip ip_mask {
+								ipv4 24
+								ipv6 32
+							}
+						}
+					}
+				}
+			}
+			`,
+			output:      `{"logging":{"logs":{"default":{"exclude":["http.log.access.log0"]},"log0":{"encoder":{"fields":{"request\u003eremote_ip":{"filter":"ip_mask","ipv4_cidr":24,"ipv6_cidr":32}},"format":"filter","wrap":{"format":"console"}},"include":["http.log.access.log0"]}}},"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{"default_logger_name":"log0"}}}}}}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				log name-override {
+					output file foo.log
+				}
+			}
+			`,
+			output:      `{"logging":{"logs":{"default":{"exclude":["http.log.access.name-override"]},"name-override":{"writer":{"filename":"foo.log","output":"file"},"include":["http.log.access.name-override"]}}},"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"logs":{"default_logger_name":"name-override"}}}}}}`,
+			expectError: false,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		out, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+
+		if string(out) != tc.output {
+			t.Errorf("Test %d error output mismatch Expected: %s, got %s", i, tc.output, out)
+		}
+	}
+}
+
+func TestRedirDirectiveSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectError bool
+	}{
+		{
+			input: `:8080 {
+				redir :8081
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir * :8081
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /api/* :8081 300
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir :8081 300
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /api/* :8081 399
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir :8081 399
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html temporary
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir https://example.com{uri} permanent
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html permanent
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html html
+			}`,
+			expectError: false,
+		},
+		{
+			// this is now allowed so a Location header
+			// can be written and consumed by JS
+			// in the case of XHR requests
+			input: `:8080 {
+				redir * :8081 401
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 402
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 {http.reverse_proxy.status_code}
+			}`,
+			expectError: false,
+		},
+		{
+			input: `:8080 {
+				redir /old.html /new.html htlm
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 200
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 temp
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 perm
+			}`,
+			expectError: true,
+		},
+		{
+			input: `:8080 {
+				redir * :8081 php
+			}`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+	}
+}
+
+func TestImportErrorLine(t *testing.T) {
+	for i, tc := range []struct {
+		input     string
+		errorFunc func(err error) bool
+	}{
+		{
+			input: `(t1) {
+					abort {args[:]}
+				}
+				:8080 {
+					import t1
+					import t1 true
+				}`,
+			errorFunc: func(err error) bool {
+				return err != nil && strings.Contains(err.Error(), "Caddyfile:6 (import t1)")
+			},
+		},
+		{
+			input: `(t1) {
+					abort {args[:]}
+				}
+				:8080 {
+					import t1 true
+				}`,
+			errorFunc: func(err error) bool {
+				return err != nil && strings.Contains(err.Error(), "Caddyfile:5 (import t1)")
+			},
+		},
+		{
+			input: `
+				import testdata/import_variadic_snippet.txt
+				:8080 {
+					import t1 true
+				}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+		{
+			input: `
+				import testdata/import_variadic_with_import.txt
+				:8080 {
+					import t1 true
+					import t2 true
+				}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+	} {
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if !tc.errorFunc(err) {
+			t.Errorf("Test %d error expectation failed, got %s", i, err)
+			continue
+		}
+	}
+}
+
+func TestNestedImport(t *testing.T) {
+	for i, tc := range []struct {
+		input     string
+		errorFunc func(err error) bool
+	}{
+		{
+			input: `(t1) {
+						respond {args[0]} {args[1]}
+					}
+					
+					(t2) {
+						import t1 {args[0]} 202
+					}
+					
+					:8080 {
+						handle {
+							import t2 "foobar"
+						}
+					}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+		{
+			input: `(t1) {
+						respond {args[:]}
+					}
+					
+					(t2) {
+						import t1 {args[0]} {args[1]}
+					}
+					
+					:8080 {
+						handle {
+							import t2 "foobar" 202
+						}
+					}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+		{
+			input: `(t1) {
+						respond {args[0]} {args[1]}
+					}
+					
+					(t2) {
+						import t1 {args[:]}
+					}
+					
+					:8080 {
+						handle {
+							import t2 "foobar" 202
+						}
+					}`,
+			errorFunc: func(err error) bool {
+				return err == nil
+			},
+		},
+	} {
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if !tc.errorFunc(err) {
+			t.Errorf("Test %d error expectation failed, got %s", i, err)
+			continue
+		}
+	}
+}

caddyconfig/httpcaddyfile/directives.go

@@ -16,7 +16,10 @@ package httpcaddyfile
 
 import (
 	"encoding/json"
+	"net"
 	"sort"
+	"strconv"
+	"strings"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
@@ -26,30 +29,57 @@ import (
 
 // directiveOrder specifies the order
 // to apply directives in HTTP routes.
+//
+// The root directive goes first in case rewrites or
+// redirects depend on existence of files, i.e. the
+// file matcher, which must know the root first.
+//
+// The header directive goes second so that headers
+// can be manipulated before doing redirects.
 var directiveOrder = []string{
-	"redir",
+	"tracing",
-	"rewrite",
 
+	"map",
+	"vars",
 	"root",
+	"skip_log",
 
-	"strip_prefix",
+	"header",
-	"strip_suffix",
+	"copy_response_headers", // only in reverse_proxy's handle_response
-	"uri_replace",
+	"request_body",
+
+	"redir",
+
+	// incoming request manipulation
+	"method",
+	"rewrite",
+	"uri",
 	"try_files",
 
+	// middleware handlers; some wrap responses
 	"basicauth",
-	"header",
+	"forward_auth",
 	"request_header",
 	"encode",
+	"push",
 	"templates",
 
+	// special routing & dispatching directives
+	"invoke",
 	"handle",
+	"handle_path",
 	"route",
 
+	// handlers that typically respond to requests
+	"abort",
+	"error",
+	"copy_response", // only in reverse_proxy's handle_response
 	"respond",
+	"metrics",
 	"reverse_proxy",
 	"php_fastcgi",
 	"file_server",
+	"acme_server",
 }
 
 // directiveIsOrdered returns true if dir is
@@ -84,20 +114,11 @@ func RegisterHandlerDirective(dir string, setupFunc UnmarshalHandlerFunc) {
 			return nil, h.ArgErr()
 		}
 
-		matcherSet, ok, err := h.MatcherToken()
+		matcherSet, err := h.ExtractMatcherSet()
 		if err != nil {
 			return nil, err
 		}
-		if ok {
-			// strip matcher token; we don't need to
-			// use the return value here because a
-			// new dispenser should have been made
-			// solely for this directive's tokens,
-			// with no other uses of same slice
-			h.Dispenser.Delete()
-		}
 
-		h.Dispenser.Reset() // pretend this lookahead never happened
 		val, err := setupFunc(h)
 		if err != nil {
 			return nil, err
@@ -107,11 +128,24 @@ func RegisterHandlerDirective(dir string, setupFunc UnmarshalHandlerFunc) {
 	})
 }
 
+// RegisterGlobalOption registers a unique global option opt with
+// an associated unmarshaling (setup) function. When the global
+// option opt is encountered in a Caddyfile, setupFunc will be
+// called to unmarshal its tokens.
+func RegisterGlobalOption(opt string, setupFunc UnmarshalGlobalFunc) {
+	if _, ok := registeredGlobalOptions[opt]; ok {
+		panic("global option " + opt + " already registered")
+	}
+	registeredGlobalOptions[opt] = setupFunc
+}
+
 // Helper is a type which helps setup a value from
 // Caddyfile tokens.
 type Helper struct {
 	*caddyfile.Dispenser
-	options      map[string]interface{}
+	// State stores intermediate variables during caddyfile adaptation.
+	State        map[string]any
+	options      map[string]any
 	warnings     *[]caddyconfig.Warning
 	matcherDefs  map[string]caddy.ModuleMap
 	parentBlock  caddyfile.ServerBlock
@@ -119,7 +153,7 @@ type Helper struct {
 }
 
 // Option gets the option keyed by name.
-func (h Helper) Option(name string) interface{} {
+func (h Helper) Option(name string) any {
 	return h.options[name]
 }
 
@@ -139,11 +173,12 @@ func (h Helper) Caddyfiles() []string {
 	for file := range files {
 		filesSlice = append(filesSlice, file)
 	}
+	sort.Strings(filesSlice)
 	return filesSlice
 }
 
 // JSON converts val into JSON. Any errors are added to warnings.
-func (h Helper) JSON(val interface{}) json.RawMessage {
+func (h Helper) JSON(val any) json.RawMessage {
 	return caddyconfig.JSON(val, h.warnings)
 }
 
@@ -158,9 +193,32 @@ func (h Helper) MatcherToken() (caddy.ModuleMap, bool, error) {
 	return matcherSetFromMatcherToken(h.Dispenser.Token(), h.matcherDefs, h.warnings)
 }
 
+// ExtractMatcherSet is like MatcherToken, except this is a higher-level
+// method that returns the matcher set described by the matcher token,
+// or nil if there is none, and deletes the matcher token from the
+// dispenser and resets it as if this look-ahead never happened. Useful
+// when wrapping a route (one or more handlers) in a user-defined matcher.
+func (h Helper) ExtractMatcherSet() (caddy.ModuleMap, error) {
+	matcherSet, hasMatcher, err := h.MatcherToken()
+	if err != nil {
+		return nil, err
+	}
+	if hasMatcher {
+		// strip matcher token; we don't need to
+		// use the return value here because a
+		// new dispenser should have been made
+		// solely for this directive's tokens,
+		// with no other uses of same slice
+		h.Dispenser.Delete()
+	}
+	h.Dispenser.Reset() // pretend this lookahead never happened
+	return matcherSet, nil
+}
+
 // NewRoute returns config values relevant to creating a new HTTP route.
 func (h Helper) NewRoute(matcherSet caddy.ModuleMap,
-	handler caddyhttp.MiddlewareHandler) []ConfigValue {
+	handler caddyhttp.MiddlewareHandler,
+) []ConfigValue {
 	mod, err := caddy.GetModule(caddy.GetModuleID(handler))
 	if err != nil {
 		*h.warnings = append(*h.warnings, caddyconfig.Warning{
@@ -218,6 +276,94 @@ func (h Helper) NewBindAddresses(addrs []string) []ConfigValue {
 	return []ConfigValue{{Class: "bind", Value: addrs}}
 }
 
+// WithDispenser returns a new instance based on d. All others Helper
+// fields are copied, so typically maps are shared with this new instance.
+func (h Helper) WithDispenser(d *caddyfile.Dispenser) Helper {
+	h.Dispenser = d
+	return h
+}
+
+// ParseSegmentAsSubroute parses the segment such that its subdirectives
+// are themselves treated as directives, from which a subroute is built
+// and returned.
+func ParseSegmentAsSubroute(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	allResults, err := parseSegmentAsConfig(h)
+	if err != nil {
+		return nil, err
+	}
+
+	return buildSubroute(allResults, h.groupCounter, true)
+}
+
+// parseSegmentAsConfig parses the segment such that its subdirectives
+// are themselves treated as directives, including named matcher definitions,
+// and the raw Config structs are returned.
+func parseSegmentAsConfig(h Helper) ([]ConfigValue, error) {
+	var allResults []ConfigValue
+
+	for h.Next() {
+		// don't allow non-matcher args on the first line
+		if h.NextArg() {
+			return nil, h.ArgErr()
+		}
+
+		// slice the linear list of tokens into top-level segments
+		var segments []caddyfile.Segment
+		for nesting := h.Nesting(); h.NextBlock(nesting); {
+			segments = append(segments, h.NextSegment())
+		}
+
+		// copy existing matcher definitions so we can augment
+		// new ones that are defined only in this scope
+		matcherDefs := make(map[string]caddy.ModuleMap, len(h.matcherDefs))
+		for key, val := range h.matcherDefs {
+			matcherDefs[key] = val
+		}
+
+		// find and extract any embedded matcher definitions in this scope
+		for i := 0; i < len(segments); i++ {
+			seg := segments[i]
+			if strings.HasPrefix(seg.Directive(), matcherPrefix) {
+				// parse, then add the matcher to matcherDefs
+				err := parseMatcherDefinitions(caddyfile.NewDispenser(seg), matcherDefs)
+				if err != nil {
+					return nil, err
+				}
+				// remove the matcher segment (consumed), then step back the loop
+				segments = append(segments[:i], segments[i+1:]...)
+				i--
+			}
+		}
+
+		// with matchers ready to go, evaluate each directive's segment
+		for _, seg := range segments {
+			dir := seg.Directive()
+			dirFunc, ok := registeredDirectives[dir]
+			if !ok {
+				return nil, h.Errf("unrecognized directive: %s - are you sure your Caddyfile structure (nesting and braces) is correct?", dir)
+			}
+
+			subHelper := h
+			subHelper.Dispenser = caddyfile.NewDispenser(seg)
+			subHelper.matcherDefs = matcherDefs
+
+			results, err := dirFunc(subHelper)
+			if err != nil {
+				return nil, h.Errf("parsing caddyfile tokens for '%s': %v", dir, err)
+			}
+
+			dir = normalizeDirectiveName(dir)
+
+			for _, result := range results {
+				result.directive = dir
+				allResults = append(allResults, result)
+			}
+		}
+	}
+
+	return allResults, nil
+}
+
 // ConfigValue represents a value to be added to the final
 // configuration, or a value to be consulted when building
 // the final configuration.
@@ -233,7 +379,7 @@ type ConfigValue struct {
 	// The value to be used when building the config.
 	// Generally its type is associated with the
 	// name of the Class.
-	Value interface{}
+	Value any
 
 	directive string
 }
@@ -244,16 +390,17 @@ func sortRoutes(routes []ConfigValue) {
 		dirPositions[dir] = i
 	}
 
-	// while we are sorting, we will need to decode a route's path matcher
-	// in order to sub-sort by path length; we can amortize this operation
-	// for efficiency by storing the decoded matchers in a slice
-	decodedMatchers := make([]caddyhttp.MatchPath, len(routes))
-
 	sort.SliceStable(routes, func(i, j int) bool {
+		// if the directives are different, just use the established directive order
 		iDir, jDir := routes[i].directive, routes[j].directive
-		if iDir == jDir {
+		if iDir != jDir {
-			// directives are the same; sub-sort by path matcher length
+			return dirPositions[iDir] < dirPositions[jDir]
-			// if there's only one matcher set and one path (common case)
+		}
+
+		// directives are the same; sub-sort by path matcher length if there's
+		// only one matcher set and one path (this is a very common case and
+		// usually -- but not always -- helpful/expected, oh well; user can
+		// always take manual control of order using handler or route blocks)
 		iRoute, ok := routes[i].Value.(caddyhttp.Route)
 		if !ok {
 			return false
@@ -263,40 +410,152 @@ func sortRoutes(routes []ConfigValue) {
 			return false
 		}
 
-			if len(iRoute.MatcherSetsRaw) == 1 && len(jRoute.MatcherSetsRaw) == 1 {
+		// decode the path matchers if there is just one matcher set
-				// use already-decoded matcher, or decode if it's the first time seeing it
+		var iPM, jPM caddyhttp.MatchPath
-				iPM, jPM := decodedMatchers[i], decodedMatchers[j]
+		if len(iRoute.MatcherSetsRaw) == 1 {
-				if iPM == nil {
+			_ = json.Unmarshal(iRoute.MatcherSetsRaw[0]["path"], &iPM)
-					var pathMatcher caddyhttp.MatchPath
-					_ = json.Unmarshal(iRoute.MatcherSetsRaw[0]["path"], &pathMatcher)
-					decodedMatchers[i] = pathMatcher
-					iPM = pathMatcher
 		}
-				if jPM == nil {
+		if len(jRoute.MatcherSetsRaw) == 1 {
-					var pathMatcher caddyhttp.MatchPath
+			_ = json.Unmarshal(jRoute.MatcherSetsRaw[0]["path"], &jPM)
-					_ = json.Unmarshal(jRoute.MatcherSetsRaw[0]["path"], &pathMatcher)
-					decodedMatchers[j] = pathMatcher
-					jPM = pathMatcher
 		}
 
-				// if there is only one path in the matcher, sort by
+		// if there is only one path in the path matcher, sort by longer path
-				// longer path (more specific) first
+		// (more specific) first; missing path matchers or multi-matchers are
-				if len(iPM) == 1 && len(jPM) == 1 {
+		// treated as zero-length paths
-					return len(iPM[0]) > len(jPM[0])
+		var iPathLen, jPathLen int
-				}
+		if len(iPM) == 1 {
+			iPathLen = len(iPM[0])
 		}
+		if len(jPM) == 1 {
+			jPathLen = len(jPM[0])
 		}
 
-		return dirPositions[iDir] < dirPositions[jDir]
+		sortByPath := func() bool {
+			// we can only confidently compare path lengths if both
+			// directives have a single path to match (issue #5037)
+			if iPathLen > 0 && jPathLen > 0 {
+				// if both paths are the same except for a trailing wildcard,
+				// sort by the shorter path first (which is more specific)
+				if strings.TrimSuffix(iPM[0], "*") == strings.TrimSuffix(jPM[0], "*") {
+					return iPathLen < jPathLen
+				}
+
+				// sort most-specific (longest) path first
+				return iPathLen > jPathLen
+			}
+
+			// if both directives don't have a single path to compare,
+			// sort whichever one has a matcher first; if both have
+			// a matcher, sort equally (stable sort preserves order)
+			return len(iRoute.MatcherSetsRaw) > 0 && len(jRoute.MatcherSetsRaw) == 0
+		}()
+
+		// some directives involve setting values which can overwrite
+		// each other, so it makes most sense to reverse the order so
+		// that the least-specific matcher is first, allowing the last
+		// matching one to win
+		if iDir == "vars" {
+			return !sortByPath
+		}
+
+		// everything else is most-specific matcher first
+		return sortByPath
 	})
 }
 
-// serverBlock pairs a Caddyfile server block
+// serverBlock pairs a Caddyfile server block with
-// with a "pile" of config values, keyed by class
+// a "pile" of config values, keyed by class name,
-// name.
+// as well as its parsed keys for convenience.
 type serverBlock struct {
 	block caddyfile.ServerBlock
 	pile  map[string][]ConfigValue // config values obtained from directives
+	keys  []Address
+}
+
+// hostsFromKeys returns a list of all the non-empty hostnames found in
+// the keys of the server block sb. If logger mode is false, a key with
+// an empty hostname portion will return an empty slice, since that
+// server block is interpreted to effectively match all hosts. An empty
+// string is never added to the slice.
+//
+// If loggerMode is true, then the non-standard ports of keys will be
+// joined to the hostnames. This is to effectively match the Host
+// header of requests that come in for that key.
+//
+// The resulting slice is not sorted but will never have duplicates.
+func (sb serverBlock) hostsFromKeys(loggerMode bool) []string {
+	// ensure each entry in our list is unique
+	hostMap := make(map[string]struct{})
+	for _, addr := range sb.keys {
+		if addr.Host == "" {
+			if !loggerMode {
+				// server block contains a key like ":443", i.e. the host portion
+				// is empty / catch-all, which means to match all hosts
+				return []string{}
+			}
+			// never append an empty string
+			continue
+		}
+		if loggerMode &&
+			addr.Port != "" &&
+			addr.Port != strconv.Itoa(caddyhttp.DefaultHTTPPort) &&
+			addr.Port != strconv.Itoa(caddyhttp.DefaultHTTPSPort) {
+			hostMap[net.JoinHostPort(addr.Host, addr.Port)] = struct{}{}
+		} else {
+			hostMap[addr.Host] = struct{}{}
+		}
+	}
+
+	// convert map to slice
+	sblockHosts := make([]string, 0, len(hostMap))
+	for host := range hostMap {
+		sblockHosts = append(sblockHosts, host)
+	}
+
+	return sblockHosts
+}
+
+func (sb serverBlock) hostsFromKeysNotHTTP(httpPort string) []string {
+	// ensure each entry in our list is unique
+	hostMap := make(map[string]struct{})
+	for _, addr := range sb.keys {
+		if addr.Host == "" {
+			continue
+		}
+		if addr.Scheme != "http" && addr.Port != httpPort {
+			hostMap[addr.Host] = struct{}{}
+		}
+	}
+
+	// convert map to slice
+	sblockHosts := make([]string, 0, len(hostMap))
+	for host := range hostMap {
+		sblockHosts = append(sblockHosts, host)
+	}
+
+	return sblockHosts
+}
+
+// hasHostCatchAllKey returns true if sb has a key that
+// omits a host portion, i.e. it "catches all" hosts.
+func (sb serverBlock) hasHostCatchAllKey() bool {
+	for _, addr := range sb.keys {
+		if addr.Host == "" {
+			return true
+		}
+	}
+	return false
+}
+
+// isAllHTTP returns true if all sb keys explicitly specify
+// the http:// scheme
+func (sb serverBlock) isAllHTTP() bool {
+	for _, addr := range sb.keys {
+		if addr.Scheme != "http" {
+			return false
+		}
+	}
+	return true
 }
 
 type (
@@ -315,6 +574,14 @@ type (
 	// for you. These are passed to a call to
 	// RegisterHandlerDirective.
 	UnmarshalHandlerFunc func(h Helper) (caddyhttp.MiddlewareHandler, error)
+
+	// UnmarshalGlobalFunc is a function which can unmarshal Caddyfile
+	// tokens from a global option. It is passed the tokens to parse and
+	// existing value from the previous instance of this global option
+	// (if any). It returns the value to associate with this global option.
+	UnmarshalGlobalFunc func(d *caddyfile.Dispenser, existingVal any) (any, error)
 )
 
 var registeredDirectives = make(map[string]UnmarshalFunc)
+
+var registeredGlobalOptions = make(map[string]UnmarshalGlobalFunc)

caddyconfig/httpcaddyfile/directives_test.go

@@ -0,0 +1,94 @@
+package httpcaddyfile
+
+import (
+	"reflect"
+	"sort"
+	"testing"
+)
+
+func TestHostsFromKeys(t *testing.T) {
+	for i, tc := range []struct {
+		keys             []Address
+		expectNormalMode []string
+		expectLoggerMode []string
+	}{
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+			},
+			[]string{"foo"},
+			[]string{"foo"},
+		},
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+				{Original: "bar", Host: "bar"},
+			},
+			[]string{"bar", "foo"},
+			[]string{"bar", "foo"},
+		},
+		{
+			[]Address{
+				{Original: ":2015", Port: "2015"},
+			},
+			[]string{}, []string{},
+		},
+		{
+			[]Address{
+				{Original: ":443", Port: "443"},
+			},
+			[]string{}, []string{},
+		},
+		{
+			[]Address{
+				{Original: "foo", Host: "foo"},
+				{Original: ":2015", Port: "2015"},
+			},
+			[]string{}, []string{"foo"},
+		},
+		{
+			[]Address{
+				{Original: "example.com:2015", Host: "example.com", Port: "2015"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com:2015"},
+		},
+		{
+			[]Address{
+				{Original: "example.com:80", Host: "example.com", Port: "80"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com"},
+		},
+		{
+			[]Address{
+				{Original: "https://:2015/foo", Scheme: "https", Port: "2015", Path: "/foo"},
+			},
+			[]string{},
+			[]string{},
+		},
+		{
+			[]Address{
+				{Original: "https://example.com:2015/foo", Scheme: "https", Host: "example.com", Port: "2015", Path: "/foo"},
+			},
+			[]string{"example.com"},
+			[]string{"example.com:2015"},
+		},
+	} {
+		sb := serverBlock{keys: tc.keys}
+
+		// test in normal mode
+		actual := sb.hostsFromKeys(false)
+		sort.Strings(actual)
+		if !reflect.DeepEqual(tc.expectNormalMode, actual) {
+			t.Errorf("Test %d (loggerMode=false): Expected: %v Actual: %v", i, tc.expectNormalMode, actual)
+		}
+
+		// test in logger mode
+		actual = sb.hostsFromKeys(true)
+		sort.Strings(actual)
+		if !reflect.DeepEqual(tc.expectLoggerMode, actual) {
+			t.Errorf("Test %d (loggerMode=true): Expected: %v Actual: %v", i, tc.expectLoggerMode, actual)
+		}
+	}
+}

caddyconfig/httpcaddyfile/httptype_test.go

@@ -1,6 +1,78 @@
 package httpcaddyfile
 
-import "testing"
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+)
+
+func TestMatcherSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectError bool
+	}{
+		{
+			input: `http://localhost
+			@debug {
+				query showdebug=1
+			}
+			`,
+			expectError: false,
+		},
+		{
+			input: `http://localhost
+			@debug {
+				query bad format
+			}
+			`,
+			expectError: true,
+		},
+		{
+			input: `http://localhost
+			@debug {
+				not {
+					path /somepath*
+				}
+			}
+			`,
+			expectError: false,
+		},
+		{
+			input: `http://localhost
+			@debug {
+				not path /somepath*
+			}
+			`,
+			expectError: false,
+		},
+		{
+			input: `http://localhost
+			@debug not path /somepath*
+			`,
+			expectError: false,
+		},
+		{
+			input: `@matcher {
+				path /matcher-not-allowed/outside-of-site-block/*
+			}
+			http://localhost
+			`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+	}
+}
 
 func TestSpecificity(t *testing.T) {
 	for i, tc := range []struct {
@@ -31,3 +103,109 @@ func TestSpecificity(t *testing.T) {
 		}
 	}
 }
+
+func TestGlobalOptions(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		expectError bool
+	}{
+		{
+			input: `
+				{
+					email test@example.com
+				}
+				:80
+			`,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin off
+				}
+				:80
+			`,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 127.0.0.1:2020
+				}
+				:80
+			`,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin {
+						disabled false
+					}
+				}
+				:80
+			`,
+			expectError: true,
+		},
+		{
+			input: `
+				{
+					admin {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 127.0.0.1:2020 {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectError: false,
+		},
+		{
+			input: `
+				{
+					admin 192.168.1.1:2020 127.0.0.1:2020 {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectError: true,
+		},
+		{
+			input: `
+				{
+					admin off {
+						enforce_origin
+						origins 192.168.1.1:2020 127.0.0.1:2020
+					}
+				}
+				:80
+			`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		_, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
+			continue
+		}
+	}
+}

caddyconfig/httpcaddyfile/options.go

@@ -15,14 +15,53 @@
 package httpcaddyfile
 
 import (
-	"fmt"
 	"strconv"
 
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
+
 	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	"github.com/caddyserver/caddy/v2/modules/caddytls"
 )
 
-func parseOptHTTPPort(d *caddyfile.Dispenser) (int, error) {
+func init() {
+	RegisterGlobalOption("debug", parseOptTrue)
+	RegisterGlobalOption("http_port", parseOptHTTPPort)
+	RegisterGlobalOption("https_port", parseOptHTTPSPort)
+	RegisterGlobalOption("default_bind", parseOptStringList)
+	RegisterGlobalOption("grace_period", parseOptDuration)
+	RegisterGlobalOption("shutdown_delay", parseOptDuration)
+	RegisterGlobalOption("default_sni", parseOptSingleString)
+	RegisterGlobalOption("fallback_sni", parseOptSingleString)
+	RegisterGlobalOption("order", parseOptOrder)
+	RegisterGlobalOption("storage", parseOptStorage)
+	RegisterGlobalOption("storage_clean_interval", parseOptDuration)
+	RegisterGlobalOption("renew_interval", parseOptDuration)
+	RegisterGlobalOption("ocsp_interval", parseOptDuration)
+	RegisterGlobalOption("acme_ca", parseOptSingleString)
+	RegisterGlobalOption("acme_ca_root", parseOptSingleString)
+	RegisterGlobalOption("acme_dns", parseOptACMEDNS)
+	RegisterGlobalOption("acme_eab", parseOptACMEEAB)
+	RegisterGlobalOption("cert_issuer", parseOptCertIssuer)
+	RegisterGlobalOption("skip_install_trust", parseOptTrue)
+	RegisterGlobalOption("email", parseOptSingleString)
+	RegisterGlobalOption("admin", parseOptAdmin)
+	RegisterGlobalOption("on_demand_tls", parseOptOnDemand)
+	RegisterGlobalOption("local_certs", parseOptTrue)
+	RegisterGlobalOption("key_type", parseOptSingleString)
+	RegisterGlobalOption("auto_https", parseOptAutoHTTPS)
+	RegisterGlobalOption("servers", parseServerOptions)
+	RegisterGlobalOption("ocsp_stapling", parseOCSPStaplingOptions)
+	RegisterGlobalOption("log", parseLogOptions)
+	RegisterGlobalOption("preferred_chains", parseOptPreferredChains)
+	RegisterGlobalOption("persist_config", parseOptPersistConfig)
+}
+
+func parseOptTrue(d *caddyfile.Dispenser, _ any) (any, error) { return true, nil }
+
+func parseOptHTTPPort(d *caddyfile.Dispenser, _ any) (any, error) {
 	var httpPort int
 	for d.Next() {
 		var httpPortStr string
@@ -38,7 +77,7 @@ func parseOptHTTPPort(d *caddyfile.Dispenser) (int, error) {
 	return httpPort, nil
 }
 
-func parseOptHTTPSPort(d *caddyfile.Dispenser) (int, error) {
+func parseOptHTTPSPort(d *caddyfile.Dispenser, _ any) (any, error) {
 	var httpsPort int
 	for d.Next() {
 		var httpsPortStr string
@@ -54,11 +93,7 @@ func parseOptHTTPSPort(d *caddyfile.Dispenser) (int, error) {
 	return httpsPort, nil
 }
 
-func parseOptExperimentalHTTP3(d *caddyfile.Dispenser) (bool, error) {
+func parseOptOrder(d *caddyfile.Dispenser, _ any) (any, error) {
-	return true, nil
-}
-
-func parseOptOrder(d *caddyfile.Dispenser) ([]string, error) {
 	newOrder := directiveOrder
 
 	for d.Next() {
@@ -68,7 +103,7 @@ func parseOptOrder(d *caddyfile.Dispenser) ([]string, error) {
 		}
 		dirName := d.Val()
 		if _, ok := registeredDirectives[dirName]; !ok {
-			return nil, fmt.Errorf("%s is not a registered directive", dirName)
+			return nil, d.Errf("%s is not a registered directive", dirName)
 		}
 
 		// get positional token
@@ -104,7 +139,7 @@ func parseOptOrder(d *caddyfile.Dispenser) ([]string, error) {
 		case "before":
 		case "after":
 		default:
-			return nil, fmt.Errorf("unknown positional '%s'", pos)
+			return nil, d.Errf("unknown positional '%s'", pos)
 		}
 
 		// get name of other directive
@@ -134,35 +169,110 @@ func parseOptOrder(d *caddyfile.Dispenser) ([]string, error) {
 	return newOrder, nil
 }
 
-func parseOptStorage(d *caddyfile.Dispenser) (caddy.StorageConverter, error) {
+func parseOptStorage(d *caddyfile.Dispenser, _ any) (any, error) {
-	if !d.Next() {
+	if !d.Next() { // consume option name
 		return nil, d.ArgErr()
 	}
-	args := d.RemainingArgs()
+	if !d.Next() { // get storage module name
-	if len(args) != 1 {
 		return nil, d.ArgErr()
 	}
-	modName := args[0]
+	modID := "caddy.storage." + d.Val()
-	mod, err := caddy.GetModule("caddy.storage." + modName)
+	unm, err := caddyfile.UnmarshalModule(d, modID)
-	if err != nil {
-		return nil, fmt.Errorf("getting storage module '%s': %v", modName, err)
-	}
-	unm, ok := mod.New().(caddyfile.Unmarshaler)
-	if !ok {
-		return nil, fmt.Errorf("storage module '%s' is not a Caddyfile unmarshaler", mod.ID)
-	}
-	err = unm.UnmarshalCaddyfile(d.NewFromNextTokens())
 	if err != nil {
 		return nil, err
 	}
 	storage, ok := unm.(caddy.StorageConverter)
 	if !ok {
-		return nil, fmt.Errorf("module %s is not a StorageConverter", mod.ID)
+		return nil, d.Errf("module %s is not a caddy.StorageConverter", modID)
 	}
 	return storage, nil
 }
 
-func parseOptACMECA(d *caddyfile.Dispenser) (string, error) {
+func parseOptDuration(d *caddyfile.Dispenser, _ any) (any, error) {
+	if !d.Next() { // consume option name
+		return nil, d.ArgErr()
+	}
+	if !d.Next() { // get duration value
+		return nil, d.ArgErr()
+	}
+	dur, err := caddy.ParseDuration(d.Val())
+	if err != nil {
+		return nil, err
+	}
+	return caddy.Duration(dur), nil
+}
+
+func parseOptACMEDNS(d *caddyfile.Dispenser, _ any) (any, error) {
+	if !d.Next() { // consume option name
+		return nil, d.ArgErr()
+	}
+	if !d.Next() { // get DNS module name
+		return nil, d.ArgErr()
+	}
+	modID := "dns.providers." + d.Val()
+	unm, err := caddyfile.UnmarshalModule(d, modID)
+	if err != nil {
+		return nil, err
+	}
+	prov, ok := unm.(certmagic.ACMEDNSProvider)
+	if !ok {
+		return nil, d.Errf("module %s (%T) is not a certmagic.ACMEDNSProvider", modID, unm)
+	}
+	return prov, nil
+}
+
+func parseOptACMEEAB(d *caddyfile.Dispenser, _ any) (any, error) {
+	eab := new(acme.EAB)
+	for d.Next() {
+		if d.NextArg() {
+			return nil, d.ArgErr()
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "key_id":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				eab.KeyID = d.Val()
+
+			case "mac_key":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				eab.MACKey = d.Val()
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
+		}
+	}
+	return eab, nil
+}
+
+func parseOptCertIssuer(d *caddyfile.Dispenser, existing any) (any, error) {
+	var issuers []certmagic.Issuer
+	if existing != nil {
+		issuers = existing.([]certmagic.Issuer)
+	}
+	for d.Next() { // consume option name
+		if !d.Next() { // get issuer module name
+			return nil, d.ArgErr()
+		}
+		modID := "tls.issuance." + d.Val()
+		unm, err := caddyfile.UnmarshalModule(d, modID)
+		if err != nil {
+			return nil, err
+		}
+		iss, ok := unm.(certmagic.Issuer)
+		if !ok {
+			return nil, d.Errf("module %s (%T) is not a certmagic.Issuer", modID, unm)
+		}
+		issuers = append(issuers, iss)
+	}
+	return issuers, nil
+}
+
+func parseOptSingleString(d *caddyfile.Dispenser, _ any) (any, error) {
 	d.Next() // consume parameter name
 	if !d.Next() {
 		return "", d.ArgErr()
@@ -174,7 +284,112 @@ func parseOptACMECA(d *caddyfile.Dispenser) (string, error) {
 	return val, nil
 }
 
-func parseOptEmail(d *caddyfile.Dispenser) (string, error) {
+func parseOptStringList(d *caddyfile.Dispenser, _ any) (any, error) {
+	d.Next() // consume parameter name
+	val := d.RemainingArgs()
+	if len(val) == 0 {
+		return "", d.ArgErr()
+	}
+	return val, nil
+}
+
+func parseOptAdmin(d *caddyfile.Dispenser, _ any) (any, error) {
+	adminCfg := new(caddy.AdminConfig)
+	for d.Next() {
+		if d.NextArg() {
+			listenAddress := d.Val()
+			if listenAddress == "off" {
+				adminCfg.Disabled = true
+				if d.Next() { // Do not accept any remaining options including block
+					return nil, d.Err("No more option is allowed after turning off admin config")
+				}
+			} else {
+				adminCfg.Listen = listenAddress
+				if d.NextArg() { // At most 1 arg is allowed
+					return nil, d.ArgErr()
+				}
+			}
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "enforce_origin":
+				adminCfg.EnforceOrigin = true
+
+			case "origins":
+				adminCfg.Origins = d.RemainingArgs()
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
+		}
+	}
+	if adminCfg.Listen == "" && !adminCfg.Disabled {
+		adminCfg.Listen = caddy.DefaultAdminListen
+	}
+	return adminCfg, nil
+}
+
+func parseOptOnDemand(d *caddyfile.Dispenser, _ any) (any, error) {
+	var ond *caddytls.OnDemandConfig
+	for d.Next() {
+		if d.NextArg() {
+			return nil, d.ArgErr()
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "ask":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				if ond == nil {
+					ond = new(caddytls.OnDemandConfig)
+				}
+				ond.Ask = d.Val()
+
+			case "interval":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				dur, err := caddy.ParseDuration(d.Val())
+				if err != nil {
+					return nil, err
+				}
+				if ond == nil {
+					ond = new(caddytls.OnDemandConfig)
+				}
+				if ond.RateLimit == nil {
+					ond.RateLimit = new(caddytls.RateLimit)
+				}
+				ond.RateLimit.Interval = caddy.Duration(dur)
+
+			case "burst":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				burst, err := strconv.Atoi(d.Val())
+				if err != nil {
+					return nil, err
+				}
+				if ond == nil {
+					ond = new(caddytls.OnDemandConfig)
+				}
+				if ond.RateLimit == nil {
+					ond.RateLimit = new(caddytls.RateLimit)
+				}
+				ond.RateLimit.Burst = burst
+
+			default:
+				return nil, d.Errf("unrecognized parameter '%s'", d.Val())
+			}
+		}
+	}
+	if ond == nil {
+		return nil, d.Err("expected at least one config parameter for on_demand_tls")
+	}
+	return ond, nil
+}
+
+func parseOptPersistConfig(d *caddyfile.Dispenser, _ any) (any, error) {
 	d.Next() // consume parameter name
 	if !d.Next() {
 		return "", d.ArgErr()
@@ -183,19 +398,93 @@ func parseOptEmail(d *caddyfile.Dispenser) (string, error) {
 	if d.Next() {
 		return "", d.ArgErr()
 	}
+	if val != "off" {
+		return "", d.Errf("persist_config must be 'off'")
+	}
 	return val, nil
 }
 
-func parseOptAdmin(d *caddyfile.Dispenser) (string, error) {
+func parseOptAutoHTTPS(d *caddyfile.Dispenser, _ any) (any, error) {
+	d.Next() // consume parameter name
+	if !d.Next() {
+		return "", d.ArgErr()
+	}
+	val := d.Val()
 	if d.Next() {
-		var listenAddress string
+		return "", d.ArgErr()
-		d.AllArgs(&listenAddress)
-
-		if listenAddress == "" {
-			listenAddress = caddy.DefaultAdminListen
 	}
-
+	if val != "off" && val != "disable_redirects" && val != "disable_certs" && val != "ignore_loaded_certs" {
-		return listenAddress, nil
+		return "", d.Errf("auto_https must be one of 'off', 'disable_redirects', 'disable_certs', or 'ignore_loaded_certs'")
 	}
-	return "", nil
+	return val, nil
+}
+
+func parseServerOptions(d *caddyfile.Dispenser, _ any) (any, error) {
+	return unmarshalCaddyfileServerOptions(d)
+}
+
+func parseOCSPStaplingOptions(d *caddyfile.Dispenser, _ any) (any, error) {
+	d.Next() // consume option name
+	var val string
+	if !d.AllArgs(&val) {
+		return nil, d.ArgErr()
+	}
+	if val != "off" {
+		return nil, d.Errf("invalid argument '%s'", val)
+	}
+	return certmagic.OCSPConfig{
+		DisableStapling: val == "off",
+	}, nil
+}
+
+// parseLogOptions parses the global log option. Syntax:
+//
+//	log [name] {
+//	    output  <writer_module> ...
+//	    format  <encoder_module> ...
+//	    level   <level>
+//	    include <namespaces...>
+//	    exclude <namespaces...>
+//	}
+//
+// When the name argument is unspecified, this directive modifies the default
+// logger.
+func parseLogOptions(d *caddyfile.Dispenser, existingVal any) (any, error) {
+	currentNames := make(map[string]struct{})
+	if existingVal != nil {
+		innerVals, ok := existingVal.([]ConfigValue)
+		if !ok {
+			return nil, d.Errf("existing log values of unexpected type: %T", existingVal)
+		}
+		for _, rawVal := range innerVals {
+			val, ok := rawVal.Value.(namedCustomLog)
+			if !ok {
+				return nil, d.Errf("existing log value of unexpected type: %T", existingVal)
+			}
+			currentNames[val.name] = struct{}{}
+		}
+	}
+
+	var warnings []caddyconfig.Warning
+	// Call out the same parser that handles server-specific log configuration.
+	configValues, err := parseLogHelper(
+		Helper{
+			Dispenser: d,
+			warnings:  &warnings,
+		},
+		currentNames,
+	)
+	if err != nil {
+		return nil, err
+	}
+	if len(warnings) > 0 {
+		return nil, d.Errf("warnings found in parsing global log options: %+v", warnings)
+	}
+
+	return configValues, nil
+}
+
+func parseOptPreferredChains(d *caddyfile.Dispenser, _ any) (any, error) {
+	d.Next()
+	return caddytls.ParseCaddyfilePreferredChainsOptions(d)
 }

caddyconfig/httpcaddyfile/options_test.go

@@ -0,0 +1,64 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	_ "github.com/caddyserver/caddy/v2/modules/logging"
+)
+
+func TestGlobalLogOptionSyntax(t *testing.T) {
+	for i, tc := range []struct {
+		input       string
+		output      string
+		expectError bool
+	}{
+		// NOTE: Additional test cases of successful Caddyfile parsing
+		// are present in: caddytest/integration/caddyfile_adapt/
+		{
+			input: `{
+				log default
+			}
+			`,
+			output:      `{}`,
+			expectError: false,
+		},
+		{
+			input: `{
+				log example {
+					output file foo.log
+				}
+				log example {
+					format json
+				}
+			}
+			`,
+			expectError: true,
+		},
+		{
+			input: `{
+				log example /foo {
+					output file foo.log
+				}
+			}
+			`,
+			expectError: true,
+		},
+	} {
+
+		adapter := caddyfile.Adapter{
+			ServerType: ServerType{},
+		}
+
+		out, _, err := adapter.Adapt([]byte(tc.input), nil)
+
+		if err != nil != tc.expectError {
+			t.Errorf("Test %d error expectation failed Expected: %v, got %v", i, tc.expectError, err)
+			continue
+		}
+
+		if string(out) != tc.output {
+			t.Errorf("Test %d error output mismatch Expected: %s, got %s", i, tc.output, out)
+		}
+	}
+}

caddyconfig/httpcaddyfile/parser_test.go

@@ -1,66 +0,0 @@
-// Copyright 2015 Matthew Holt and The Caddy Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package httpcaddyfile
-
-import (
-	"testing"
-
-	caddyfile "github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
-)
-
-func TestParse(t *testing.T) {
-
-	for i, tc := range []struct {
-		input       string
-		expectWarn  bool
-		expectError bool
-	}{
-		{
-			input: `http://localhost
-			@debug {
-			  query showdebug=1
-			}
-			`,
-			expectWarn:  false,
-			expectError: false,
-		},
-		{
-			input: `http://localhost
-			@debug {
-			  query bad format
-			}
-			`,
-			expectWarn:  false,
-			expectError: true,
-		},
-	} {
-
-		adapter := caddyfile.Adapter{
-			ServerType: ServerType{},
-		}
-
-		_, warnings, err := adapter.Adapt([]byte(tc.input), nil)
-
-		if len(warnings) > 0 != tc.expectWarn {
-			t.Errorf("Test %d warning expectation failed Expected: %v, got %v", i, tc.expectWarn, warnings)
-			continue
-		}
-
-		if err != nil != tc.expectError {
-			t.Errorf("Test %d error expectation failed Expected: %v, got %s", i, tc.expectError, err)
-			continue
-		}
-	}
-}

caddyconfig/httpcaddyfile/pkiapp.go

@@ -0,0 +1,229 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package httpcaddyfile
+
+import (
+	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	"github.com/caddyserver/caddy/v2/modules/caddypki"
+)
+
+func init() {
+	RegisterGlobalOption("pki", parsePKIApp)
+}
+
+// parsePKIApp parses the global log option. Syntax:
+//
+//	pki {
+//	    ca [<id>] {
+//	        name                  <name>
+//	        root_cn               <name>
+//	        intermediate_cn       <name>
+//	        intermediate_lifetime <duration>
+//	        root {
+//	            cert   <path>
+//	            key    <path>
+//	            format <format>
+//	        }
+//	        intermediate {
+//	            cert   <path>
+//	            key    <path>
+//	            format <format>
+//	        }
+//	    }
+//	}
+//
+// When the CA ID is unspecified, 'local' is assumed.
+func parsePKIApp(d *caddyfile.Dispenser, existingVal any) (any, error) {
+	pki := &caddypki.PKI{CAs: make(map[string]*caddypki.CA)}
+
+	for d.Next() {
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "ca":
+				pkiCa := new(caddypki.CA)
+				if d.NextArg() {
+					pkiCa.ID = d.Val()
+					if d.NextArg() {
+						return nil, d.ArgErr()
+					}
+				}
+				if pkiCa.ID == "" {
+					pkiCa.ID = caddypki.DefaultCAID
+				}
+
+				for nesting := d.Nesting(); d.NextBlock(nesting); {
+					switch d.Val() {
+					case "name":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						pkiCa.Name = d.Val()
+
+					case "root_cn":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						pkiCa.RootCommonName = d.Val()
+
+					case "intermediate_cn":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						pkiCa.IntermediateCommonName = d.Val()
+
+					case "intermediate_lifetime":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, err
+						}
+						pkiCa.IntermediateLifetime = caddy.Duration(dur)
+
+					case "root":
+						if pkiCa.Root == nil {
+							pkiCa.Root = new(caddypki.KeyPair)
+						}
+						for nesting := d.Nesting(); d.NextBlock(nesting); {
+							switch d.Val() {
+							case "cert":
+								if !d.NextArg() {
+									return nil, d.ArgErr()
+								}
+								pkiCa.Root.Certificate = d.Val()
+
+							case "key":
+								if !d.NextArg() {
+									return nil, d.ArgErr()
+								}
+								pkiCa.Root.PrivateKey = d.Val()
+
+							case "format":
+								if !d.NextArg() {
+									return nil, d.ArgErr()
+								}
+								pkiCa.Root.Format = d.Val()
+
+							default:
+								return nil, d.Errf("unrecognized pki ca root option '%s'", d.Val())
+							}
+						}
+
+					case "intermediate":
+						if pkiCa.Intermediate == nil {
+							pkiCa.Intermediate = new(caddypki.KeyPair)
+						}
+						for nesting := d.Nesting(); d.NextBlock(nesting); {
+							switch d.Val() {
+							case "cert":
+								if !d.NextArg() {
+									return nil, d.ArgErr()
+								}
+								pkiCa.Intermediate.Certificate = d.Val()
+
+							case "key":
+								if !d.NextArg() {
+									return nil, d.ArgErr()
+								}
+								pkiCa.Intermediate.PrivateKey = d.Val()
+
+							case "format":
+								if !d.NextArg() {
+									return nil, d.ArgErr()
+								}
+								pkiCa.Intermediate.Format = d.Val()
+
+							default:
+								return nil, d.Errf("unrecognized pki ca intermediate option '%s'", d.Val())
+							}
+						}
+
+					default:
+						return nil, d.Errf("unrecognized pki ca option '%s'", d.Val())
+					}
+				}
+
+				pki.CAs[pkiCa.ID] = pkiCa
+
+			default:
+				return nil, d.Errf("unrecognized pki option '%s'", d.Val())
+			}
+		}
+	}
+
+	return pki, nil
+}
+
+func (st ServerType) buildPKIApp(
+	pairings []sbAddrAssociation,
+	options map[string]any,
+	warnings []caddyconfig.Warning,
+) (*caddypki.PKI, []caddyconfig.Warning, error) {
+	skipInstallTrust := false
+	if _, ok := options["skip_install_trust"]; ok {
+		skipInstallTrust = true
+	}
+	falseBool := false
+
+	// Load the PKI app configured via global options
+	var pkiApp *caddypki.PKI
+	unwrappedPki, ok := options["pki"].(*caddypki.PKI)
+	if ok {
+		pkiApp = unwrappedPki
+	} else {
+		pkiApp = &caddypki.PKI{CAs: make(map[string]*caddypki.CA)}
+	}
+	for _, ca := range pkiApp.CAs {
+		if skipInstallTrust {
+			ca.InstallTrust = &falseBool
+		}
+		pkiApp.CAs[ca.ID] = ca
+	}
+
+	// Add in the CAs configured via directives
+	for _, p := range pairings {
+		for _, sblock := range p.serverBlocks {
+			// find all the CAs that were defined and add them to the app config
+			// i.e. from any "acme_server" directives
+			for _, caCfgValue := range sblock.pile["pki.ca"] {
+				ca := caCfgValue.Value.(*caddypki.CA)
+				if skipInstallTrust {
+					ca.InstallTrust = &falseBool
+				}
+
+				// the CA might already exist from global options, so
+				// don't overwrite it in that case
+				if _, ok := pkiApp.CAs[ca.ID]; !ok {
+					pkiApp.CAs[ca.ID] = ca
+				}
+			}
+		}
+	}
+
+	// if there was no CAs defined in any of the servers,
+	// and we were requested to not install trust, then
+	// add one for the default/local CA to do so
+	if len(pkiApp.CAs) == 0 && skipInstallTrust {
+		ca := new(caddypki.CA)
+		ca.ID = caddypki.DefaultCAID
+		ca.InstallTrust = &falseBool
+		pkiApp.CAs[ca.ID] = ca
+	}
+
+	return pkiApp, warnings, nil
+}

caddyconfig/httpcaddyfile/serveroptions.go

@@ -0,0 +1,363 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package httpcaddyfile
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/dustin/go-humanize"
+
+	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
+)
+
+// serverOptions collects server config overrides parsed from Caddyfile global options
+type serverOptions struct {
+	// If set, will only apply these options to servers that contain a
+	// listener address that matches exactly. If empty, will apply to all
+	// servers that were not already matched by another serverOptions.
+	ListenerAddress string
+
+	// These will all map 1:1 to the caddyhttp.Server struct
+	Name                 string
+	ListenerWrappersRaw  []json.RawMessage
+	ReadTimeout          caddy.Duration
+	ReadHeaderTimeout    caddy.Duration
+	WriteTimeout         caddy.Duration
+	IdleTimeout          caddy.Duration
+	KeepAliveInterval    caddy.Duration
+	MaxHeaderBytes       int
+	EnableFullDuplex     bool
+	Protocols            []string
+	StrictSNIHost        *bool
+	TrustedProxiesRaw    json.RawMessage
+	ClientIPHeaders      []string
+	ShouldLogCredentials bool
+	Metrics              *caddyhttp.Metrics
+}
+
+func unmarshalCaddyfileServerOptions(d *caddyfile.Dispenser) (any, error) {
+	serverOpts := serverOptions{}
+	for d.Next() {
+		if d.NextArg() {
+			serverOpts.ListenerAddress = d.Val()
+			if d.NextArg() {
+				return nil, d.ArgErr()
+			}
+		}
+		for nesting := d.Nesting(); d.NextBlock(nesting); {
+			switch d.Val() {
+			case "name":
+				if serverOpts.ListenerAddress == "" {
+					return nil, d.Errf("cannot set a name for a server without a listener address")
+				}
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				serverOpts.Name = d.Val()
+
+			case "listener_wrappers":
+				for nesting := d.Nesting(); d.NextBlock(nesting); {
+					modID := "caddy.listeners." + d.Val()
+					unm, err := caddyfile.UnmarshalModule(d, modID)
+					if err != nil {
+						return nil, err
+					}
+					listenerWrapper, ok := unm.(caddy.ListenerWrapper)
+					if !ok {
+						return nil, fmt.Errorf("module %s (%T) is not a listener wrapper", modID, unm)
+					}
+					jsonListenerWrapper := caddyconfig.JSONModuleObject(
+						listenerWrapper,
+						"wrapper",
+						listenerWrapper.(caddy.Module).CaddyModule().ID.Name(),
+						nil,
+					)
+					serverOpts.ListenerWrappersRaw = append(serverOpts.ListenerWrappersRaw, jsonListenerWrapper)
+				}
+
+			case "timeouts":
+				for nesting := d.Nesting(); d.NextBlock(nesting); {
+					switch d.Val() {
+					case "read_body":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, d.Errf("parsing read_body timeout duration: %v", err)
+						}
+						serverOpts.ReadTimeout = caddy.Duration(dur)
+
+					case "read_header":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, d.Errf("parsing read_header timeout duration: %v", err)
+						}
+						serverOpts.ReadHeaderTimeout = caddy.Duration(dur)
+
+					case "write":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, d.Errf("parsing write timeout duration: %v", err)
+						}
+						serverOpts.WriteTimeout = caddy.Duration(dur)
+
+					case "idle":
+						if !d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						dur, err := caddy.ParseDuration(d.Val())
+						if err != nil {
+							return nil, d.Errf("parsing idle timeout duration: %v", err)
+						}
+						serverOpts.IdleTimeout = caddy.Duration(dur)
+
+					default:
+						return nil, d.Errf("unrecognized timeouts option '%s'", d.Val())
+					}
+				}
+			case "keepalive_interval":
+				if !d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				dur, err := caddy.ParseDuration(d.Val())
+				if err != nil {
+					return nil, d.Errf("parsing keepalive interval duration: %v", err)
+				}
+				serverOpts.KeepAliveInterval = caddy.Duration(dur)
+
+			case "max_header_size":
+				var sizeStr string
+				if !d.AllArgs(&sizeStr) {
+					return nil, d.ArgErr()
+				}
+				size, err := humanize.ParseBytes(sizeStr)
+				if err != nil {
+					return nil, d.Errf("parsing max_header_size: %v", err)
+				}
+				serverOpts.MaxHeaderBytes = int(size)
+
+			case "enable_full_duplex":
+				if d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				serverOpts.EnableFullDuplex = true
+
+			case "log_credentials":
+				if d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				serverOpts.ShouldLogCredentials = true
+
+			case "protocols":
+				protos := d.RemainingArgs()
+				for _, proto := range protos {
+					if proto != "h1" && proto != "h2" && proto != "h2c" && proto != "h3" {
+						return nil, d.Errf("unknown protocol '%s': expected h1, h2, h2c, or h3", proto)
+					}
+					if sliceContains(serverOpts.Protocols, proto) {
+						return nil, d.Errf("protocol %s specified more than once", proto)
+					}
+					serverOpts.Protocols = append(serverOpts.Protocols, proto)
+				}
+				if nesting := d.Nesting(); d.NextBlock(nesting) {
+					return nil, d.ArgErr()
+				}
+
+			case "strict_sni_host":
+				if d.NextArg() && d.Val() != "insecure_off" && d.Val() != "on" {
+					return nil, d.Errf("strict_sni_host only supports 'on' or 'insecure_off', got '%s'", d.Val())
+				}
+				boolVal := true
+				if d.Val() == "insecure_off" {
+					boolVal = false
+				}
+				serverOpts.StrictSNIHost = &boolVal
+
+			case "trusted_proxies":
+				if !d.NextArg() {
+					return nil, d.Err("trusted_proxies expects an IP range source module name as its first argument")
+				}
+				modID := "http.ip_sources." + d.Val()
+				unm, err := caddyfile.UnmarshalModule(d, modID)
+				if err != nil {
+					return nil, err
+				}
+				source, ok := unm.(caddyhttp.IPRangeSource)
+				if !ok {
+					return nil, fmt.Errorf("module %s (%T) is not an IP range source", modID, unm)
+				}
+				jsonSource := caddyconfig.JSONModuleObject(
+					source,
+					"source",
+					source.(caddy.Module).CaddyModule().ID.Name(),
+					nil,
+				)
+				serverOpts.TrustedProxiesRaw = jsonSource
+
+			case "client_ip_headers":
+				headers := d.RemainingArgs()
+				for _, header := range headers {
+					if sliceContains(serverOpts.ClientIPHeaders, header) {
+						return nil, d.Errf("client IP header %s specified more than once", header)
+					}
+					serverOpts.ClientIPHeaders = append(serverOpts.ClientIPHeaders, header)
+				}
+				if nesting := d.Nesting(); d.NextBlock(nesting) {
+					return nil, d.ArgErr()
+				}
+
+			case "metrics":
+				if d.NextArg() {
+					return nil, d.ArgErr()
+				}
+				if nesting := d.Nesting(); d.NextBlock(nesting) {
+					return nil, d.ArgErr()
+				}
+				serverOpts.Metrics = new(caddyhttp.Metrics)
+
+			// TODO: DEPRECATED. (August 2022)
+			case "protocol":
+				caddy.Log().Named("caddyfile").Warn("DEPRECATED: protocol sub-option will be removed soon")
+
+				for nesting := d.Nesting(); d.NextBlock(nesting); {
+					switch d.Val() {
+					case "allow_h2c":
+						caddy.Log().Named("caddyfile").Warn("DEPRECATED: allow_h2c will be removed soon; use protocols option instead")
+
+						if d.NextArg() {
+							return nil, d.ArgErr()
+						}
+						if sliceContains(serverOpts.Protocols, "h2c") {
+							return nil, d.Errf("protocol h2c already specified")
+						}
+						serverOpts.Protocols = append(serverOpts.Protocols, "h2c")
+
+					case "strict_sni_host":
+						caddy.Log().Named("caddyfile").Warn("DEPRECATED: protocol > strict_sni_host in this position will be removed soon; move up to the servers block instead")
+
+						if d.NextArg() && d.Val() != "insecure_off" && d.Val() != "on" {
+							return nil, d.Errf("strict_sni_host only supports 'on' or 'insecure_off', got '%s'", d.Val())
+						}
+						boolVal := true
+						if d.Val() == "insecure_off" {
+							boolVal = false
+						}
+						serverOpts.StrictSNIHost = &boolVal
+
+					default:
+						return nil, d.Errf("unrecognized protocol option '%s'", d.Val())
+					}
+				}
+
+			default:
+				return nil, d.Errf("unrecognized servers option '%s'", d.Val())
+			}
+		}
+	}
+	return serverOpts, nil
+}
+
+// applyServerOptions sets the server options on the appropriate servers
+func applyServerOptions(
+	servers map[string]*caddyhttp.Server,
+	options map[string]any,
+	warnings *[]caddyconfig.Warning,
+) error {
+	serverOpts, ok := options["servers"].([]serverOptions)
+	if !ok {
+		return nil
+	}
+
+	// check for duplicate names, which would clobber the config
+	existingNames := map[string]bool{}
+	for _, opts := range serverOpts {
+		if opts.Name == "" {
+			continue
+		}
+		if existingNames[opts.Name] {
+			return fmt.Errorf("cannot use duplicate server name '%s'", opts.Name)
+		}
+		existingNames[opts.Name] = true
+	}
+
+	// collect the server name overrides
+	nameReplacements := map[string]string{}
+
+	for key, server := range servers {
+		// find the options that apply to this server
+		opts := func() *serverOptions {
+			for _, entry := range serverOpts {
+				if entry.ListenerAddress == "" {
+					return &entry
+				}
+				for _, listener := range server.Listen {
+					if entry.ListenerAddress == listener {
+						return &entry
+					}
+				}
+			}
+			return nil
+		}()
+
+		// if none apply, then move to the next server
+		if opts == nil {
+			continue
+		}
+
+		// set all the options
+		server.ListenerWrappersRaw = opts.ListenerWrappersRaw
+		server.ReadTimeout = opts.ReadTimeout
+		server.ReadHeaderTimeout = opts.ReadHeaderTimeout
+		server.WriteTimeout = opts.WriteTimeout
+		server.IdleTimeout = opts.IdleTimeout
+		server.KeepAliveInterval = opts.KeepAliveInterval
+		server.MaxHeaderBytes = opts.MaxHeaderBytes
+		server.EnableFullDuplex = opts.EnableFullDuplex
+		server.Protocols = opts.Protocols
+		server.StrictSNIHost = opts.StrictSNIHost
+		server.TrustedProxiesRaw = opts.TrustedProxiesRaw
+		server.ClientIPHeaders = opts.ClientIPHeaders
+		server.Metrics = opts.Metrics
+		if opts.ShouldLogCredentials {
+			if server.Logs == nil {
+				server.Logs = &caddyhttp.ServerLogConfig{}
+			}
+			server.Logs.ShouldLogCredentials = opts.ShouldLogCredentials
+		}
+
+		if opts.Name != "" {
+			nameReplacements[key] = opts.Name
+		}
+	}
+
+	// rename the servers if marked to do so
+	for old, new := range nameReplacements {
+		servers[new] = servers[old]
+		delete(servers, old)
+	}
+
+	return nil
+}

caddyconfig/httpcaddyfile/shorthands.go

@@ -0,0 +1,93 @@
+package httpcaddyfile
+
+import (
+	"regexp"
+	"strings"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
+)
+
+type ComplexShorthandReplacer struct {
+	search  *regexp.Regexp
+	replace string
+}
+
+type ShorthandReplacer struct {
+	complex []ComplexShorthandReplacer
+	simple  *strings.Replacer
+}
+
+func NewShorthandReplacer() ShorthandReplacer {
+	// replace shorthand placeholders (which are convenient
+	// when writing a Caddyfile) with their actual placeholder
+	// identifiers or variable names
+	replacer := strings.NewReplacer(placeholderShorthands()...)
+
+	// these are placeholders that allow a user-defined final
+	// parameters, but we still want to provide a shorthand
+	// for those, so we use a regexp to replace
+	regexpReplacements := []ComplexShorthandReplacer{
+		{regexp.MustCompile(`{header\.([\w-]*)}`), "{http.request.header.$1}"},
+		{regexp.MustCompile(`{cookie\.([\w-]*)}`), "{http.request.cookie.$1}"},
+		{regexp.MustCompile(`{labels\.([\w-]*)}`), "{http.request.host.labels.$1}"},
+		{regexp.MustCompile(`{path\.([\w-]*)}`), "{http.request.uri.path.$1}"},
+		{regexp.MustCompile(`{file\.([\w-]*)}`), "{http.request.uri.path.file.$1}"},
+		{regexp.MustCompile(`{query\.([\w-]*)}`), "{http.request.uri.query.$1}"},
+		{regexp.MustCompile(`{re\.([\w-]*)\.([\w-]*)}`), "{http.regexp.$1.$2}"},
+		{regexp.MustCompile(`{vars\.([\w-]*)}`), "{http.vars.$1}"},
+		{regexp.MustCompile(`{rp\.([\w-\.]*)}`), "{http.reverse_proxy.$1}"},
+		{regexp.MustCompile(`{err\.([\w-\.]*)}`), "{http.error.$1}"},
+		{regexp.MustCompile(`{file_match\.([\w-]*)}`), "{http.matchers.file.$1}"},
+	}
+
+	return ShorthandReplacer{
+		complex: regexpReplacements,
+		simple:  replacer,
+	}
+}
+
+// placeholderShorthands returns a slice of old-new string pairs,
+// where the left of the pair is a placeholder shorthand that may
+// be used in the Caddyfile, and the right is the replacement.
+func placeholderShorthands() []string {
+	return []string{
+		"{dir}", "{http.request.uri.path.dir}",
+		"{file}", "{http.request.uri.path.file}",
+		"{host}", "{http.request.host}",
+		"{hostport}", "{http.request.hostport}",
+		"{port}", "{http.request.port}",
+		"{method}", "{http.request.method}",
+		"{path}", "{http.request.uri.path}",
+		"{query}", "{http.request.uri.query}",
+		"{remote}", "{http.request.remote}",
+		"{remote_host}", "{http.request.remote.host}",
+		"{remote_port}", "{http.request.remote.port}",
+		"{scheme}", "{http.request.scheme}",
+		"{uri}", "{http.request.uri}",
+		"{uuid}", "{http.request.uuid}",
+		"{tls_cipher}", "{http.request.tls.cipher_suite}",
+		"{tls_version}", "{http.request.tls.version}",
+		"{tls_client_fingerprint}", "{http.request.tls.client.fingerprint}",
+		"{tls_client_issuer}", "{http.request.tls.client.issuer}",
+		"{tls_client_serial}", "{http.request.tls.client.serial}",
+		"{tls_client_subject}", "{http.request.tls.client.subject}",
+		"{tls_client_certificate_pem}", "{http.request.tls.client.certificate_pem}",
+		"{tls_client_certificate_der_base64}", "{http.request.tls.client.certificate_der_base64}",
+		"{upstream_hostport}", "{http.reverse_proxy.upstream.hostport}",
+		"{client_ip}", "{http.vars.client_ip}",
+	}
+}
+
+// ApplyToSegment replaces shorthand placeholder to its full placeholder, understandable by Caddy.
+func (s ShorthandReplacer) ApplyToSegment(segment *caddyfile.Segment) {
+	if segment != nil {
+		for i := 0; i < len(*segment); i++ {
+			// simple string replacements
+			(*segment)[i].Text = s.simple.Replace((*segment)[i].Text)
+			// complex regexp replacements
+			for _, r := range s.complex {
+				(*segment)[i].Text = r.search.ReplaceAllString((*segment)[i].Text, r.replace)
+			}
+		}
+	}
+}

caddyconfig/httpcaddyfile/testdata/import_variadic.txt

@@ -0,0 +1,9 @@
+(t2) {
+    respond 200 {
+        body {args[:]}
+    }
+}
+
+:8082 {
+    import t2 false
+}

caddyconfig/httpcaddyfile/testdata/import_variadic_snippet.txt

@@ -0,0 +1,9 @@
+(t1) {
+    respond 200 {
+        body {args[:]}
+    }
+}
+
+:8081 {
+    import t1 false
+}

caddyconfig/httpcaddyfile/testdata/import_variadic_with_import.txt

@@ -0,0 +1,15 @@
+(t1) {
+    respond 200 {
+        body {args[:]}
+    }
+}
+
+:8081 {
+    import t1 false
+}
+
+import import_variadic.txt
+
+:8083 {
+    import t2 true
+}

caddyconfig/httpcaddyfile/tlsapp.go

@@ -0,0 +1,676 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package httpcaddyfile
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"reflect"
+	"sort"
+	"strconv"
+	"strings"
+
+	"github.com/caddyserver/certmagic"
+	"github.com/mholt/acmez/acme"
+
+	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	"github.com/caddyserver/caddy/v2/modules/caddyhttp"
+	"github.com/caddyserver/caddy/v2/modules/caddytls"
+)
+
+func (st ServerType) buildTLSApp(
+	pairings []sbAddrAssociation,
+	options map[string]any,
+	warnings []caddyconfig.Warning,
+) (*caddytls.TLS, []caddyconfig.Warning, error) {
+	tlsApp := &caddytls.TLS{CertificatesRaw: make(caddy.ModuleMap)}
+	var certLoaders []caddytls.CertificateLoader
+
+	httpPort := strconv.Itoa(caddyhttp.DefaultHTTPPort)
+	if hp, ok := options["http_port"].(int); ok {
+		httpPort = strconv.Itoa(hp)
+	}
+	autoHTTPS := "on"
+	if ah, ok := options["auto_https"].(string); ok {
+		autoHTTPS = ah
+	}
+
+	// find all hosts that share a server block with a hostless
+	// key, so that they don't get forgotten/omitted by auto-HTTPS
+	// (since they won't appear in route matchers)
+	httpsHostsSharedWithHostlessKey := make(map[string]struct{})
+	if autoHTTPS != "off" {
+		for _, pair := range pairings {
+			for _, sb := range pair.serverBlocks {
+				for _, addr := range sb.keys {
+					if addr.Host == "" {
+						// this server block has a hostless key, now
+						// go through and add all the hosts to the set
+						for _, otherAddr := range sb.keys {
+							if otherAddr.Original == addr.Original {
+								continue
+							}
+							if otherAddr.Host != "" && otherAddr.Scheme != "http" && otherAddr.Port != httpPort {
+								httpsHostsSharedWithHostlessKey[otherAddr.Host] = struct{}{}
+							}
+						}
+						break
+					}
+				}
+			}
+		}
+	}
+
+	// a catch-all automation policy is used as a "default" for all subjects that
+	// don't have custom configuration explicitly associated with them; this
+	// is only to add if the global settings or defaults are non-empty
+	catchAllAP, err := newBaseAutomationPolicy(options, warnings, false)
+	if err != nil {
+		return nil, warnings, err
+	}
+	if catchAllAP != nil {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, catchAllAP)
+	}
+
+	for _, p := range pairings {
+		// avoid setting up TLS automation policies for a server that is HTTP-only
+		if !listenersUseAnyPortOtherThan(p.addresses, httpPort) {
+			continue
+		}
+
+		for _, sblock := range p.serverBlocks {
+			// check the scheme of all the site addresses,
+			// skip building AP if they all had http://
+			if sblock.isAllHTTP() {
+				continue
+			}
+
+			// get values that populate an automation policy for this block
+			ap, err := newBaseAutomationPolicy(options, warnings, true)
+			if err != nil {
+				return nil, warnings, err
+			}
+
+			sblockHosts := sblock.hostsFromKeys(false)
+			if len(sblockHosts) == 0 && catchAllAP != nil {
+				ap = catchAllAP
+			}
+
+			// on-demand tls
+			if _, ok := sblock.pile["tls.on_demand"]; ok {
+				ap.OnDemand = true
+			}
+
+			if keyTypeVals, ok := sblock.pile["tls.key_type"]; ok {
+				ap.KeyType = keyTypeVals[0].Value.(string)
+			}
+
+			// certificate issuers
+			if issuerVals, ok := sblock.pile["tls.cert_issuer"]; ok {
+				var issuers []certmagic.Issuer
+				for _, issuerVal := range issuerVals {
+					issuers = append(issuers, issuerVal.Value.(certmagic.Issuer))
+				}
+				if ap == catchAllAP && !reflect.DeepEqual(ap.Issuers, issuers) {
+					// this more correctly implements an error check that was removed
+					// below; try it with this config:
+					//
+					// :443 {
+					// 	bind 127.0.0.1
+					// }
+					//
+					// :443 {
+					// 	bind ::1
+					// 	tls {
+					// 		issuer acme
+					// 	}
+					// }
+					return nil, warnings, fmt.Errorf("automation policy from site block is also default/catch-all policy because of key without hostname, and the two are in conflict: %#v != %#v", ap.Issuers, issuers)
+				}
+				ap.Issuers = issuers
+			}
+
+			// certificate managers
+			if certManagerVals, ok := sblock.pile["tls.cert_manager"]; ok {
+				for _, certManager := range certManagerVals {
+					certGetterName := certManager.Value.(caddy.Module).CaddyModule().ID.Name()
+					ap.ManagersRaw = append(ap.ManagersRaw, caddyconfig.JSONModuleObject(certManager.Value, "via", certGetterName, &warnings))
+				}
+			}
+			// custom bind host
+			for _, cfgVal := range sblock.pile["bind"] {
+				for _, iss := range ap.Issuers {
+					// if an issuer was already configured and it is NOT an ACME issuer,
+					// skip, since we intend to adjust only ACME issuers; ensure we
+					// include any issuer that embeds/wraps an underlying ACME issuer
+					var acmeIssuer *caddytls.ACMEIssuer
+					if acmeWrapper, ok := iss.(acmeCapable); ok {
+						acmeIssuer = acmeWrapper.GetACMEIssuer()
+					}
+					if acmeIssuer == nil {
+						continue
+					}
+
+					// proceed to configure the ACME issuer's bind host, without
+					// overwriting any existing settings
+					if acmeIssuer.Challenges == nil {
+						acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
+					}
+					if acmeIssuer.Challenges.BindHost == "" {
+						// only binding to one host is supported
+						var bindHost string
+						if bindHosts, ok := cfgVal.Value.([]string); ok && len(bindHosts) > 0 {
+							bindHost = bindHosts[0]
+						}
+						acmeIssuer.Challenges.BindHost = bindHost
+					}
+				}
+			}
+
+			// we used to ensure this block is allowed to create an automation policy;
+			// doing so was forbidden if it has a key with no host (i.e. ":443")
+			// and if there is a different server block that also has a key with no
+			// host -- since a key with no host matches any host, we need its
+			// associated automation policy to have an empty Subjects list, i.e. no
+			// host filter, which is indistinguishable between the two server blocks
+			// because automation is not done in the context of a particular server...
+			// this is an example of a poor mapping from Caddyfile to JSON but that's
+			// the least-leaky abstraction I could figure out -- however, this check
+			// was preventing certain listeners, like those provided by plugins, from
+			// being used as desired (see the Tailscale listener plugin), so I removed
+			// the check: and I think since I originally wrote the check I added a new
+			// check above which *properly* detects this ambiguity without breaking the
+			// listener plugin; see the check above with a commented example config
+			if len(sblockHosts) == 0 && catchAllAP == nil {
+				// this server block has a key with no hosts, but there is not yet
+				// a catch-all automation policy (probably because no global options
+				// were set), so this one becomes it
+				catchAllAP = ap
+			}
+
+			// associate our new automation policy with this server block's hosts
+			ap.SubjectsRaw = sblock.hostsFromKeysNotHTTP(httpPort)
+			sort.Strings(ap.SubjectsRaw) // solely for deterministic test results
+
+			// if a combination of public and internal names were given
+			// for this same server block and no issuer was specified, we
+			// need to separate them out in the automation policies so
+			// that the internal names can use the internal issuer and
+			// the other names can use the default/public/ACME issuer
+			var ap2 *caddytls.AutomationPolicy
+			if len(ap.Issuers) == 0 {
+				var internal, external []string
+				for _, s := range ap.SubjectsRaw {
+					// do not create Issuers for Tailscale domains; they will be given a Manager instead
+					if strings.HasSuffix(strings.ToLower(s), ".ts.net") {
+						continue
+					}
+					if !certmagic.SubjectQualifiesForCert(s) {
+						return nil, warnings, fmt.Errorf("subject does not qualify for certificate: '%s'", s)
+					}
+					// we don't use certmagic.SubjectQualifiesForPublicCert() because of one nuance:
+					// names like *.*.tld that may not qualify for a public certificate are actually
+					// fine when used with OnDemand, since OnDemand (currently) does not obtain
+					// wildcards (if it ever does, there will be a separate config option to enable
+					// it that we would need to check here) since the hostname is known at handshake;
+					// and it is unexpected to switch to internal issuer when the user wants to get
+					// regular certificates on-demand for a class of certs like *.*.tld.
+					if subjectQualifiesForPublicCert(ap, s) {
+						external = append(external, s)
+					} else {
+						internal = append(internal, s)
+					}
+				}
+				if len(external) > 0 && len(internal) > 0 {
+					ap.SubjectsRaw = external
+					apCopy := *ap
+					ap2 = &apCopy
+					ap2.SubjectsRaw = internal
+					ap2.IssuersRaw = []json.RawMessage{caddyconfig.JSONModuleObject(caddytls.InternalIssuer{}, "module", "internal", &warnings)}
+				}
+			}
+			if tlsApp.Automation == nil {
+				tlsApp.Automation = new(caddytls.AutomationConfig)
+			}
+			tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap)
+			if ap2 != nil {
+				tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap2)
+			}
+
+			// certificate loaders
+			if clVals, ok := sblock.pile["tls.cert_loader"]; ok {
+				for _, clVal := range clVals {
+					certLoaders = append(certLoaders, clVal.Value.(caddytls.CertificateLoader))
+				}
+			}
+		}
+	}
+
+	// group certificate loaders by module name, then add to config
+	if len(certLoaders) > 0 {
+		loadersByName := make(map[string]caddytls.CertificateLoader)
+		for _, cl := range certLoaders {
+			name := caddy.GetModuleName(cl)
+			// ugh... technically, we may have multiple FileLoader and FolderLoader
+			// modules (because the tls directive returns one per occurrence), but
+			// the config structure expects only one instance of each kind of loader
+			// module, so we have to combine them... instead of enumerating each
+			// possible cert loader module in a type switch, we can use reflection,
+			// which works on any cert loaders that are slice types
+			if reflect.TypeOf(cl).Kind() == reflect.Slice {
+				combined := reflect.ValueOf(loadersByName[name])
+				if !combined.IsValid() {
+					combined = reflect.New(reflect.TypeOf(cl)).Elem()
+				}
+				clVal := reflect.ValueOf(cl)
+				for i := 0; i < clVal.Len(); i++ {
+					combined = reflect.Append(combined, clVal.Index(i))
+				}
+				loadersByName[name] = combined.Interface().(caddytls.CertificateLoader)
+			}
+		}
+		for certLoaderName, loaders := range loadersByName {
+			tlsApp.CertificatesRaw[certLoaderName] = caddyconfig.JSON(loaders, &warnings)
+		}
+	}
+
+	// set any of the on-demand options, for if/when on-demand TLS is enabled
+	if onDemand, ok := options["on_demand_tls"].(*caddytls.OnDemandConfig); ok {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.OnDemand = onDemand
+	}
+
+	// set the storage clean interval if configured
+	if storageCleanInterval, ok := options["storage_clean_interval"].(caddy.Duration); ok {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.StorageCleanInterval = storageCleanInterval
+	}
+
+	// set the expired certificates renew interval if configured
+	if renewCheckInterval, ok := options["renew_interval"].(caddy.Duration); ok {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.RenewCheckInterval = renewCheckInterval
+	}
+
+	// set the OCSP check interval if configured
+	if ocspCheckInterval, ok := options["ocsp_interval"].(caddy.Duration); ok {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.OCSPCheckInterval = ocspCheckInterval
+	}
+
+	// set whether OCSP stapling should be disabled for manually-managed certificates
+	if ocspConfig, ok := options["ocsp_stapling"].(certmagic.OCSPConfig); ok {
+		tlsApp.DisableOCSPStapling = ocspConfig.DisableStapling
+	}
+
+	// if any hostnames appear on the same server block as a key with
+	// no host, they will not be used with route matchers because the
+	// hostless key matches all hosts, therefore, it wouldn't be
+	// considered for auto-HTTPS, so we need to make sure those hosts
+	// are manually considered for managed certificates; we also need
+	// to make sure that any of these names which are internal-only
+	// get internal certificates by default rather than ACME
+	var al caddytls.AutomateLoader
+	internalAP := &caddytls.AutomationPolicy{
+		IssuersRaw: []json.RawMessage{json.RawMessage(`{"module":"internal"}`)},
+	}
+	if autoHTTPS != "off" {
+		for h := range httpsHostsSharedWithHostlessKey {
+			al = append(al, h)
+			if !certmagic.SubjectQualifiesForPublicCert(h) {
+				internalAP.SubjectsRaw = append(internalAP.SubjectsRaw, h)
+			}
+		}
+	}
+	if len(al) > 0 {
+		tlsApp.CertificatesRaw["automate"] = caddyconfig.JSON(al, &warnings)
+	}
+	if len(internalAP.SubjectsRaw) > 0 {
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, internalAP)
+	}
+
+	// if there are any global options set for issuers (ACME ones in particular), make sure they
+	// take effect in every automation policy that does not have any issuers
+	if tlsApp.Automation != nil {
+		globalEmail := options["email"]
+		globalACMECA := options["acme_ca"]
+		globalACMECARoot := options["acme_ca_root"]
+		globalACMEDNS := options["acme_dns"]
+		globalACMEEAB := options["acme_eab"]
+		globalPreferredChains := options["preferred_chains"]
+		hasGlobalACMEDefaults := globalEmail != nil || globalACMECA != nil || globalACMECARoot != nil || globalACMEDNS != nil || globalACMEEAB != nil || globalPreferredChains != nil
+		if hasGlobalACMEDefaults {
+			for i := 0; i < len(tlsApp.Automation.Policies); i++ {
+				ap := tlsApp.Automation.Policies[i]
+				if len(ap.Issuers) == 0 && automationPolicyHasAllPublicNames(ap) {
+					// for public names, create default issuers which will later be filled in with configured global defaults
+					// (internal names will implicitly use the internal issuer at auto-https time)
+					ap.Issuers = caddytls.DefaultIssuers()
+
+					// if a specific endpoint is configured, can't use multiple default issuers
+					if globalACMECA != nil {
+						if strings.Contains(globalACMECA.(string), "zerossl") {
+							ap.Issuers = []certmagic.Issuer{&caddytls.ZeroSSLIssuer{ACMEIssuer: new(caddytls.ACMEIssuer)}}
+						} else {
+							ap.Issuers = []certmagic.Issuer{new(caddytls.ACMEIssuer)}
+						}
+					}
+				}
+			}
+		}
+	}
+
+	// finalize and verify policies; do cleanup
+	if tlsApp.Automation != nil {
+		for i, ap := range tlsApp.Automation.Policies {
+			// ensure all issuers have global defaults filled in
+			for j, issuer := range ap.Issuers {
+				err := fillInGlobalACMEDefaults(issuer, options)
+				if err != nil {
+					return nil, warnings, fmt.Errorf("filling in global issuer defaults for AP %d, issuer %d: %v", i, j, err)
+				}
+			}
+
+			// encode all issuer values we created, so they will be rendered in the output
+			if len(ap.Issuers) > 0 && ap.IssuersRaw == nil {
+				for _, iss := range ap.Issuers {
+					issuerName := iss.(caddy.Module).CaddyModule().ID.Name()
+					ap.IssuersRaw = append(ap.IssuersRaw, caddyconfig.JSONModuleObject(iss, "module", issuerName, &warnings))
+				}
+			}
+		}
+
+		// consolidate automation policies that are the exact same
+		tlsApp.Automation.Policies = consolidateAutomationPolicies(tlsApp.Automation.Policies)
+
+		// ensure automation policies don't overlap subjects (this should be
+		// an error at provision-time as well, but catch it in the adapt phase
+		// for convenience)
+		automationHostSet := make(map[string]struct{})
+		for _, ap := range tlsApp.Automation.Policies {
+			for _, s := range ap.SubjectsRaw {
+				if _, ok := automationHostSet[s]; ok {
+					return nil, warnings, fmt.Errorf("hostname appears in more than one automation policy, making certificate management ambiguous: %s", s)
+				}
+				automationHostSet[s] = struct{}{}
+			}
+		}
+
+		// if nothing remains, remove any excess values to clean up the resulting config
+		if len(tlsApp.Automation.Policies) == 0 {
+			tlsApp.Automation.Policies = nil
+		}
+		if reflect.DeepEqual(tlsApp.Automation, new(caddytls.AutomationConfig)) {
+			tlsApp.Automation = nil
+		}
+	}
+
+	return tlsApp, warnings, nil
+}
+
+type acmeCapable interface{ GetACMEIssuer() *caddytls.ACMEIssuer }
+
+func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]any) error {
+	acmeWrapper, ok := issuer.(acmeCapable)
+	if !ok {
+		return nil
+	}
+	acmeIssuer := acmeWrapper.GetACMEIssuer()
+	if acmeIssuer == nil {
+		return nil
+	}
+
+	globalEmail := options["email"]
+	globalACMECA := options["acme_ca"]
+	globalACMECARoot := options["acme_ca_root"]
+	globalACMEDNS := options["acme_dns"]
+	globalACMEEAB := options["acme_eab"]
+	globalPreferredChains := options["preferred_chains"]
+
+	if globalEmail != nil && acmeIssuer.Email == "" {
+		acmeIssuer.Email = globalEmail.(string)
+	}
+	if globalACMECA != nil && acmeIssuer.CA == "" {
+		acmeIssuer.CA = globalACMECA.(string)
+	}
+	if globalACMECARoot != nil && !sliceContains(acmeIssuer.TrustedRootsPEMFiles, globalACMECARoot.(string)) {
+		acmeIssuer.TrustedRootsPEMFiles = append(acmeIssuer.TrustedRootsPEMFiles, globalACMECARoot.(string))
+	}
+	if globalACMEDNS != nil && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.DNS == nil) {
+		acmeIssuer.Challenges = &caddytls.ChallengesConfig{
+			DNS: &caddytls.DNSChallengeConfig{
+				ProviderRaw: caddyconfig.JSONModuleObject(globalACMEDNS, "name", globalACMEDNS.(caddy.Module).CaddyModule().ID.Name(), nil),
+			},
+		}
+	}
+	if globalACMEEAB != nil && acmeIssuer.ExternalAccount == nil {
+		acmeIssuer.ExternalAccount = globalACMEEAB.(*acme.EAB)
+	}
+	if globalPreferredChains != nil && acmeIssuer.PreferredChains == nil {
+		acmeIssuer.PreferredChains = globalPreferredChains.(*caddytls.ChainPreference)
+	}
+	return nil
+}
+
+// newBaseAutomationPolicy returns a new TLS automation policy that gets
+// its values from the global options map. It should be used as the base
+// for any other automation policies. A nil policy (and no error) will be
+// returned if there are no default/global options. However, if always is
+// true, a non-nil value will always be returned (unless there is an error).
+func newBaseAutomationPolicy(options map[string]any, warnings []caddyconfig.Warning, always bool) (*caddytls.AutomationPolicy, error) {
+	issuers, hasIssuers := options["cert_issuer"]
+	_, hasLocalCerts := options["local_certs"]
+	keyType, hasKeyType := options["key_type"]
+	ocspStapling, hasOCSPStapling := options["ocsp_stapling"]
+
+	hasGlobalAutomationOpts := hasIssuers || hasLocalCerts || hasKeyType || hasOCSPStapling
+
+	// if there are no global options related to automation policies
+	// set, then we can just return right away
+	if !hasGlobalAutomationOpts {
+		if always {
+			return new(caddytls.AutomationPolicy), nil
+		}
+		return nil, nil
+	}
+
+	ap := new(caddytls.AutomationPolicy)
+	if hasKeyType {
+		ap.KeyType = keyType.(string)
+	}
+
+	if hasIssuers && hasLocalCerts {
+		return nil, fmt.Errorf("global options are ambiguous: local_certs is confusing when combined with cert_issuer, because local_certs is also a specific kind of issuer")
+	}
+
+	if hasIssuers {
+		ap.Issuers = issuers.([]certmagic.Issuer)
+	} else if hasLocalCerts {
+		ap.Issuers = []certmagic.Issuer{new(caddytls.InternalIssuer)}
+	}
+
+	if hasOCSPStapling {
+		ocspConfig := ocspStapling.(certmagic.OCSPConfig)
+		ap.DisableOCSPStapling = ocspConfig.DisableStapling
+		ap.OCSPOverrides = ocspConfig.ResponderOverrides
+	}
+
+	return ap, nil
+}
+
+// consolidateAutomationPolicies combines automation policies that are the same,
+// for a cleaner overall output.
+func consolidateAutomationPolicies(aps []*caddytls.AutomationPolicy) []*caddytls.AutomationPolicy {
+	// sort from most specific to least specific; we depend on this ordering
+	sort.SliceStable(aps, func(i, j int) bool {
+		if automationPolicyIsSubset(aps[i], aps[j]) {
+			return true
+		}
+		if automationPolicyIsSubset(aps[j], aps[i]) {
+			return false
+		}
+		return len(aps[i].SubjectsRaw) > len(aps[j].SubjectsRaw)
+	})
+
+	emptyAPCount := 0
+	origLenAPs := len(aps)
+	// compute the number of empty policies (disregarding subjects) - see #4128
+	emptyAP := new(caddytls.AutomationPolicy)
+	for i := 0; i < len(aps); i++ {
+		emptyAP.SubjectsRaw = aps[i].SubjectsRaw
+		if reflect.DeepEqual(aps[i], emptyAP) {
+			emptyAPCount++
+			if !automationPolicyHasAllPublicNames(aps[i]) {
+				// if this automation policy has internal names, we might as well remove it
+				// so auto-https can implicitly use the internal issuer
+				aps = append(aps[:i], aps[i+1:]...)
+				i--
+			}
+		}
+	}
+	// If all policies are empty, we can return nil, as there is no need to set any policy
+	if emptyAPCount == origLenAPs {
+		return nil
+	}
+
+	// remove or combine duplicate policies
+outer:
+	for i := 0; i < len(aps); i++ {
+		// compare only with next policies; we sorted by specificity so we must not delete earlier policies
+		for j := i + 1; j < len(aps); j++ {
+			// if they're exactly equal in every way, just keep one of them
+			if reflect.DeepEqual(aps[i], aps[j]) {
+				aps = append(aps[:j], aps[j+1:]...)
+				// must re-evaluate current i against next j; can't skip it!
+				// even if i decrements to -1, will be incremented to 0 immediately
+				i--
+				continue outer
+			}
+
+			// if the policy is the same, we can keep just one, but we have
+			// to be careful which one we keep; if only one has any hostnames
+			// defined, then we need to keep the one without any hostnames,
+			// otherwise the one without any subjects (a catch-all) would be
+			// eaten up by the one with subjects; and if both have subjects, we
+			// need to combine their lists
+			if reflect.DeepEqual(aps[i].IssuersRaw, aps[j].IssuersRaw) &&
+				reflect.DeepEqual(aps[i].ManagersRaw, aps[j].ManagersRaw) &&
+				bytes.Equal(aps[i].StorageRaw, aps[j].StorageRaw) &&
+				aps[i].MustStaple == aps[j].MustStaple &&
+				aps[i].KeyType == aps[j].KeyType &&
+				aps[i].OnDemand == aps[j].OnDemand &&
+				aps[i].RenewalWindowRatio == aps[j].RenewalWindowRatio {
+				if len(aps[i].SubjectsRaw) > 0 && len(aps[j].SubjectsRaw) == 0 {
+					// later policy (at j) has no subjects ("catch-all"), so we can
+					// remove the identical-but-more-specific policy that comes first
+					// AS LONG AS it is not shadowed by another policy before it; e.g.
+					// if policy i is for example.com, policy i+1 is '*.com', and policy
+					// j is catch-all, we cannot remove policy i because that would
+					// cause example.com to be served by the less specific policy for
+					// '*.com', which might be different (yes we've seen this happen)
+					if automationPolicyShadows(i, aps) >= j {
+						aps = append(aps[:i], aps[i+1:]...)
+						i--
+						continue outer
+					}
+				} else {
+					// avoid repeated subjects
+					for _, subj := range aps[j].SubjectsRaw {
+						if !sliceContains(aps[i].SubjectsRaw, subj) {
+							aps[i].SubjectsRaw = append(aps[i].SubjectsRaw, subj)
+						}
+					}
+					aps = append(aps[:j], aps[j+1:]...)
+					j--
+				}
+			}
+		}
+	}
+
+	return aps
+}
+
+// automationPolicyIsSubset returns true if a's subjects are a subset
+// of b's subjects.
+func automationPolicyIsSubset(a, b *caddytls.AutomationPolicy) bool {
+	if len(b.SubjectsRaw) == 0 {
+		return true
+	}
+	if len(a.SubjectsRaw) == 0 {
+		return false
+	}
+	for _, aSubj := range a.SubjectsRaw {
+		var inSuperset bool
+		for _, bSubj := range b.SubjectsRaw {
+			if certmagic.MatchWildcard(aSubj, bSubj) {
+				inSuperset = true
+				break
+			}
+		}
+		if !inSuperset {
+			return false
+		}
+	}
+	return true
+}
+
+// automationPolicyShadows returns the index of a policy that aps[i] shadows;
+// in other words, for all policies after position i, if that policy covers
+// the same subjects but is less specific, that policy's position is returned,
+// or -1 if no shadowing is found. For example, if policy i is for
+// "foo.example.com" and policy i+2 is for "*.example.com", then i+2 will be
+// returned, since that policy is shadowed by i, which is in front.
+func automationPolicyShadows(i int, aps []*caddytls.AutomationPolicy) int {
+	for j := i + 1; j < len(aps); j++ {
+		if automationPolicyIsSubset(aps[i], aps[j]) {
+			return j
+		}
+	}
+	return -1
+}
+
+// subjectQualifiesForPublicCert is like certmagic.SubjectQualifiesForPublicCert() except
+// that this allows domains with multiple wildcard levels like '*.*.example.com' to qualify
+// if the automation policy has OnDemand enabled (i.e. this function is more lenient).
+func subjectQualifiesForPublicCert(ap *caddytls.AutomationPolicy, subj string) bool {
+	return !certmagic.SubjectIsIP(subj) &&
+		!certmagic.SubjectIsInternal(subj) &&
+		(strings.Count(subj, "*.") < 2 || ap.OnDemand)
+}
+
+func automationPolicyHasAllPublicNames(ap *caddytls.AutomationPolicy) bool {
+	for _, subj := range ap.SubjectsRaw {
+		if !subjectQualifiesForPublicCert(ap, subj) {
+			return false
+		}
+	}
+	return true
+}

caddyconfig/httpcaddyfile/tlsapp_test.go

@@ -0,0 +1,56 @@
+package httpcaddyfile
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/modules/caddytls"
+)
+
+func TestAutomationPolicyIsSubset(t *testing.T) {
+	for i, test := range []struct {
+		a, b   []string
+		expect bool
+	}{
+		{
+			a:      []string{"example.com"},
+			b:      []string{},
+			expect: true,
+		},
+		{
+			a:      []string{},
+			b:      []string{"example.com"},
+			expect: false,
+		},
+		{
+			a:      []string{"foo.example.com"},
+			b:      []string{"*.example.com"},
+			expect: true,
+		},
+		{
+			a:      []string{"foo.example.com"},
+			b:      []string{"foo.example.com"},
+			expect: true,
+		},
+		{
+			a:      []string{"foo.example.com"},
+			b:      []string{"example.com"},
+			expect: false,
+		},
+		{
+			a:      []string{"example.com", "foo.example.com"},
+			b:      []string{"*.com", "*.*.com"},
+			expect: true,
+		},
+		{
+			a:      []string{"example.com", "foo.example.com"},
+			b:      []string{"*.com"},
+			expect: false,
+		},
+	} {
+		apA := &caddytls.AutomationPolicy{SubjectsRaw: test.a}
+		apB := &caddytls.AutomationPolicy{SubjectsRaw: test.b}
+		if actual := automationPolicyIsSubset(apA, apB); actual != test.expect {
+			t.Errorf("Test %d: Expected %t but got %t (A: %v  B: %v)", i, test.expect, actual, test.a, test.b)
+		}
+	}
+}

caddyconfig/httploader.go

@@ -0,0 +1,221 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyconfig
+
+import (
+	"crypto/tls"
+	"crypto/x509"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"time"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func init() {
+	caddy.RegisterModule(HTTPLoader{})
+}
+
+// HTTPLoader can load Caddy configs over HTTP(S).
+//
+// If the response is not a JSON config, a config adapter must be specified
+// either in the loader config (`adapter`), or in the Content-Type HTTP header
+// returned in the HTTP response from the server. The Content-Type header is
+// read just like the admin API's `/load` endpoint. Uf you don't have control
+// over the HTTP server (but can still trust its response), you can override
+// the Content-Type header by setting the `adapter` property in this config.
+type HTTPLoader struct {
+	// The method for the request. Default: GET
+	Method string `json:"method,omitempty"`
+
+	// The URL of the request.
+	URL string `json:"url,omitempty"`
+
+	// HTTP headers to add to the request.
+	Headers http.Header `json:"header,omitempty"`
+
+	// Maximum time allowed for a complete connection and request.
+	Timeout caddy.Duration `json:"timeout,omitempty"`
+
+	// The name of the config adapter to use, if any. Only needed
+	// if the HTTP response is not a JSON config and if the server's
+	// Content-Type header is missing or incorrect.
+	Adapter string `json:"adapter,omitempty"`
+
+	TLS *struct {
+		// Present this instance's managed remote identity credentials to the server.
+		UseServerIdentity bool `json:"use_server_identity,omitempty"`
+
+		// PEM-encoded client certificate filename to present to the server.
+		ClientCertificateFile string `json:"client_certificate_file,omitempty"`
+
+		// PEM-encoded key to use with the client certificate.
+		ClientCertificateKeyFile string `json:"client_certificate_key_file,omitempty"`
+
+		// List of PEM-encoded CA certificate files to add to the same trust
+		// store as RootCAPool (or root_ca_pool in the JSON).
+		RootCAPEMFiles []string `json:"root_ca_pem_files,omitempty"`
+	} `json:"tls,omitempty"`
+}
+
+// CaddyModule returns the Caddy module information.
+func (HTTPLoader) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "caddy.config_loaders.http",
+		New: func() caddy.Module { return new(HTTPLoader) },
+	}
+}
+
+// LoadConfig loads a Caddy config.
+func (hl HTTPLoader) LoadConfig(ctx caddy.Context) ([]byte, error) {
+	repl := caddy.NewReplacer()
+
+	client, err := hl.makeClient(ctx)
+	if err != nil {
+		return nil, err
+	}
+
+	method := repl.ReplaceAll(hl.Method, "")
+	if method == "" {
+		method = http.MethodGet
+	}
+
+	url := repl.ReplaceAll(hl.URL, "")
+	req, err := http.NewRequest(method, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	for key, vals := range hl.Headers {
+		for _, val := range vals {
+			req.Header.Add(repl.ReplaceAll(key, ""), repl.ReplaceKnown(val, ""))
+		}
+	}
+
+	resp, err := doHttpCallWithRetries(ctx, client, req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode >= 400 {
+		return nil, fmt.Errorf("server responded with HTTP %d", resp.StatusCode)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	// adapt the config based on either manually-configured adapter or server's response header
+	ct := resp.Header.Get("Content-Type")
+	if hl.Adapter != "" {
+		ct = "text/" + hl.Adapter
+	}
+	result, warnings, err := adaptByContentType(ct, body)
+	if err != nil {
+		return nil, err
+	}
+	for _, warn := range warnings {
+		ctx.Logger().Warn(warn.String())
+	}
+
+	return result, nil
+}
+
+func attemptHttpCall(client *http.Client, request *http.Request) (*http.Response, error) {
+	resp, err := client.Do(request)
+	if err != nil {
+		return nil, fmt.Errorf("problem calling http loader url: %v", err)
+	} else if resp.StatusCode < 200 || resp.StatusCode > 499 {
+		resp.Body.Close()
+		return nil, fmt.Errorf("bad response status code from http loader url: %v", resp.StatusCode)
+	}
+	return resp, nil
+}
+
+func doHttpCallWithRetries(ctx caddy.Context, client *http.Client, request *http.Request) (*http.Response, error) {
+	var resp *http.Response
+	var err error
+	const maxAttempts = 10
+
+	for i := 0; i < maxAttempts; i++ {
+		resp, err = attemptHttpCall(client, request)
+		if err != nil && i < maxAttempts-1 {
+			select {
+			case <-time.After(time.Millisecond * 500):
+			case <-ctx.Done():
+				return resp, ctx.Err()
+			}
+		} else {
+			break
+		}
+	}
+
+	return resp, err
+}
+
+func (hl HTTPLoader) makeClient(ctx caddy.Context) (*http.Client, error) {
+	client := &http.Client{
+		Timeout: time.Duration(hl.Timeout),
+	}
+
+	if hl.TLS != nil {
+		var tlsConfig *tls.Config
+
+		// client authentication
+		if hl.TLS.UseServerIdentity {
+			certs, err := ctx.IdentityCredentials(ctx.Logger())
+			if err != nil {
+				return nil, fmt.Errorf("getting server identity credentials: %v", err)
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.Certificates = certs
+		} else if hl.TLS.ClientCertificateFile != "" && hl.TLS.ClientCertificateKeyFile != "" {
+			cert, err := tls.LoadX509KeyPair(hl.TLS.ClientCertificateFile, hl.TLS.ClientCertificateKeyFile)
+			if err != nil {
+				return nil, err
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.Certificates = []tls.Certificate{cert}
+		}
+
+		// trusted server certs
+		if len(hl.TLS.RootCAPEMFiles) > 0 {
+			rootPool := x509.NewCertPool()
+			for _, pemFile := range hl.TLS.RootCAPEMFiles {
+				pemData, err := os.ReadFile(pemFile)
+				if err != nil {
+					return nil, fmt.Errorf("failed reading ca cert: %v", err)
+				}
+				rootPool.AppendCertsFromPEM(pemData)
+			}
+			if tlsConfig == nil {
+				tlsConfig = new(tls.Config)
+			}
+			tlsConfig.RootCAs = rootPool
+		}
+
+		client.Transport = &http.Transport{TLSClientConfig: tlsConfig}
+	}
+
+	return client, nil
+}
+
+var _ caddy.ConfigLoader = (*HTTPLoader)(nil)

caddyconfig/jsonc/jsonc.go

@@ -1,49 +0,0 @@
-// Copyright 2015 Matthew Holt and The Caddy Authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package jsoncadapter
-
-import (
-	"encoding/json"
-
-	"github.com/caddyserver/caddy/v2/caddyconfig"
-	"github.com/muhammadmuzzammil1998/jsonc"
-)
-
-func init() {
-	caddyconfig.RegisterAdapter("jsonc", Adapter{})
-}
-
-// Adapter adapts JSON-C to Caddy JSON.
-type Adapter struct{}
-
-// Adapt converts the JSON-C config in body to Caddy JSON.
-func (a Adapter) Adapt(body []byte, options map[string]interface{}) (result []byte, warnings []caddyconfig.Warning, err error) {
-	result = jsonc.ToJSON(body)
-
-	// any errors in the JSON will be
-	// reported during config load, but
-	// we can at least warn here that
-	// it is not valid JSON
-	if !json.Valid(result) {
-		warnings = append(warnings, caddyconfig.Warning{
-			Message: "Resulting JSON is invalid.",
-		})
-	}
-
-	return
-}
-
-// Interface guard
-var _ caddyconfig.Adapter = (*Adapter)(nil)

caddyconfig/load.go

@@ -0,0 +1,214 @@
+// Copyright 2015 Matthew Holt and The Caddy Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package caddyconfig
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"mime"
+	"net/http"
+	"strings"
+	"sync"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func init() {
+	caddy.RegisterModule(adminLoad{})
+}
+
+// adminLoad is a module that provides the /load endpoint
+// for the Caddy admin API. The only reason it's not baked
+// into the caddy package directly is because of the import
+// of the caddyconfig package for its GetAdapter function.
+// If the caddy package depends on the caddyconfig package,
+// then the caddyconfig package will not be able to import
+// the caddy package, and it can more easily cause backward
+// edges in the dependency tree (i.e. import cycle).
+// Fortunately, the admin API has first-class support for
+// adding endpoints from modules.
+type adminLoad struct{}
+
+// CaddyModule returns the Caddy module information.
+func (adminLoad) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "admin.api.load",
+		New: func() caddy.Module { return new(adminLoad) },
+	}
+}
+
+// Routes returns a route for the /load endpoint.
+func (al adminLoad) Routes() []caddy.AdminRoute {
+	return []caddy.AdminRoute{
+		{
+			Pattern: "/load",
+			Handler: caddy.AdminHandlerFunc(al.handleLoad),
+		},
+		{
+			Pattern: "/adapt",
+			Handler: caddy.AdminHandlerFunc(al.handleAdapt),
+		},
+	}
+}
+
+// handleLoad replaces the entire current configuration with
+// a new one provided in the response body. It supports config
+// adapters through the use of the Content-Type header. A
+// config that is identical to the currently-running config
+// will be a no-op unless Cache-Control: must-revalidate is set.
+func (adminLoad) handleLoad(w http.ResponseWriter, r *http.Request) error {
+	if r.Method != http.MethodPost {
+		return caddy.APIError{
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method not allowed"),
+		}
+	}
+
+	buf := bufPool.Get().(*bytes.Buffer)
+	buf.Reset()
+	defer bufPool.Put(buf)
+
+	_, err := io.Copy(buf, r.Body)
+	if err != nil {
+		return caddy.APIError{
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("reading request body: %v", err),
+		}
+	}
+	body := buf.Bytes()
+
+	// if the config is formatted other than Caddy's native
+	// JSON, we need to adapt it before loading it
+	if ctHeader := r.Header.Get("Content-Type"); ctHeader != "" {
+		result, warnings, err := adaptByContentType(ctHeader, body)
+		if err != nil {
+			return caddy.APIError{
+				HTTPStatus: http.StatusBadRequest,
+				Err:        err,
+			}
+		}
+		if len(warnings) > 0 {
+			respBody, err := json.Marshal(warnings)
+			if err != nil {
+				caddy.Log().Named("admin.api.load").Error(err.Error())
+			}
+			_, _ = w.Write(respBody)
+		}
+		body = result
+	}
+
+	forceReload := r.Header.Get("Cache-Control") == "must-revalidate"
+
+	err = caddy.Load(body, forceReload)
+	if err != nil {
+		return caddy.APIError{
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("loading config: %v", err),
+		}
+	}
+
+	caddy.Log().Named("admin.api").Info("load complete")
+
+	return nil
+}
+
+// handleAdapt adapts the given Caddy config to JSON and responds with the result.
+func (adminLoad) handleAdapt(w http.ResponseWriter, r *http.Request) error {
+	if r.Method != http.MethodPost {
+		return caddy.APIError{
+			HTTPStatus: http.StatusMethodNotAllowed,
+			Err:        fmt.Errorf("method not allowed"),
+		}
+	}
+
+	buf := bufPool.Get().(*bytes.Buffer)
+	buf.Reset()
+	defer bufPool.Put(buf)
+
+	_, err := io.Copy(buf, r.Body)
+	if err != nil {
+		return caddy.APIError{
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("reading request body: %v", err),
+		}
+	}
+
+	result, warnings, err := adaptByContentType(r.Header.Get("Content-Type"), buf.Bytes())
+	if err != nil {
+		return caddy.APIError{
+			HTTPStatus: http.StatusBadRequest,
+			Err:        err,
+		}
+	}
+
+	out := struct {
+		Warnings []Warning       `json:"warnings,omitempty"`
+		Result   json.RawMessage `json:"result"`
+	}{
+		Warnings: warnings,
+		Result:   result,
+	}
+
+	w.Header().Set("Content-Type", "application/json")
+	return json.NewEncoder(w).Encode(out)
+}
+
+// adaptByContentType adapts body to Caddy JSON using the adapter specified by contentType.
+// If contentType is empty or ends with "/json", the input will be returned, as a no-op.
+func adaptByContentType(contentType string, body []byte) ([]byte, []Warning, error) {
+	// assume JSON as the default
+	if contentType == "" {
+		return body, nil, nil
+	}
+
+	ct, _, err := mime.ParseMediaType(contentType)
+	if err != nil {
+		return nil, nil, caddy.APIError{
+			HTTPStatus: http.StatusBadRequest,
+			Err:        fmt.Errorf("invalid Content-Type: %v", err),
+		}
+	}
+
+	// if already JSON, no need to adapt
+	if strings.HasSuffix(ct, "/json") {
+		return body, nil, nil
+	}
+
+	// adapter name should be suffix of MIME type
+	_, adapterName, slashFound := strings.Cut(ct, "/")
+	if !slashFound {
+		return nil, nil, fmt.Errorf("malformed Content-Type")
+	}
+
+	cfgAdapter := GetAdapter(adapterName)
+	if cfgAdapter == nil {
+		return nil, nil, fmt.Errorf("unrecognized config adapter '%s'", adapterName)
+	}
+
+	result, warnings, err := cfgAdapter.Adapt(body, nil)
+	if err != nil {
+		return nil, nil, fmt.Errorf("adapting config using %s adapter: %v", adapterName, err)
+	}
+
+	return result, warnings, nil
+}
+
+var bufPool = sync.Pool{
+	New: func() any {
+		return new(bytes.Buffer)
+	},
+}

caddytest/a.caddy.localhost.crt

@@ -0,0 +1,23 @@
+-----BEGIN CERTIFICATE-----
+MIID5zCCAs8CFG4+w/pqR5AZQ+aVB330uRRRKMF0MA0GCSqGSIb3DQEBCwUAMIGv
+MQswCQYDVQQGEwJVUzELMAkGA1UECAwCTlkxGzAZBgNVBAoMEkxvY2FsIERldmVs
+b3BlbWVudDEbMBkGA1UEBwwSTG9jYWwgRGV2ZWxvcGVtZW50MRowGAYDVQQDDBFh
+LmNhZGR5LmxvY2FsaG9zdDEbMBkGA1UECwwSTG9jYWwgRGV2ZWxvcGVtZW50MSAw
+HgYJKoZIhvcNAQkBFhFhZG1pbkBjYWRkeS5sb2NhbDAeFw0yMDAzMTMxODUwMTda
+Fw0zMDAzMTExODUwMTdaMIGvMQswCQYDVQQGEwJVUzELMAkGA1UECAwCTlkxGzAZ
+BgNVBAoMEkxvY2FsIERldmVsb3BlbWVudDEbMBkGA1UEBwwSTG9jYWwgRGV2ZWxv
+cGVtZW50MRowGAYDVQQDDBFhLmNhZGR5LmxvY2FsaG9zdDEbMBkGA1UECwwSTG9j
+YWwgRGV2ZWxvcGVtZW50MSAwHgYJKoZIhvcNAQkBFhFhZG1pbkBjYWRkeS5sb2Nh
+bDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMd9pC9wF7j0459FndPs
+Deud/rq41jEZFsVOVtjQgjS1A5ct6NfeMmSlq8i1F7uaTMPZjbOHzY6y6hzLc9+y
+/VWNgyUC543HjXnNTnp9Xug6tBBxOxvRMw5mv2nAyzjBGDePPgN84xKhOXG2Wj3u
+fOZ+VPVISefRNvjKfN87WLJ0B0HI9wplG5ASVdPQsWDY1cndrZgt2sxQ/3fjIno4
+VvrgRWC9Penizgps/a0ZcFZMD/6HJoX/mSZVa1LjopwbMTXvyHCpXkth21E+rBt6
+I9DMHerdioVQcX25CqPmAwePxPZSNGEQo/Qu32kzcmscmYxTtYBhDa+yLuHgGggI
+j7ECAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAP/94KPtkpYtkWADnhtzDmgQ6Q1pH
+SubTUZdCwQtm6/LrvpT+uFNsOj4L3Mv3TVUnIQDmKd5VvR42W2MRBiTN2LQptgEn
+C7g9BB+UA9kjL3DPk1pJMjzxLHohh0uNLi7eh4mAj8eNvjz9Z4qMWPQoVS0y7/ZK
+cCBRKh2GkIqKm34ih6pX7xmMpPEQsFoTVPRHYJfYD1SZ8Iui+EN+7WqLuJWPsPXw
+JM1HuZKn7pZmJU2MZZBsrupHGUvNMbBg2mFJcxt4D1VvU+p+a67PSjpFQ6dJG2re
+pZoF+N1vMGAFkxe6UqhcC/bXDX+ILVQHJ+RNhzDO6DcWf8dRrC2LaJk3WA==
+-----END CERTIFICATE-----

caddytest/a.caddy.localhost.key

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpAIBAAKCAQEAx32kL3AXuPTjn0Wd0+wN653+urjWMRkWxU5W2NCCNLUDly3o
+194yZKWryLUXu5pMw9mNs4fNjrLqHMtz37L9VY2DJQLnjceNec1Oen1e6Dq0EHE7
+G9EzDma/acDLOMEYN48+A3zjEqE5cbZaPe585n5U9UhJ59E2+Mp83ztYsnQHQcj3
+CmUbkBJV09CxYNjVyd2tmC3azFD/d+MiejhW+uBFYL096eLOCmz9rRlwVkwP/ocm
+hf+ZJlVrUuOinBsxNe/IcKleS2HbUT6sG3oj0Mwd6t2KhVBxfbkKo+YDB4/E9lI0
+YRCj9C7faTNyaxyZjFO1gGENr7Iu4eAaCAiPsQIDAQABAoIBAQDD/YFIBeWYlifn
+e9risQDAIrp3sk7lb9O6Rwv1+Wxi4hBEABvJsYhq74VFK/3EF4UhyWR5JIvkjYyK
+e6w887oGyoA05ZSe65XoO7fFidSrbbkoikZbPv3dQT7/ZCWEfdkQBNAVVyY0UGeC
+e3hPbjYRsb5AOSQ694X9idqC6uhqcOrBDjITFrctUoP4S6l9A6a+mLSUIwiICcuh
+mrNl+j0lzy7DMXRp/Z5Hyo5kuUlrC0dCLa1UHqtrrK7MR55AVEOihSNp1w+OC+vw
+f0VjE4JUtO7RQEQUmD1tDfLXwNfMFeWaobB2W0WMvRg0IqoitiqPxsPHRm56OxfM
+SRo/Q7QBAoGBAP8DapzBMuaIcJ7cE8Yl07ZGndWWf8buIKIItGF8rkEO3BXhrIke
+EmpOi+ELtpbMOG0APhORZyQ58f4ZOVrqZfneNKtDiEZV4mJZaYUESm1pU+2Y6+y5
+g4bpQSVKN0ow0xR+MH7qDYtSlsmBU7qAOz775L7BmMA1Bnu72aN/H1JBAoGBAMhD
+OzqCSakHOjUbEd22rPwqWmcIyVyo04gaSmcVVT2dHbqR4/t0gX5a9D9U2qwyO6xi
+/R+PXyMd32xIeVR2D/7SQ0x6dK68HXICLV8ofHZ5UQcHbxy5og4v/YxSZVTkN374
+cEsUeyB0s/UPOHLktFU5hpIlON72/Rp7b+pNIwFxAoGAczpq+Qu/YTWzlcSh1r4O
+7OT5uqI3eH7vFehTAV3iKxl4zxZa7NY+wfRd9kFhrr/2myIp6pOgBFl+hC+HoBIc
+JAyIxf5M3GNAWOpH6MfojYmzV7/qktu8l8BcJGplk0t+hVsDtMUze4nFAqZCXBpH
+Kw2M7bjyuZ78H/rgu6TcVUECgYEAo1M5ldE2U/VCApeuLX1TfWDpU8i1uK0zv3d5
+oLKkT1i5KzTak3SEO9HgC1qf8PoS8tfUio26UICHe99rnHehOfivzEq+qNdgyF+A
+M3BoeZMdgzcL5oh640k+Zte4LtDlddcWdhUhCepD7iPYrNNbQ3pkBwL2a9lRuOxc
+7OC2IPECgYBH8f3OrwXjDltIG1dDvuDPNljxLZbFEFbQyVzMePYNftgZknAyGEdh
+NW/LuWeTzstnmz/s6RE3jN5ZrrMa4sW77VA9+yU9QW2dkHqFyukQ4sfuNg6kDDNZ
++lqZYMCLw0M5P9fIbmnIYwey7tXkHfmzoCpnYHGQDN6hL0Bh0zGwmg==
+-----END RSA PRIVATE KEY-----

caddytest/caddy.ca.cer

@@ -0,0 +1,20 @@
+-----BEGIN CERTIFICATE-----
+MIIDSzCCAjOgAwIBAgIUfIRObjWNUA4jxQ/0x8BOCvE2Vw4wDQYJKoZIhvcNAQEL
+BQAwFjEUMBIGA1UEAwwLRWFzeS1SU0EgQ0EwHhcNMTkwODI4MTYyNTU5WhcNMjkw
+ODI1MTYyNTU5WjAWMRQwEgYDVQQDDAtFYXN5LVJTQSBDQTCCASIwDQYJKoZIhvcN
+AQEBBQADggEPADCCAQoCggEBAK5m5elxhQfMp/3aVJ4JnpN9PUSz6LlP6LePAPFU
+7gqohVVFVtDkChJAG3FNkNQNlieVTja/bgH9IcC6oKbROwdY1h0MvNV8AHHigvl0
+3WuJD8g2ReVFXXwsnrPmKXCFzQyMI6TYk3m2gYrXsZOU1GLnfMRC3KAMRgE2F45t
+wOs9hqG169YJ6mM2eQjzjCHWI6S2/iUYvYxRkCOlYUbLsMD/AhgAf1plzg6LPqNx
+tdlwxZnA0ytgkmhK67HtzJu0+ovUCsMv0RwcMhsEo9T8nyFAGt9XLZ63X5WpBCTU
+ApaAUhnG0XnerjmUWb6eUWw4zev54sEfY5F3x002iQaW6cECAwEAAaOBkDCBjTAd
+BgNVHQ4EFgQU4CBUbZsS2GaNIkGRz/cBsD5ivjswUQYDVR0jBEowSIAU4CBUbZsS
+2GaNIkGRz/cBsD5ivjuhGqQYMBYxFDASBgNVBAMMC0Vhc3ktUlNBIENBghR8hE5u
+NY1QDiPFD/THwE4K8TZXDjAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkq
+hkiG9w0BAQsFAAOCAQEAKB3V4HIzoiO/Ch6WMj9bLJ2FGbpkMrcb/Eq01hT5zcfK
+D66lVS1MlK+cRL446Z2b2KDP1oFyVs+qmrmtdwrWgD+nfe2sBmmIHo9m9KygMkEO
+fG3MghGTEcS+0cTKEcoHYWYyOqQh6jnedXY8Cdm4GM1hAc9MiL3/sqV8YCVSLNnk
+oNysmr06/rZ0MCUZPGUtRmfd0heWhrfzAKw2HLgX+RAmpOE2MZqWcjvqKGyaRiaZ
+ks4nJkP6521aC2Lgp0HhCz1j8/uQ5ldoDszCnu/iro0NAsNtudTMD+YoLQxLqdle
+Ih6CW+illc2VdXwj7mn6J04yns9jfE2jRjW/yTLFuQ==
+-----END CERTIFICATE-----

caddytest/caddy.localhost.crt

@@ -0,0 +1,23 @@
+-----BEGIN CERTIFICATE-----
+MIID5zCCAs8CFFmAAFKV79uhzxc5qXbUw3oBNsYXMA0GCSqGSIb3DQEBCwUAMIGv
+MQswCQYDVQQGEwJVUzELMAkGA1UECAwCTlkxGzAZBgNVBAoMEkxvY2FsIERldmVs
+b3BlbWVudDEbMBkGA1UEBwwSTG9jYWwgRGV2ZWxvcGVtZW50MRowGAYDVQQDDBEq
+LmNhZGR5LmxvY2FsaG9zdDEbMBkGA1UECwwSTG9jYWwgRGV2ZWxvcGVtZW50MSAw
+HgYJKoZIhvcNAQkBFhFhZG1pbkBjYWRkeS5sb2NhbDAeFw0yMDAzMDIwODAxMTZa
+Fw0zMDAyMjgwODAxMTZaMIGvMQswCQYDVQQGEwJVUzELMAkGA1UECAwCTlkxGzAZ
+BgNVBAoMEkxvY2FsIERldmVsb3BlbWVudDEbMBkGA1UEBwwSTG9jYWwgRGV2ZWxv
+cGVtZW50MRowGAYDVQQDDBEqLmNhZGR5LmxvY2FsaG9zdDEbMBkGA1UECwwSTG9j
+YWwgRGV2ZWxvcGVtZW50MSAwHgYJKoZIhvcNAQkBFhFhZG1pbkBjYWRkeS5sb2Nh
+bDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJngfeirQkWaU8ihgIC5
+SKpRQX/3koRjljDK/oCbhLs+wg592kIwVv06l7+mn7NSaNBloabjuA1GqyLRsNLL
+ptrv0HvXa5qLx28+icsb2Ny3dJnQaj9w9PwjxQ1qZqEJfWRH1D8Vz9AmB+QSV/Gu
+8e8alGFewlYZVfH1kbxoTT6QorF37TeA3bh1fgKFtzsGYKswcaZNdDBBHzLunCKZ
+HU6U6L45hm+yLADj3mmDLafUeiVOt6MRLLoSD1eLRVSXGrNo+brJ87zkZntI9+W1
+JxOBoXtZCwka7k2DlAtLihsrmBZA2ZC9yVeu/SQy3qb3iCNnTFTCyAnWeTCr6Tcq
+6w8CAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAOWfXqpAmD4C3wGiMeZAeaaS4hDAR
++JmN+avPDA6F6Bq7DB4NJuIwVUlaDL2s07w5VJJtW52aZVKoBlgHR5yG/XUli6J7
+YUJRmdQJvHUSu26cmKvyoOaTrEYbmvtGICWtZc8uTlMf9wQZbJA4KyxTgEQJDXsZ
+B2XFe+wVdhAgEpobYDROi+l/p8TL5z3U24LpwVTcJy5sEZVv7Wfs886IyxU8ORt8
+VZNcDiH6V53OIGeiufIhia/mPe6jbLntfGZfIFxtCcow4IA/lTy1ned7K5fmvNNb
+ZilxOQUk+wVK8genjdrZVAnAxsYLHJIb5yf9O7rr6fWciVMF3a0k5uNK1w==
+-----END CERTIFICATE-----

caddytest/caddy.localhost.key

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEogIBAAKCAQEAmeB96KtCRZpTyKGAgLlIqlFBf/eShGOWMMr+gJuEuz7CDn3a
+QjBW/TqXv6afs1Jo0GWhpuO4DUarItGw0sum2u/Qe9drmovHbz6JyxvY3Ld0mdBq
+P3D0/CPFDWpmoQl9ZEfUPxXP0CYH5BJX8a7x7xqUYV7CVhlV8fWRvGhNPpCisXft
+N4DduHV+AoW3OwZgqzBxpk10MEEfMu6cIpkdTpTovjmGb7IsAOPeaYMtp9R6JU63
+oxEsuhIPV4tFVJcas2j5usnzvORme0j35bUnE4Ghe1kLCRruTYOUC0uKGyuYFkDZ
+kL3JV679JDLepveII2dMVMLICdZ5MKvpNyrrDwIDAQABAoIBAFcPK01zb6hfm12c
++k5aBiHOnUdgc/YRPg1XHEz5MEycQkDetZjTLrRQ7UBSbnKPgpu9lIsOtbhVLkgh
+6XAqJroiCou2oruqr+hhsqZGmBiwdvj7cNF6ADGTr05az7v22YneFdinZ481pStF
+sZocx+bm2+KHMV5zMSwXKyA0xtdJLxs2yklniDBxSZRppgppq1pDPprP5DkgKPfe
+3ekUmbQd5bHmivhW8ItbJLuf82XSsMBZ9ZhKiKIlWlbKAgiSV3SqnUQb5fi7l8hG
+yYZxbuCUIGFwKmEpUBBt/nyxrOlMiNtDh9JhrPmijTV3slq70pCLwLL/Ai2aeear
+EVA5VhkCgYEAyAmxfPqc2P7BsDAp67/sA7OEPso9qM4WyuWiVdlX2gb9TLNLYbPX
+Kk/UmpAIVzpoTAGY5Zp3wkvdD/ou8uUQsE8ioNn4S1a4G9XURH1wVhcEbUiAKI1S
+QVBH9B/Pj3eIp5OTKwob0Wj7DNdxoH7ed/Eok0EaTWzOA8pCWADKv/MCgYEAxOzY
+YsX7Nl+eyZr2+9unKyeAK/D1DCT/o99UUAHx72/xaBVP/06cfzpvKBNcF9iYc+fq
+R1yIUIrDRoSmYKBq+Kb3+nOg1nrqih/NBTokbTiI4Q+/30OQt0Al1e7y9iNKqV8H
+jYZItzluGNrWKedZbATwBwbVCY2jnNl6RMDnS3UCgYBxj3cwQUHLuoyQjjcuO80r
+qLzZvIxWiXDNDKIk5HcIMlGYOmz/8U2kGp/SgxQJGQJeq8V2C0QTjGfaCyieAcaA
+oNxCvptDgd6RBsoze5bLeNOtiqwe2WOp6n5+q5R0mOJ+Z7vzghCayGNFPgWmnH+F
+TeW/+wSIkc0+v5L8TK7NWwKBgBrlWlyLO9deUfqpHqihhICBYaEexOlGuF+yZfqT
+eW7BdFBJ8OYm33sFCR+JHV/oZlIWT8o1Wizd9vPPtEWoQ1P4wg/D8Si6GwSIeWEI
+YudD/HX4x7T/rmlI6qIAg9CYW18sqoRq3c2gm2fro6qPfYgiWIItLbWjUcBfd7Ki
+QjTtAoGARKdRv3jMWL84rlEx1nBRgL3pe9Dt+Uxzde2xT3ZeF+5Hp9NfU01qE6M6
+1I6H64smqpetlsXmCEVKwBemP3pJa6avLKgIYiQvHAD/v4rs9mqgy1RTqtYyGNhR
+1A/6dKkbiZ6wzePLLPasXVZxSKEviXf5gJooqumQVSVhCswyCZ0=
+-----END RSA PRIVATE KEY-----

caddytest/caddytest.go

@@ -0,0 +1,554 @@
+package caddytest
+
+import (
+	"bytes"
+	"context"
+	"crypto/tls"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log"
+	"net"
+	"net/http"
+	"net/http/cookiejar"
+	"os"
+	"path"
+	"reflect"
+	"regexp"
+	"runtime"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/aryann/difflib"
+
+	caddycmd "github.com/caddyserver/caddy/v2/cmd"
+
+	"github.com/caddyserver/caddy/v2/caddyconfig"
+	// plug in Caddy modules here
+	_ "github.com/caddyserver/caddy/v2/modules/standard"
+)
+
+// Defaults store any configuration required to make the tests run
+type Defaults struct {
+	// Port we expect caddy to listening on
+	AdminPort int
+	// Certificates we expect to be loaded before attempting to run the tests
+	Certifcates []string
+	// TestRequestTimeout is the time to wait for a http request to
+	TestRequestTimeout time.Duration
+	// LoadRequestTimeout is the time to wait for the config to be loaded against the caddy server
+	LoadRequestTimeout time.Duration
+}
+
+// Default testing values
+var Default = Defaults{
+	AdminPort:          2999, // different from what a real server also running on a developer's machine might be
+	Certifcates:        []string{"/caddy.localhost.crt", "/caddy.localhost.key"},
+	TestRequestTimeout: 5 * time.Second,
+	LoadRequestTimeout: 5 * time.Second,
+}
+
+var (
+	matchKey  = regexp.MustCompile(`(/[\w\d\.]+\.key)`)
+	matchCert = regexp.MustCompile(`(/[\w\d\.]+\.crt)`)
+)
+
+// Tester represents an instance of a test client.
+type Tester struct {
+	Client       *http.Client
+	configLoaded bool
+	t            *testing.T
+}
+
+// NewTester will create a new testing client with an attached cookie jar
+func NewTester(t *testing.T) *Tester {
+	jar, err := cookiejar.New(nil)
+	if err != nil {
+		t.Fatalf("failed to create cookiejar: %s", err)
+	}
+
+	return &Tester{
+		Client: &http.Client{
+			Transport: CreateTestingTransport(),
+			Jar:       jar,
+			Timeout:   Default.TestRequestTimeout,
+		},
+		configLoaded: false,
+		t:            t,
+	}
+}
+
+type configLoadError struct {
+	Response string
+}
+
+func (e configLoadError) Error() string { return e.Response }
+
+func timeElapsed(start time.Time, name string) {
+	elapsed := time.Since(start)
+	log.Printf("%s took %s", name, elapsed)
+}
+
+// InitServer this will configure the server with a configurion of a specific
+// type. The configType must be either "json" or the adapter type.
+func (tc *Tester) InitServer(rawConfig string, configType string) {
+	if err := tc.initServer(rawConfig, configType); err != nil {
+		tc.t.Logf("failed to load config: %s", err)
+		tc.t.Fail()
+	}
+	if err := tc.ensureConfigRunning(rawConfig, configType); err != nil {
+		tc.t.Logf("failed ensuring config is running: %s", err)
+		tc.t.Fail()
+	}
+}
+
+// InitServer this will configure the server with a configurion of a specific
+// type. The configType must be either "json" or the adapter type.
+func (tc *Tester) initServer(rawConfig string, configType string) error {
+	if testing.Short() {
+		tc.t.SkipNow()
+		return nil
+	}
+
+	err := validateTestPrerequisites(tc.t)
+	if err != nil {
+		tc.t.Skipf("skipping tests as failed integration prerequisites. %s", err)
+		return nil
+	}
+
+	tc.t.Cleanup(func() {
+		if tc.t.Failed() && tc.configLoaded {
+
+			res, err := http.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
+			if err != nil {
+				tc.t.Log("unable to read the current config")
+				return
+			}
+			defer res.Body.Close()
+			body, _ := io.ReadAll(res.Body)
+
+			var out bytes.Buffer
+			_ = json.Indent(&out, body, "", "  ")
+			tc.t.Logf("----------- failed with config -----------\n%s", out.String())
+		}
+	})
+
+	rawConfig = prependCaddyFilePath(rawConfig)
+	client := &http.Client{
+		Timeout: Default.LoadRequestTimeout,
+	}
+	start := time.Now()
+	req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/load", Default.AdminPort), strings.NewReader(rawConfig))
+	if err != nil {
+		tc.t.Errorf("failed to create request. %s", err)
+		return err
+	}
+
+	if configType == "json" {
+		req.Header.Add("Content-Type", "application/json")
+	} else {
+		req.Header.Add("Content-Type", "text/"+configType)
+	}
+
+	res, err := client.Do(req)
+	if err != nil {
+		tc.t.Errorf("unable to contact caddy server. %s", err)
+		return err
+	}
+	timeElapsed(start, "caddytest: config load time")
+
+	defer res.Body.Close()
+	body, err := io.ReadAll(res.Body)
+	if err != nil {
+		tc.t.Errorf("unable to read response. %s", err)
+		return err
+	}
+
+	if res.StatusCode != 200 {
+		return configLoadError{Response: string(body)}
+	}
+
+	tc.configLoaded = true
+	return nil
+}
+
+func (tc *Tester) ensureConfigRunning(rawConfig string, configType string) error {
+	expectedBytes := []byte(prependCaddyFilePath(rawConfig))
+	if configType != "json" {
+		adapter := caddyconfig.GetAdapter(configType)
+		if adapter == nil {
+			return fmt.Errorf("adapter of config type is missing: %s", configType)
+		}
+		expectedBytes, _, _ = adapter.Adapt([]byte(rawConfig), nil)
+	}
+
+	var expected any
+	err := json.Unmarshal(expectedBytes, &expected)
+	if err != nil {
+		return err
+	}
+
+	client := &http.Client{
+		Timeout: Default.LoadRequestTimeout,
+	}
+
+	fetchConfig := func(client *http.Client) any {
+		resp, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
+		if err != nil {
+			return nil
+		}
+		defer resp.Body.Close()
+		actualBytes, err := io.ReadAll(resp.Body)
+		if err != nil {
+			return nil
+		}
+		var actual any
+		err = json.Unmarshal(actualBytes, &actual)
+		if err != nil {
+			return nil
+		}
+		return actual
+	}
+
+	for retries := 10; retries > 0; retries-- {
+		if reflect.DeepEqual(expected, fetchConfig(client)) {
+			return nil
+		}
+		time.Sleep(1 * time.Second)
+	}
+	tc.t.Errorf("POSTed configuration isn't active")
+	return errors.New("EnsureConfigRunning: POSTed configuration isn't active")
+}
+
+const initConfig = `{
+	admin localhost:2999
+}
+`
+
+// validateTestPrerequisites ensures the certificates are available in the
+// designated path and Caddy sub-process is running.
+func validateTestPrerequisites(t *testing.T) error {
+	// check certificates are found
+	for _, certName := range Default.Certifcates {
+		if _, err := os.Stat(getIntegrationDir() + certName); os.IsNotExist(err) {
+			return fmt.Errorf("caddy integration test certificates (%s) not found", certName)
+		}
+	}
+
+	if isCaddyAdminRunning() != nil {
+		// setup the init config file, and set the cleanup afterwards
+		f, err := os.CreateTemp("", "")
+		if err != nil {
+			return err
+		}
+		t.Cleanup(func() {
+			os.Remove(f.Name())
+		})
+		if _, err := f.WriteString(initConfig); err != nil {
+			return err
+		}
+
+		// start inprocess caddy server
+		os.Args = []string{"caddy", "run", "--config", f.Name(), "--adapter", "caddyfile"}
+		go func() {
+			caddycmd.Main()
+		}()
+
+		// wait for caddy to start serving the initial config
+		for retries := 10; retries > 0 && isCaddyAdminRunning() != nil; retries-- {
+			time.Sleep(1 * time.Second)
+		}
+	}
+
+	// one more time to return the error
+	return isCaddyAdminRunning()
+}
+
+func isCaddyAdminRunning() error {
+	// assert that caddy is running
+	client := &http.Client{
+		Timeout: Default.LoadRequestTimeout,
+	}
+	resp, err := client.Get(fmt.Sprintf("http://localhost:%d/config/", Default.AdminPort))
+	if err != nil {
+		return fmt.Errorf("caddy integration test caddy server not running. Expected to be listening on localhost:%d", Default.AdminPort)
+	}
+	resp.Body.Close()
+
+	return nil
+}
+
+func getIntegrationDir() string {
+	_, filename, _, ok := runtime.Caller(1)
+	if !ok {
+		panic("unable to determine the current file path")
+	}
+
+	return path.Dir(filename)
+}
+
+// use the convention to replace /[certificatename].[crt|key] with the full path
+// this helps reduce the noise in test configurations and also allow this
+// to run in any path
+func prependCaddyFilePath(rawConfig string) string {
+	r := matchKey.ReplaceAllString(rawConfig, getIntegrationDir()+"$1")
+	r = matchCert.ReplaceAllString(r, getIntegrationDir()+"$1")
+	return r
+}
+
+// CreateTestingTransport creates a testing transport that forces call dialing connections to happen locally
+func CreateTestingTransport() *http.Transport {
+	dialer := net.Dialer{
+		Timeout:   5 * time.Second,
+		KeepAlive: 5 * time.Second,
+		DualStack: true,
+	}
+
+	dialContext := func(ctx context.Context, network, addr string) (net.Conn, error) {
+		parts := strings.Split(addr, ":")
+		destAddr := fmt.Sprintf("127.0.0.1:%s", parts[1])
+		log.Printf("caddytest: redirecting the dialer from %s to %s", addr, destAddr)
+		return dialer.DialContext(ctx, network, destAddr)
+	}
+
+	return &http.Transport{
+		Proxy:                 http.ProxyFromEnvironment,
+		DialContext:           dialContext,
+		ForceAttemptHTTP2:     true,
+		MaxIdleConns:          100,
+		IdleConnTimeout:       90 * time.Second,
+		TLSHandshakeTimeout:   5 * time.Second,
+		ExpectContinueTimeout: 1 * time.Second,
+		TLSClientConfig:       &tls.Config{InsecureSkipVerify: true}, //nolint:gosec
+	}
+}
+
+// AssertLoadError will load a config and expect an error
+func AssertLoadError(t *testing.T, rawConfig string, configType string, expectedError string) {
+	tc := NewTester(t)
+
+	err := tc.initServer(rawConfig, configType)
+	if !strings.Contains(err.Error(), expectedError) {
+		t.Errorf("expected error \"%s\" but got \"%s\"", expectedError, err.Error())
+	}
+}
+
+// AssertRedirect makes a request and asserts the redirection happens
+func (tc *Tester) AssertRedirect(requestURI string, expectedToLocation string, expectedStatusCode int) *http.Response {
+	redirectPolicyFunc := func(req *http.Request, via []*http.Request) error {
+		return http.ErrUseLastResponse
+	}
+
+	// using the existing client, we override the check redirect policy for this test
+	old := tc.Client.CheckRedirect
+	tc.Client.CheckRedirect = redirectPolicyFunc
+	defer func() { tc.Client.CheckRedirect = old }()
+
+	resp, err := tc.Client.Get(requestURI)
+	if err != nil {
+		tc.t.Errorf("failed to call server %s", err)
+		return nil
+	}
+
+	if expectedStatusCode != resp.StatusCode {
+		tc.t.Errorf("requesting \"%s\" expected status code: %d but got %d", requestURI, expectedStatusCode, resp.StatusCode)
+	}
+
+	loc, err := resp.Location()
+	if err != nil {
+		tc.t.Errorf("requesting \"%s\" expected location: \"%s\" but got error: %s", requestURI, expectedToLocation, err)
+	}
+	if loc == nil && expectedToLocation != "" {
+		tc.t.Errorf("requesting \"%s\" expected a Location header, but didn't get one", requestURI)
+	}
+	if loc != nil {
+		if expectedToLocation != loc.String() {
+			tc.t.Errorf("requesting \"%s\" expected location: \"%s\" but got \"%s\"", requestURI, expectedToLocation, loc.String())
+		}
+	}
+
+	return resp
+}
+
+// CompareAdapt adapts a config and then compares it against an expected result
+func CompareAdapt(t *testing.T, filename, rawConfig string, adapterName string, expectedResponse string) bool {
+	cfgAdapter := caddyconfig.GetAdapter(adapterName)
+	if cfgAdapter == nil {
+		t.Logf("unrecognized config adapter '%s'", adapterName)
+		return false
+	}
+
+	options := make(map[string]any)
+
+	result, warnings, err := cfgAdapter.Adapt([]byte(rawConfig), options)
+	if err != nil {
+		t.Logf("adapting config using %s adapter: %v", adapterName, err)
+		return false
+	}
+
+	// prettify results to keep tests human-manageable
+	var prettyBuf bytes.Buffer
+	err = json.Indent(&prettyBuf, result, "", "\t")
+	if err != nil {
+		return false
+	}
+	result = prettyBuf.Bytes()
+
+	if len(warnings) > 0 {
+		for _, w := range warnings {
+			t.Logf("warning: %s:%d: %s: %s", filename, w.Line, w.Directive, w.Message)
+		}
+	}
+
+	diff := difflib.Diff(
+		strings.Split(expectedResponse, "\n"),
+		strings.Split(string(result), "\n"))
+
+	// scan for failure
+	failed := false
+	for _, d := range diff {
+		if d.Delta != difflib.Common {
+			failed = true
+			break
+		}
+	}
+
+	if failed {
+		for _, d := range diff {
+			switch d.Delta {
+			case difflib.Common:
+				fmt.Printf("  %s\n", d.Payload)
+			case difflib.LeftOnly:
+				fmt.Printf(" - %s\n", d.Payload)
+			case difflib.RightOnly:
+				fmt.Printf(" + %s\n", d.Payload)
+			}
+		}
+		return false
+	}
+	return true
+}
+
+// AssertAdapt adapts a config and then tests it against an expected result
+func AssertAdapt(t *testing.T, rawConfig string, adapterName string, expectedResponse string) {
+	ok := CompareAdapt(t, "Caddyfile", rawConfig, adapterName, expectedResponse)
+	if !ok {
+		t.Fail()
+	}
+}
+
+// Generic request functions
+
+func applyHeaders(t *testing.T, req *http.Request, requestHeaders []string) {
+	requestContentType := ""
+	for _, requestHeader := range requestHeaders {
+		arr := strings.SplitAfterN(requestHeader, ":", 2)
+		k := strings.TrimRight(arr[0], ":")
+		v := strings.TrimSpace(arr[1])
+		if k == "Content-Type" {
+			requestContentType = v
+		}
+		t.Logf("Request header: %s => %s", k, v)
+		req.Header.Set(k, v)
+	}
+
+	if requestContentType == "" {
+		t.Logf("Content-Type header not provided")
+	}
+}
+
+// AssertResponseCode will execute the request and verify the status code, returns a response for additional assertions
+func (tc *Tester) AssertResponseCode(req *http.Request, expectedStatusCode int) *http.Response {
+	resp, err := tc.Client.Do(req)
+	if err != nil {
+		tc.t.Fatalf("failed to call server %s", err)
+	}
+
+	if expectedStatusCode != resp.StatusCode {
+		tc.t.Errorf("requesting \"%s\" expected status code: %d but got %d", req.URL.RequestURI(), expectedStatusCode, resp.StatusCode)
+	}
+
+	return resp
+}
+
+// AssertResponse request a URI and assert the status code and the body contains a string
+func (tc *Tester) AssertResponse(req *http.Request, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+	resp := tc.AssertResponseCode(req, expectedStatusCode)
+
+	defer resp.Body.Close()
+	bytes, err := io.ReadAll(resp.Body)
+	if err != nil {
+		tc.t.Fatalf("unable to read the response body %s", err)
+	}
+
+	body := string(bytes)
+
+	if body != expectedBody {
+		tc.t.Errorf("requesting \"%s\" expected response body \"%s\" but got \"%s\"", req.RequestURI, expectedBody, body)
+	}
+
+	return resp, body
+}
+
+// Verb specific test functions
+
+// AssertGetResponse GET a URI and expect a statusCode and body text
+func (tc *Tester) AssertGetResponse(requestURI string, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+	req, err := http.NewRequest("GET", requestURI, nil)
+	if err != nil {
+		tc.t.Fatalf("unable to create request %s", err)
+	}
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertDeleteResponse request a URI and expect a statusCode and body text
+func (tc *Tester) AssertDeleteResponse(requestURI string, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+	req, err := http.NewRequest("DELETE", requestURI, nil)
+	if err != nil {
+		tc.t.Fatalf("unable to create request %s", err)
+	}
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPostResponseBody POST to a URI and assert the response code and body
+func (tc *Tester) AssertPostResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+	req, err := http.NewRequest("POST", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPutResponseBody PUT to a URI and assert the response code and body
+func (tc *Tester) AssertPutResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+	req, err := http.NewRequest("PUT", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}
+
+// AssertPatchResponseBody PATCH to a URI and assert the response code and body
+func (tc *Tester) AssertPatchResponseBody(requestURI string, requestHeaders []string, requestBody *bytes.Buffer, expectedStatusCode int, expectedBody string) (*http.Response, string) {
+	req, err := http.NewRequest("PATCH", requestURI, requestBody)
+	if err != nil {
+		tc.t.Errorf("failed to create request %s", err)
+		return nil, ""
+	}
+
+	applyHeaders(tc.t, req, requestHeaders)
+
+	return tc.AssertResponse(req, expectedStatusCode, expectedBody)
+}

caddytest/caddytest_test.go

@@ -0,0 +1,33 @@
+package caddytest
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestReplaceCertificatePaths(t *testing.T) {
+	rawConfig := `a.caddy.localhost:9443 {
+		tls /caddy.localhost.crt /caddy.localhost.key {
+		}
+
+		redir / https://b.caddy.localhost:9443/version 301
+    
+		respond /version 200 {
+		  body "hello from a.caddy.localhost"
+		}	
+	  }`
+
+	r := prependCaddyFilePath(rawConfig)
+
+	if !strings.Contains(r, getIntegrationDir()+"/caddy.localhost.crt") {
+		t.Error("expected the /caddy.localhost.crt to be expanded to include the full path")
+	}
+
+	if !strings.Contains(r, getIntegrationDir()+"/caddy.localhost.key") {
+		t.Error("expected the /caddy.localhost.crt to be expanded to include the full path")
+	}
+
+	if !strings.Contains(r, "https://b.caddy.localhost:9443/version") {
+		t.Error("expected redirect uri to be unchanged")
+	}
+}

caddytest/integration/acmeserver_test.go

@@ -0,0 +1,33 @@
+package integration
+
+import (
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestACMEServerDirectory(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		skip_install_trust
+		local_certs
+		admin localhost:2999
+		http_port     9080
+		https_port    9443
+		pki {
+			ca local {
+				name "Caddy Local Authority"
+			}
+		}
+	}
+	acme.localhost:9443 {
+		acme_server
+	}
+  `, "caddyfile")
+	tester.AssertGetResponse(
+		"https://acme.localhost:9443/acme/local/directory",
+		200,
+		`{"newNonce":"https://acme.localhost:9443/acme/local/new-nonce","newAccount":"https://acme.localhost:9443/acme/local/new-account","newOrder":"https://acme.localhost:9443/acme/local/new-order","revokeCert":"https://acme.localhost:9443/acme/local/revoke-cert","keyChange":"https://acme.localhost:9443/acme/local/key-change"}
+`)
+}

caddytest/integration/autohttps_test.go

@@ -0,0 +1,145 @@
+package integration
+
+import (
+	"net/http"
+	"testing"
+
+	"github.com/caddyserver/caddy/v2/caddytest"
+)
+
+func TestAutoHTTPtoHTTPSRedirectsImplicitPort(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		admin localhost:2999
+		skip_install_trust
+		http_port     9080
+		https_port    9443
+	}
+	localhost
+	respond "Yahaha! You found me!"
+  `, "caddyfile")
+
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
+}
+
+func TestAutoHTTPtoHTTPSRedirectsExplicitPortSameAsHTTPSPort(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		skip_install_trust
+		admin localhost:2999
+		http_port     9080
+		https_port    9443
+	}
+	localhost:9443
+	respond "Yahaha! You found me!"
+  `, "caddyfile")
+
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
+}
+
+func TestAutoHTTPtoHTTPSRedirectsExplicitPortDifferentFromHTTPSPort(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		skip_install_trust
+		admin localhost:2999
+		http_port     9080
+		https_port    9443
+	}
+	localhost:1234
+	respond "Yahaha! You found me!"
+  `, "caddyfile")
+
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost:1234/", http.StatusPermanentRedirect)
+}
+
+func TestAutoHTTPRedirectsWithHTTPListenerFirstInAddresses(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+{
+  "admin": {
+	"listen": "localhost:2999"
+  },
+  "apps": {
+    "http": {
+      "http_port": 9080,
+      "https_port": 9443,
+      "servers": {
+        "ingress_server": {
+          "listen": [
+            ":9080",
+            ":9443"
+          ],
+          "routes": [
+            {
+              "match": [
+                {
+				  "host": ["localhost"]
+                }
+              ]
+            }
+          ]
+        }
+      }
+    },
+	"pki": {
+		"certificate_authorities": {
+			"local": {
+				"install_trust": false
+			}
+		}
+	}
+  }
+}
+`, "json")
+	tester.AssertRedirect("http://localhost:9080/", "https://localhost/", http.StatusPermanentRedirect)
+}
+
+func TestAutoHTTPRedirectsInsertedBeforeUserDefinedCatchAll(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		skip_install_trust
+		admin localhost:2999
+		http_port     9080
+		https_port    9443
+		local_certs
+	}
+	http://:9080 {
+		respond "Foo"
+	}
+	http://baz.localhost:9080 {
+		respond "Baz"
+	}
+	bar.localhost {
+		respond "Bar"
+	}
+  `, "caddyfile")
+	tester.AssertRedirect("http://bar.localhost:9080/", "https://bar.localhost/", http.StatusPermanentRedirect)
+	tester.AssertGetResponse("http://foo.localhost:9080/", 200, "Foo")
+	tester.AssertGetResponse("http://baz.localhost:9080/", 200, "Baz")
+}
+
+func TestAutoHTTPRedirectsInsertedBeforeUserDefinedCatchAllWithNoExplicitHTTPSite(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+	{
+		skip_install_trust
+		admin localhost:2999
+		http_port     9080
+		https_port    9443
+		local_certs
+	}
+	http://:9080 {
+		respond "Foo"
+	}
+	bar.localhost {
+		respond "Bar"
+	}
+  `, "caddyfile")
+	tester.AssertRedirect("http://bar.localhost:9080/", "https://bar.localhost/", http.StatusPermanentRedirect)
+	tester.AssertGetResponse("http://foo.localhost:9080/", 200, "Foo")
+	tester.AssertGetResponse("http://baz.localhost:9080/", 200, "Foo")
+}

caddytest/integration/caddyfile_adapt/acme_server_lifetime.txt

@@ -0,0 +1,108 @@
+{
+	pki {
+		ca internal {
+			name "Internal"
+			root_cn "Internal Root Cert"
+			intermediate_cn "Internal Intermediate Cert"
+		}
+		ca internal-long-lived {
+			name "Long-lived"
+			root_cn "Internal Root Cert 2"
+			intermediate_cn "Internal Intermediate Cert 2"
+		}
+	}
+}
+
+acme-internal.example.com {
+	acme_server {
+		ca internal
+	}
+}
+
+acme-long-lived.example.com {
+	acme_server {
+		ca internal-long-lived
+		lifetime 7d
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"acme-long-lived.example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"ca": "internal-long-lived",
+													"handler": "acme_server",
+													"lifetime": 604800000000000
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						},
+						{
+							"match": [
+								{
+									"host": [
+										"acme-internal.example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"ca": "internal",
+													"handler": "acme_server"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		},
+		"pki": {
+			"certificate_authorities": {
+				"internal": {
+					"name": "Internal",
+					"root_common_name": "Internal Root Cert",
+					"intermediate_common_name": "Internal Intermediate Cert"
+				},
+				"internal-long-lived": {
+					"name": "Long-lived",
+					"root_common_name": "Internal Root Cert 2",
+					"intermediate_common_name": "Internal Intermediate Cert 2"
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/auto_https_disable_redirects.txt

@@ -0,0 +1,34 @@
+{
+	auto_https disable_redirects
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"disable_redirects": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/auto_https_ignore_loaded_certs.txt

@@ -0,0 +1,34 @@
+{
+	auto_https ignore_loaded_certs
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"automatic_https": {
+						"ignore_loaded_certificates": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/auto_https_off.txt

@@ -0,0 +1,37 @@
+{
+	auto_https off
+}
+
+localhost
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"tls_connection_policies": [
+						{}
+					],
+					"automatic_https": {
+						"disable": true
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/bind_ipv6.txt

@@ -0,0 +1,29 @@
+example.com {
+	bind tcp6/[::]
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						"tcp6/[::]:443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/enable_tls_for_catch_all_site.txt

@@ -0,0 +1,37 @@
+:8443 {
+	tls internal {
+		on_demand
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8443"
+					],
+					"tls_connection_policies": [
+						{}
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						],
+						"on_demand": true
+					}
+				]
+			}
+		}
+	}
+}
+

caddytest/integration/caddyfile_adapt/encode_options.txt

@@ -0,0 +1,87 @@
+:80
+
+# All the options
+encode gzip zstd {
+	minimum_length 256
+	match {
+		status 2xx 4xx 500
+		header Content-Type text/*
+		header Content-Type application/json*
+		header Content-Type application/javascript*
+		header Content-Type application/xhtml+xml*
+		header Content-Type application/atom+xml*
+		header Content-Type application/rss+xml*
+		header Content-Type application/wasm*
+		header Content-Type image/svg+xml*
+	}
+}
+
+# Long way with a block for each encoding
+encode {
+	zstd
+	gzip 5
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"encodings": {
+										"gzip": {},
+										"zstd": {}
+									},
+									"handler": "encode",
+									"match": {
+										"headers": {
+											"Content-Type": [
+												"text/*",
+												"application/json*",
+												"application/javascript*",
+												"application/xhtml+xml*",
+												"application/atom+xml*",
+												"application/rss+xml*",
+												"application/wasm*",
+												"image/svg+xml*"
+											]
+										},
+										"status_code": [
+											2,
+											4,
+											500
+										]
+									},
+									"minimum_length": 256,
+									"prefer": [
+										"gzip",
+										"zstd"
+									]
+								},
+								{
+									"encodings": {
+										"gzip": {
+											"level": 5
+										},
+										"zstd": {}
+									},
+									"handler": "encode",
+									"prefer": [
+										"zstd",
+										"gzip"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/error_example.txt

@@ -0,0 +1,138 @@
+example.com {
+	root * /srv
+
+	# Trigger errors for certain paths
+	error /private* "Unauthorized" 403
+	error /hidden* "Not found" 404
+
+	# Handle the error by serving an HTML page 
+	handle_errors {
+		rewrite * /{http.error.status_code}.html
+		file_server
+	}
+
+	file_server
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "vars",
+													"root": "/srv"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"error": "Unauthorized",
+													"handler": "error",
+													"status_code": 403
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/private*"
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"error": "Not found",
+													"handler": "error",
+													"status_code": 404
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/hidden*"
+													]
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"handler": "file_server",
+													"hide": [
+														"./Caddyfile"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"errors": {
+						"routes": [
+							{
+								"match": [
+									{
+										"host": [
+											"example.com"
+										]
+									}
+								],
+								"handle": [
+									{
+										"handler": "subroute",
+										"routes": [
+											{
+												"group": "group0",
+												"handle": [
+													{
+														"handler": "rewrite",
+														"uri": "/{http.error.status_code}.html"
+													}
+												]
+											},
+											{
+												"handle": [
+													{
+														"handler": "file_server",
+														"hide": [
+															"./Caddyfile"
+														]
+													}
+												]
+											}
+										]
+									}
+								],
+								"terminal": true
+							}
+						]
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/expression_quotes.txt

@@ -0,0 +1,114 @@
+example.com
+
+@a expression {http.error.status_code} == 400
+abort @a
+
+@b expression {http.error.status_code} == "401"
+abort @b
+
+@c expression {http.error.status_code} == `402`
+abort @c
+
+@d expression "{http.error.status_code} == 403"
+abort @d
+
+@e expression `{http.error.status_code} == 404`
+abort @e
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"abort": true,
+													"handler": "static_response"
+												}
+											],
+											"match": [
+												{
+													"expression": "{http.error.status_code} == 400"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"abort": true,
+													"handler": "static_response"
+												}
+											],
+											"match": [
+												{
+													"expression": "{http.error.status_code} == \"401\""
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"abort": true,
+													"handler": "static_response"
+												}
+											],
+											"match": [
+												{
+													"expression": "{http.error.status_code} == `402`"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"abort": true,
+													"handler": "static_response"
+												}
+											],
+											"match": [
+												{
+													"expression": "{http.error.status_code} == 403"
+												}
+											]
+										},
+										{
+											"handle": [
+												{
+													"abort": true,
+													"handler": "static_response"
+												}
+											],
+											"match": [
+												{
+													"expression": "{http.error.status_code} == 404"
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_disable_canonical_uris.txt

@@ -0,0 +1,32 @@
+:80
+
+file_server {
+	disable_canonical_uris
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"canonical_uris": false,
+									"handler": "file_server",
+									"hide": [
+										"./Caddyfile"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_pass_thru.txt

@@ -0,0 +1,32 @@
+:80
+
+file_server {
+	pass_thru
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "file_server",
+									"hide": [
+										"./Caddyfile"
+									],
+									"pass_thru": true
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_precompressed.txt

@@ -0,0 +1,41 @@
+:80
+
+file_server {
+	precompressed zstd br gzip
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handler": "file_server",
+									"hide": [
+										"./Caddyfile"
+									],
+									"precompressed": {
+										"br": {},
+										"gzip": {},
+										"zstd": {}
+									},
+									"precompressed_order": [
+										"zstd",
+										"br",
+										"gzip"
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/file_server_status.txt

@@ -0,0 +1,112 @@
+localhost
+
+root * /srv
+
+handle /nope* {
+	file_server {
+		status 403
+	}
+}
+
+handle /custom-status* {
+	file_server {
+		status {env.CUSTOM_STATUS}
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handler": "vars",
+													"root": "/srv"
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "file_server",
+																	"hide": [
+																		"./Caddyfile"
+																	],
+																	"status_code": "{env.CUSTOM_STATUS}"
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/custom-status*"
+													]
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"handler": "file_server",
+																	"hide": [
+																		"./Caddyfile"
+																	],
+																	"status_code": 403
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"path": [
+														"/nope*"
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/forward_auth_authelia.txt

@@ -0,0 +1,111 @@
+app.example.com {
+	forward_auth authelia:9091 {
+		uri /api/verify?rd=https://authelia.example.com
+		copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
+	}
+
+	reverse_proxy backend:8080
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"app.example.com"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"handle": [
+												{
+													"handle_response": [
+														{
+															"match": {
+																"status_code": [
+																	2
+																]
+															},
+															"routes": [
+																{
+																	"handle": [
+																		{
+																			"handler": "headers",
+																			"request": {
+																				"set": {
+																					"Remote-Email": [
+																						"{http.reverse_proxy.header.Remote-Email}"
+																					],
+																					"Remote-Groups": [
+																						"{http.reverse_proxy.header.Remote-Groups}"
+																					],
+																					"Remote-Name": [
+																						"{http.reverse_proxy.header.Remote-Name}"
+																					],
+																					"Remote-User": [
+																						"{http.reverse_proxy.header.Remote-User}"
+																					]
+																				}
+																			}
+																		}
+																	]
+																}
+															]
+														}
+													],
+													"handler": "reverse_proxy",
+													"headers": {
+														"request": {
+															"set": {
+																"X-Forwarded-Method": [
+																	"{http.request.method}"
+																],
+																"X-Forwarded-Uri": [
+																	"{http.request.uri}"
+																]
+															}
+														}
+													},
+													"rewrite": {
+														"method": "GET",
+														"uri": "/api/verify?rd=https://authelia.example.com"
+													},
+													"upstreams": [
+														{
+															"dial": "authelia:9091"
+														}
+													]
+												},
+												{
+													"handler": "reverse_proxy",
+													"upstreams": [
+														{
+															"dial": "backend:8080"
+														}
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/forward_auth_rename_headers.txt

@@ -0,0 +1,90 @@
+:8881
+
+forward_auth localhost:9000 {
+	uri /auth
+	copy_headers A>1 B C>3 {
+		D
+		E>5
+	}
+}
+----------
+{
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":8881"
+					],
+					"routes": [
+						{
+							"handle": [
+								{
+									"handle_response": [
+										{
+											"match": {
+												"status_code": [
+													2
+												]
+											},
+											"routes": [
+												{
+													"handle": [
+														{
+															"handler": "headers",
+															"request": {
+																"set": {
+																	"1": [
+																		"{http.reverse_proxy.header.A}"
+																	],
+																	"3": [
+																		"{http.reverse_proxy.header.C}"
+																	],
+																	"5": [
+																		"{http.reverse_proxy.header.E}"
+																	],
+																	"B": [
+																		"{http.reverse_proxy.header.B}"
+																	],
+																	"D": [
+																		"{http.reverse_proxy.header.D}"
+																	]
+																}
+															}
+														}
+													]
+												}
+											]
+										}
+									],
+									"handler": "reverse_proxy",
+									"headers": {
+										"request": {
+											"set": {
+												"X-Forwarded-Method": [
+													"{http.request.method}"
+												],
+												"X-Forwarded-Uri": [
+													"{http.request.uri}"
+												]
+											}
+										}
+									},
+									"rewrite": {
+										"method": "GET",
+										"uri": "/auth"
+									},
+									"upstreams": [
+										{
+											"dial": "localhost:9000"
+										}
+									]
+								}
+							]
+						}
+					]
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options.txt

@@ -0,0 +1,82 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	grace_period 5s
+	shutdown_delay 10s
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_ca_root /path/to/ca.crt
+	ocsp_stapling off
+
+	email test@example.com
+	admin off
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	local_certs
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"disabled": true
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"grace_period": 5000000000,
+			"shutdown_delay": 10000000000,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						],
+						"key_type": "ed25519",
+						"disable_ocsp_stapling": true
+					}
+				],
+				"on_demand": {
+					"ask": "https://example.com",
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					}
+				}
+			},
+			"disable_ocsp_stapling": true
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_acme.txt

@@ -0,0 +1,93 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_eab {
+		key_id 4K2scIVbBpNd-78scadB2g
+		mac_key abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh
+	}
+	acme_ca_root /path/to/ca.crt
+	email test@example.com
+	admin off
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	storage_clean_interval 7d
+	renew_interval 1d
+	ocsp_interval 2d
+
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"disabled": true
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuers": [
+							{
+								"ca": "https://example.com",
+								"email": "test@example.com",
+								"external_account": {
+									"key_id": "4K2scIVbBpNd-78scadB2g",
+									"mac_key": "abcdefghijklmnopqrstuvwx-abcdefghijklnopqrstuvwxyz12ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgh"
+								},
+								"module": "acme",
+								"trusted_roots_pem_files": [
+									"/path/to/ca.crt"
+								]
+							}
+						],
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"ask": "https://example.com",
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					}
+				},
+				"ocsp_interval": 172800000000000,
+				"renew_interval": 86400000000000,
+				"storage_clean_interval": 604800000000000
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_admin.txt

@@ -0,0 +1,83 @@
+{
+	debug
+	http_port 8080
+	https_port 8443
+	default_sni localhost
+	order root first
+	storage file_system {
+		root /data
+	}
+	acme_ca https://example.com
+	acme_ca_root /path/to/ca.crt
+
+	email test@example.com
+	admin {
+		origins localhost:2019 [::1]:2019 127.0.0.1:2019 192.168.10.128
+	}
+	on_demand_tls {
+		ask https://example.com
+		interval 30s
+		burst 20
+	}
+	local_certs
+	key_type ed25519
+}
+
+:80
+----------
+{
+	"admin": {
+		"listen": "localhost:2019",
+		"origins": [
+			"localhost:2019",
+			"[::1]:2019",
+			"127.0.0.1:2019",
+			"192.168.10.128"
+		]
+	},
+	"logging": {
+		"logs": {
+			"default": {
+				"level": "DEBUG"
+			}
+		}
+	},
+	"storage": {
+		"module": "file_system",
+		"root": "/data"
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"https_port": 8443,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		},
+		"tls": {
+			"automation": {
+				"policies": [
+					{
+						"issuers": [
+							{
+								"module": "internal"
+							}
+						],
+						"key_type": "ed25519"
+					}
+				],
+				"on_demand": {
+					"ask": "https://example.com",
+					"rate_limit": {
+						"interval": 30000000000,
+						"burst": 20
+					}
+				}
+			}
+		}
+	}
+}

caddytest/integration/caddyfile_adapt/global_options_admin_with_persist_config_off.txt

@@ -0,0 +1,36 @@
+{
+	http_port 8080
+	persist_config off
+	admin {
+		origins localhost:2019 [::1]:2019 127.0.0.1:2019 192.168.10.128
+	}
+}
+
+:80
+----------
+{
+	"admin": {
+		"listen": "localhost:2019",
+		"origins": [
+			"localhost:2019",
+			"[::1]:2019",
+			"127.0.0.1:2019",
+			"192.168.10.128"
+		],
+		"config": {
+			"persist": false
+		}
+	},
+	"apps": {
+		"http": {
+			"http_port": 8080,
+			"servers": {
+				"srv0": {
+					"listen": [
+						":80"
+					]
+				}
+			}
+		}
+	}
+}